Building Evals

Convincing Code Is Not Working Code

An agent can produce a diff that reads beautifully and still breaks production.

Evals are the systematic frameworks that settle the question. They run an agent against representative repositories and measure the result with deterministic tests and graders.

The goal is a baseline you can trust. Without one, every claim about agent quality is anecdote.

Correctness is only the first axis. Maintainability and security matter just as much.

The Anatomy of an Eval

An eval is not a single test. It is a stack of parts that must all exist.

You start with representative repositories, add deterministic tests, then define rubrics and graders. Review closes the loop on correctness, maintainability, and security.

Skip any layer and the score becomes noise. A rubric without deterministic tests measures taste, not behavior.

Rubrics and Graders

A rubric states what a good answer looks like. A grader applies that rubric and produces a score.

Keep them separate. The rubric is the specification; the grader is the implementation.

Good rubrics score more than passing tests. They ask whether the change is maintainable and whether it introduces a security weakness.

Deterministic graders come first. Judgment-based scoring belongs on top of them, never instead of them.

Catching Regressions

A regression is behavior that used to work and no longer does.

Catch them with versioned tasks so today's score compares to last week's. Layer unit tests, integration tests, static checks, and diff review underneath.

Then run canaries before a full rollout, and sort every failure into a taxonomy so patterns become visible instead of anonymous.

Monitoring and Adversarial Cases

Evals run before release. Monitoring watches what happens after it.

Both need adversarial cases: tasks built to make the agent fail in the way you most fear.

Feed real production failures back into the versioned task set. That is how a suite stays honest instead of decaying into tasks the agent already memorized.

Least Privilege for Agents

Evals tell you how often an agent fails. Permissions decide how much a failure costs.

Apply least privilege: give the agent the smallest scope that lets it do the job.

In practice that means scoped repositories, sandboxes, network controls, and secret isolation. The agent should never see a credential it does not need.

Approvals and Reversible Changes

Least privilege limits reach. Approvals and reversibility limit consequence.

Route risky actions through a human gate. Keep every change reversible so a bad merge is an undo, not an incident.

Put it together: evals prove capability, monitoring catches drift, permissions bound the blast radius.

That combination is what makes autonomous code generation trustworthy in production.

A regression is behavior that worked before but no longer works after a change.

• True
• False

Answer: True

Correct. A regression is a previously working behavior that has stopped working, often because of a code change.

A coding agent produces a polished-looking diff. Which evaluation approach best determines whether it actually works in production-like codebases?

• Ask a reviewer whether the diff looks professional
• Run the change on representative repositories with deterministic tests and graders
• Measure how many lines the agent changed
• Accept the change if it uses familiar coding patterns

Answer: Run the change on representative repositories with deterministic tests and graders

Representative repositories, deterministic tests, and graders provide repeatable evidence about behavior instead of relying on how convincing the diff looks.

A rubric specifies what good code looks like, while a grader applies that specification and assigns a score.

• True
• False

Answer: True

Correct. The rubric is the evaluation specification; the grader is the implementation that evaluates work against it.