Evaluate the repository change, not the chat
Coding-agent evals should measure whether an agent can produce a correct, reviewable repository change from a frozen task. They must preserve the repository state, tools, permissions, and acceptance checks that shaped the outcome. This page owns one search job: evaluate coding agents on reproducible repository tasks. It does not promise a universal product ranking, an undisclosed benchmark, or hands-on results that are not present in the evidence ledger.
Give developers a source-led, reproducible answer for how to evaluate coding agents on reproducible repository tasks, with explicit version and stop conditions. In practice, that means separating documented behavior from inference, naming the consequence of being wrong, and defining the evidence that would change the decision.
For the coding agent evals decision, the surrounding Agent Evals: Trajectories, Tools, and Outcomes guide defines the nearest architectural boundary and prevents this page from absorbing a broader search job.
Freeze task, environment, and authority
Package each task with a base commit, issue statement, allowed resources, environment image, time and network policy, acceptance tests, hidden checks, and a trace that connects actions to the final patch. The artifact should be portable enough for another engineer to inspect without relying on a private chat transcript or the memory of the person who ran it.
For coding agent evals, the source ledger uses current first-party material from OpenAI and OpenAI to define documented concepts and interfaces. Those sources do not prove performance on this site's hypothetical setup, so every comparative or operational conclusion remains tied to the recorded artifact and a local verification step.
After the coding agent evals evidence is recorded, use Tool-Use Evals for LLM Agents; it covers the adjacent implementation handoff without duplicating the protocol here.
Run a reproducible repository task
The order matters for coding agent evals. Starting with tooling or a score before the evidence boundary is defined makes later results hard to interpret. Keep each step small enough that its input, authority, output, and failure state can be reviewed independently.
- Choose repository tasks that resemble the maintenance, debugging, and feature work the team actually delegates.
- Freeze the base commit and environment while keeping acceptance tests independent from the agent prompt.
- Run the agent with recorded tool and permission boundaries, then capture patch, tests, retries, and stop reason.
- Review correctness, scope discipline, and reproducibility; rerun failures before attributing them to capability.
Record the exact coding agent evals configuration and environment beside the artifact, but do not invent a version number in evergreen copy. At execution time, pin the tested release, preserve command output or trace evidence, and stop when the next action requires new authority or an unverifiable assumption.
Repository-task evaluation loop
Separate capability from environment
For coding agent evals, the architecture flags three recurring risks for this family: metrics are chosen before error analysis, an LLM judge is not calibrated, and the evaluation dataset leaks or drifts. They are not abstract caveats; each can make a polished result unusable for the decision this page owns.
- Public benchmark exposure can let model or prompt design absorb task-specific solutions.
- A passing test suite may miss unnecessary edits, insecure shortcuts, or changes outside the issue scope.
- Comparing runs with different tools, network access, or time budgets confounds the agent with its environment.
Treat a coding agent evals failure label as the start of investigation, not as an explanation. Preserve the case, identify which evidence or control was missing, and rerun one changed condition at a time. That discipline separates a tool limitation from a bad task definition, weak context, an unsafe permission, or a broken test harness.
Verify outcome and process together
Verification for “evaluate coding agents on reproducible repository tasks” needs a stopping rule that another engineer can apply. The checks below favor direct artifacts and observable state over confidence, verbosity, or vendor reputation. A failed check keeps the conclusion provisional even when the generated output appears convincing.
| Check | Evidence to retain | Stop condition |
|---|---|---|
| Task fidelity | The issue and repository reflect real delegated work | A toy task rewards a narrow trick |
| Outcome | Visible and hidden checks test the requested behavior | Only patch presence is scored |
| Process | Permissions, tool calls, retries, and human intervention are logged | The final summary hides the path |
Run the coding agent evals gate against both an expected success and at least one denied, malformed, or recovery path. Store disagreements and residual risk beside the result. If the evidence cannot distinguish a system failure from an evaluation failure, improve the instrument before using its score to approve a release.
When the coding agent evals gate exposes a neighboring problem, continue with LLM Evals: Build Evidence Before Metrics and carry this page's evidence record into that step.
Coding-agent evidence matrix
Apply results only to matched work
Use a coding-agent score only for the task family and control surface that produced it. For adoption, prefer a small internal suite with reviewed diffs over a broad headline benchmark that does not resemble the repository. This rule applies to the documented search job, not to every use of coding agent evals. A different repository, data boundary, model, tool set, or consequence requires a new dated check.
End the coding agent evals record with the owner, next review trigger, and one of four outcomes: proceed within the tested boundary, reduce scope, gather missing evidence, or reject the approach. This preserves a useful negative result and prevents scheduled editorial copy from implying an experiment that was never run.