Make reproducibility a harness property
A coding-agent benchmark harness reconstructs a repository task, runs an agent under declared controls, captures the patch and trace, and verifies the result before resetting all state. Repetition depends on the harness, not on a prose prompt alone. This page owns one search job: run coding agents against repeatable repository tasks. It does not promise a universal product ranking, an undisclosed benchmark, or hands-on results that are not present in the evidence ledger.
Give developers a source-led, reproducible answer for how to run coding agents against repeatable repository tasks, with explicit version and stop conditions. In practice, that means separating documented behavior from inference, naming the consequence of being wrong, and defining the evidence that would change the decision.
For the coding agent benchmark decision, the surrounding Coding Agent Benchmarks: Measure Repository Work guide defines the nearest architectural boundary and prevents this page from absorbing a broader search job.
Bundle environment, policy, trace, and checks
Package a task manifest, base repository archive or commit, container image digest, setup and reset scripts, policy file, tests, resource budget, event log schema, result bundle, and adjudication record. The artifact should be portable enough for another engineer to inspect without relying on a private chat transcript or the memory of the person who ran it.
For coding agent benchmark, the source ledger uses current first-party material from Google and GitHub to define documented concepts and interfaces. Those sources do not prove performance on this site's hypothetical setup, so every comparative or operational conclusion remains tied to the recorded artifact and a local verification step.
After the coding agent benchmark evidence is recorded, use AI Coding Agents: Capabilities, Limits, and Tests; it covers the adjacent implementation handoff without duplicating the protocol here.
Reset, run, verify, and destroy
The order matters for coding agent benchmark. Starting with tooling or a score before the evidence boundary is defined makes later results hard to interpret. Keep each step small enough that its input, authority, output, and failure state can be reviewed independently.
- Build an immutable base environment and verify that setup reaches the same starting state twice.
- Define the task, allowed tools, network and credential policy, time budget, and external stop conditions.
- Run the agent while capturing commands, file changes, resource use, approvals, tests, and terminal reason.
- Archive the result, execute hidden checks and diff review, then destroy and rebuild the environment for reruns.
Record the exact coding agent benchmark configuration and environment beside the artifact, but do not invent a version number in evergreen copy. At execution time, pin the tested release, preserve command output or trace evidence, and stop when the next action requires new authority or an unverifiable assumption.
Benchmark harness lifecycle
Control mutable infrastructure
For coding agent benchmark, the architecture flags three recurring risks for this family: benchmark tasks do not match real repository work, autonomy claims exceed the tested setup, and tools or secrets are granted without containment. They are not abstract caveats; each can make a polished result unusable for the decision this page owns.
- Mutable package sources and remote APIs can change a supposedly frozen task between runs.
- Hidden tests that encode one implementation rather than behavior can penalize valid solutions.
- Incomplete cleanup can leak files, ports, processes, caches, or credentials into the next candidate run.
Treat a coding agent benchmark failure label as the start of investigation, not as an explanation. Preserve the case, identify which evidence or control was missing, and rerun one changed condition at a time. That discipline separates a tool limitation from a bad task definition, weak context, an unsafe permission, or a broken test harness.
Verify reset, capture, and isolation
Verification for “run coding agents against repeatable repository tasks” needs a stopping rule that another engineer can apply. The checks below favor direct artifacts and observable state over confidence, verbosity, or vendor reputation. A failed check keeps the conclusion provisional even when the generated output appears convincing.
| Check | Evidence to retain | Stop condition |
|---|---|---|
| Reset | Two clean starts produce the same dependency and repository state | A previous run affects the next |
| Capture | Patch, tests, trace, policy, and versions share one run ID | Evidence must be reconstructed manually |
| Isolation | Task code cannot reach harness secrets or other runs | The benchmark grants host-level access |
Run the coding agent benchmark gate against both an expected success and at least one denied, malformed, or recovery path. Store disagreements and residual risk beside the result. If the evidence cannot distinguish a system failure from an evaluation failure, improve the instrument before using its score to approve a release.
Harness acceptance gate
Qualify results with harness health
Trust the harness only after repeat-start, failure-path, and cleanup tests pass without agent participation. When infrastructure noise remains, report it separately and rerun before labeling an agent failure. This rule applies to the documented search job, not to every use of coding agent benchmark. A different repository, data boundary, model, tool set, or consequence requires a new dated check.
End the coding agent benchmark record with the owner, next review trigger, and one of four outcomes: proceed within the tested boundary, reduce scope, gather missing evidence, or reject the approach. This preserves a useful negative result and prevents scheduled editorial copy from implying an experiment that was never run.