Start with the job: evaluate multi-step agents beyond final-answer accuracy

Agent evals must inspect both what the system achieved and how it used tools and state to get there. A correct final answer can conceal excessive calls, unauthorized reads or a failed action summarized as success.

Exact trajectory matching is usually too brittle when several valid paths exist. Evaluate necessary and prohibited events, final state and resource constraints instead of demanding one hidden reasoning sequence.

Keep this page's decision boundary canonical

Define the agent as a system under test, including tools and external state. This cluster owns trajectory events, outcome assertions, side effects and recovery behavior; coding-agent and tool-use pages apply that contract to narrower systems. The scope avoids duplicating general evaluator selection while making agent-only failure modes visible. A run passes only when both the intended outcome and the prohibited-path checks satisfy the release rule.

Define outcome evidence at the system boundary. If the task is to update an issue, inspect the issue; if it is to change code, inspect the repository and checks. The agent's final message is only a claim about completion. Store external identifiers or sanitized state hashes that let the evaluator confirm the intended effect and detect duplicate, partial or cross-target actions.

Trajectory criteria should focus on observable safety and efficiency properties rather than hidden reasoning. Examples include using only allowed tools, requesting approval before a write, staying within a call budget, recovering from a declared failure and terminating without loops. Allow multiple valid sequences when they satisfy those properties. For model-based review, present the same redacted trace fields to human calibrators and the judge so disagreement reflects the rubric rather than unequal evidence.

Make the operating boundary visible

An agent run produces a trajectory of model turns, tool calls, observations, state changes and external effects. The eval combines deterministic assertions over events and systems of record with calibrated judgment for properties that remain semantic.

FIG. 01 / Conceptual model

Outcome plus trajectory

Trace map of agent turns, tool calls, state changes and final system outcome
Conceptual model: agent quality includes the path, permissions and external state—not only the final sentence.

Build a reproducible path

For Agent Evals: Trajectories, Tools, and Outcomes, use a small fixture that another developer can repeat without privileged production data. Change one boundary at a time and preserve the exact configuration needed to explain how the page's decision was reached.

  1. Define the task outcome and all prohibited consequences.
  2. Instrument tool calls, state transitions, approvals and costs.
  3. Assert deterministic properties before adding an LLM judge.
  4. Inject tool errors and compare recovery, final state and side effects.

Keep secrets outside the agent evals artifact. Record variable names, scopes and owners, then verify the relevant system of record whenever this tool or workflow can change external state.

Record evidence that survives a rerun

Keep the full versioned run configuration and sanitized trace beside downstream evidence. Token and latency totals require a common boundary; otherwise frameworks count different parts of the workflow.

  • Task fixture and final-state assertion
  • Tool calls, parameters and permission decisions
  • Trajectory events, retries and termination reason
  • Model, prompt, tool versions, usage and downstream audit

Date the Agent Evals: Trajectories, Tools, and Outcomes record and keep factual observations separate from inference. If a claim depends on a hosted service, preview feature or moving SDK, name that dependency beside the claim.

Use a decision rule and a stopping rule

Gate releases on prohibited actions and required outcomes, then use efficiency and qualitative scores to compare survivors. Do not average a severe security failure into a favorable overall score.

Replay deterministic tool fixtures, force timeout and malformed responses and audit the system of record for duplicate effects. Calibrate semantic trajectory judgments against reviewers who can see the same trace.

FIG. 02 / Decision aid

Choose checks by event type

Decision matrix matching agent outcomes and trajectory properties to deterministic or judgment evaluators
Decision aid: assert observable events in code and reserve calibrated judgment for semantic properties.

Protect against predictable failure and continue deliberately

For Agent Evals: Trajectories, Tools, and Outcomes, the architecture review flags three recurring failure modes: metrics are chosen before error analysis; an LLM judge is not calibrated; the evaluation dataset leaks or drifts. Treat them as release checks, not footnotes. This page remains draft when its exact implementation or intent evidence is still research-gated.

Use the LLM evals field guide next: it builds evaluation from observed failures and release decisions.

Use the coding-agent eval guide next: it applies outcome and process checks to repository work.

Use the tool-use eval guide next: it tests tool choice, parameters, permissions and downstream state.

Use the eval framework selection guide next: it matches harness choice to datasets, CI and trace requirements.