Start with the job: design an evaluation system from observed errors

LLM evals are repeatable evidence for a specific product decision. Start with failures users or reviewers can recognize, convert them into cases and criteria, then choose the simplest evaluator that can apply those criteria reliably.

A public benchmark or one aggregate score rarely represents a private application. Evaluation must include the system prompt, retrieval, tools and other context that shape the behavior being shipped.

Keep this page's decision boundary canonical

Treat evaluation as a maintained evidence system. This pillar owns the loop from error sampling through release decisions, while framework, dataset, judge and agent pages own narrower implementation questions. The boundary keeps a metric library from defining product quality and ensures that every supporting technique remains tied to a failure class, a versioned case and a decision the team is prepared to make.

Connect each criterion to a product action. A retrieval-citation check may block release, trigger document reindexing or route a response to review; if no one knows what a failure changes, the metric is collecting information without supporting a decision. Define owners and thresholds before the comparison run, then inspect cases near the boundary. This prevents teams from selecting the most favorable score after seeing results.

Evaluation maintenance needs change control. New cases, rubric edits, evaluator-model updates and candidate changes can all move results, so version them independently. Run a stable reference candidate alongside the current candidate to distinguish evaluator drift from product change. When a grader or dataset changes materially, avoid comparing the new score directly with an old dashboard series until a bridged run establishes how the measurement changed.

Make the operating boundary visible

The loop moves from sampled failures to a taxonomy, versioned dataset, evaluator and run comparison. Error analysis after each run adds new failure classes and removes redundant cases rather than merely growing the test set.

FIG. 01 / Process map

Failure-led evaluation loop

Process map from observed LLM failures through taxonomy, cases, evaluators and regression decisions
Process map: observed failures become versioned cases and repeatable release evidence.

Build a reproducible path

For LLM Evals: Build Evidence Before Metrics, use a small fixture that another developer can repeat without privileged production data. Change one boundary at a time and preserve the exact configuration needed to explain how the page's decision was reached.

  1. Collect representative failures and preserve their full system context.
  2. Group errors by observable user or system consequence.
  3. Write criteria and use deterministic checks wherever possible.
  4. Run candidates, inspect disagreements and version cases, graders and configuration.

Keep secrets outside the llm evals artifact. Record variable names, scopes and owners, then verify the relevant system of record whenever this tool or workflow can change external state.

Record evidence that survives a rerun

Every result should identify dataset version, candidate configuration, evaluator version and run date. Pair aggregate rates with examples from major failure classes so a score remains connected to engineering action.

  • Case origin, consent and dataset split
  • Input, context and expected property
  • Evaluator, rubric and calibration evidence
  • Model, prompt, tool configuration and run output

Date the LLM Evals: Build Evidence Before Metrics record and keep factual observations separate from inference. If a claim depends on a hosted service, preview feature or moving SDK, name that dependency beside the claim.

Use a decision rule and a stopping rule

Ship or roll back from predeclared criteria tied to user risk, not from whichever metric moved favorably. Treat improvements within evaluator disagreement or unstable samples as unresolved.

Blind-review a sample, compare evaluator judgments with humans and rerun unchanged cases to detect nondeterminism. Add production failures to a held-out regression set without leaking them into prompt tuning first.

FIG. 02 / Decision aid

Match evaluator to property

Decision matrix matching deterministic and judgment-based evaluators to output properties
Decision aid: use deterministic checks when possible and calibrated judgment only when the criterion requires it.

Protect against predictable failure and continue deliberately

For LLM Evals: Build Evidence Before Metrics, the architecture review flags three recurring failure modes: metrics are chosen before error analysis; an LLM judge is not calibrated; the evaluation dataset leaks or drifts. Treat them as release checks, not footnotes. This page remains draft when its exact implementation or intent evidence is still research-gated.

Use the eval framework selection guide next: it matches harness choice to datasets, CI and trace requirements.

Use the evaluation design guide next: it turns production errors into leakage-aware cases and criteria.

Use the agent eval guide next: it evaluates trajectories, tools, state and outcomes together.