LLM Evaluation Design: From Failure to Test Case

Start with the job: turn production failure modes into evaluation cases

Evaluation design begins with a concrete failure and the decision it should inform. Preserve the input and system context, describe the unacceptable consequence and write an observable criterion before selecting a metric.

Examples used repeatedly during prompt development are no longer an independent release check. Separate development, validation and held-out regression uses even when the dataset is small.

Keep this page's decision boundary canonical

Keep case design independent from the harness that eventually runs it. This cluster owns failure taxonomy, case structure, criteria, provenance and split policy. Dataset and LLM-as-judge pages handle storage and evaluator-specific questions, while the agent cluster extends the model to trajectories. That boundary lets teams migrate tooling without losing the meaning of the regression set or accidentally turning prompt-development examples into held-out evidence.

Write criteria at the level a reviewer can observe. Replace broad labels such as helpful or correct with required facts, prohibited claims, supported citations, valid tool parameters or task-specific rubric dimensions. When several properties matter, score them separately before defining any aggregate. This makes failure analysis actionable and prevents one severe defect from disappearing inside a generally positive holistic judgment.

Dataset composition should reflect the decision boundary, not raw traffic alone. Sample common paths, difficult boundary cases and high-consequence rare events, then record their intended role. Weighting may reflect product exposure, but release gates can still preserve zero-tolerance cases for prohibited actions. Inspect near-duplicates so paraphrase families do not dominate, and keep metadata that permits slice analysis without exposing user identity. A single average across heterogeneous cases is rarely enough to locate the next engineering change.

Make the operating boundary visible

Error analysis groups failures into a taxonomy. Each case stores inputs, relevant context, expected properties and metadata; evaluators apply criteria, while review of disagreements improves both rubric and dataset.

FIG. 01 / Process map

Failure to versioned case

Build a reproducible path

For LLM Evaluation Design: From Failure to Test Case, use a small fixture that another developer can repeat without privileged production data. Change one boundary at a time and preserve the exact configuration needed to explain how the page's decision was reached.

1. Sample failures from real or controlled representative workflows.
2. Label the consequence and assign one canonical failure class.
3. Write a case with only the context the shipped system receives.
4. Calibrate its evaluator, split the dataset and define the release decision.

Keep secrets outside the llm evaluation design artifact. Record variable names, scopes and owners, then verify the relevant system of record whenever this tool or workflow can change external state.

Record evidence that survives a rerun

Case provenance matters because duplicates and synthetic variants can create false confidence. Record source, consent, transformations, split and when the case entered the evaluation set.

• Failure source and user consequence
• Input, system context and expected properties
• Taxonomy label, rubric and reviewer notes
• Dataset version, split and leakage controls

Date the LLM Evaluation Design: From Failure to Test Case record and keep factual observations separate from inference. If a claim depends on a hosted service, preview feature or moving SDK, name that dependency beside the claim.

Use a decision rule and a stopping rule

Prioritize cases by frequency and consequence, while retaining rare catastrophic failures as explicit gates. Merge duplicate intent so a large family of near-identical prompts does not dominate the score.

Have reviewers label a blind sample, inspect disagreement and test whether tiny irrelevant wording changes alter the result. Hold new production incidents out until the current candidate is frozen.

FIG. 02 / Decision aid

Place cases without leakage

Protect against predictable failure and continue deliberately

For LLM Evaluation Design: From Failure to Test Case, the architecture review flags three recurring failure modes: metrics are chosen before error analysis; an LLM judge is not calibrated; the evaluation dataset leaks or drifts. Treat them as release checks, not footnotes. This page remains draft when its exact implementation or intent evidence is still research-gated.

Use the LLM evals field guide next: it builds evaluation from observed failures and release decisions.

Use the eval dataset guide next: it handles provenance, splits, duplicates and drift.

Use the LLM-as-judge guide next: it calibrates model judgments against explicit rubrics and reviewers.

Use the agent eval guide next: it evaluates trajectories, tools, state and outcomes together.