Start with the job: choose an evaluation framework for a defined system and dataset

An evaluation framework is a harness for cases, candidates, evaluators and results. Choose it after defining the system and dataset; otherwise the framework's built-in metric menu will shape the question backward.

Libraries, hosted platforms and observability products solve overlapping but different jobs. Compare the concrete deployment mode you would use, including where prompts, outputs and labels are stored.

Keep this page's decision boundary canonical

This cluster owns harness selection, not a universal tool ranking. Compare code-first libraries and hosted platforms only after defining the case schema, candidate interface, evaluator types and data boundary. DeepEval and Langfuse pages can then document distinct workflows without pretending their features are equivalent. The outcome should be a reproducible choice for one evaluation system, with clear costs for model calls, traces and collaborative review.

Evaluate debugging behavior, not just suite execution. When a case fails, a developer needs the candidate input and output, relevant context, evaluator rationale, configuration and a path back to the source case without exposing unrelated private data. Run a deliberately malformed case and a failing evaluator to see whether infrastructure errors remain distinct from product failures. A framework that reports both as a low quality score can create misleading release decisions.

Concurrency and caching also affect evidence. Parallel model calls can hit rate limits or reorder records, while cached outputs can make a rerun appear stable even when the candidate was never invoked. Document retry, cache and sampling settings and include unique run identifiers. Cost comparisons should use the same candidate and evaluator boundaries, counting both product and grading calls. Otherwise a harness can seem cheaper simply because part of its work is omitted from the ledger.

Make the operating boundary visible

A harness loads versioned cases, invokes the candidate system, applies deterministic or model-based evaluators and persists results for comparison. Some tools also connect cases to production traces or CI gates.

FIG. 01 / Conceptual model

An eval harness around the system

Architecture map of dataset, candidate system, evaluators, result store and CI
Conceptual model: the framework coordinates evidence but does not define the product criterion.

Build a reproducible path

For LLM Evaluation Frameworks: Choose an Eval Harness, use a small fixture that another developer can repeat without privileged production data. Change one boundary at a time and preserve the exact configuration needed to explain how the page's decision was reached.

  1. Define case schema, candidate interface and required evaluator types.
  2. Set hard gates for data location, CI and trace integration.
  3. Implement the same small suite in shortlisted harnesses.
  4. Compare reproducibility, debugging, export and ongoing cost.

Keep secrets outside the llm evaluation framework artifact. Record variable names, scopes and owners, then verify the relevant system of record whenever this tool or workflow can change external state.

Record evidence that survives a rerun

Preserve framework version, configuration, dataset hash and raw per-case output. Built-in metric names are not equivalent across frameworks unless their prompts, models, thresholds and aggregation are also aligned.

  • Dataset and candidate-system interface
  • Evaluator implementation and model dependency
  • Result store, trace link and export format
  • CI behavior, concurrency, cost and failure reporting

Date the LLM Evaluation Frameworks: Choose an Eval Harness record and keep factual observations separate from inference. If a claim depends on a hosted service, preview feature or moving SDK, name that dependency beside the claim.

Use a decision rule and a stopping rule

Choose the harness that makes your defined criteria easiest to reproduce and inspect. Prefer a simple test library when CI is central; prefer a platform when trace-linked review and collaborative labeling justify its data boundary.

Run the suite twice on a frozen candidate, export results and recreate one comparison outside the UI. Differences should be attributable to model nondeterminism or evaluator behavior, not hidden framework state.

FIG. 02 / Decision aid

Choose the harness from the job

Decision matrix for LLM evaluation frameworks by CI, trace and data-boundary needs
Decision aid: workflow and data boundaries determine fit before metric count.

Protect against predictable failure and continue deliberately

For LLM Evaluation Frameworks: Choose an Eval Harness, the architecture review flags three recurring failure modes: metrics are chosen before error analysis; an LLM judge is not calibrated; the evaluation dataset leaks or drifts. Treat them as release checks, not footnotes. This page remains draft when its exact implementation or intent evidence is still research-gated.

Use the LLM evals field guide next: it builds evaluation from observed failures and release decisions.

Use the DeepEval lab note next: it turns a small set of criteria into an inspectable code-first suite.

Use the Langfuse eval guide next: it covers trace-linked evaluation and collaborative review.

Use the evaluation design guide next: it turns production errors into leakage-aware cases and criteria.