AI Labnotes
Index
Evidence layer

LLM Evals & Reliability

Evaluation design, error analysis, datasets and repeatable checks for agent behavior.

11mapped notes1 live · 3 scheduled · 7 research
Content graph / 11 nodes

Research path

Start with the pillar, move into a focused cluster, then follow its supporting field notes. Every node in the LLM Evals & Reliability architecture is exposed here.

Live
1
Scheduled
3
Research
7
Parent–child pathPPillarCClusterSSupporting note
  1. 01PillarLLM Evals: Build Evidence Before MetricsBuild LLM evals from observed failures, explicit criteria, versioned cases and decisions the results can change.llm evalsLive590 US/mo · KD 45
    1. 01.01ClusterLLM Evaluation Frameworks: Choose an Eval HarnessChoose an LLM evaluation framework from dataset, evaluator, CI, trace and operating requirements.llm evaluation frameworkScheduled320 US/mo · KD 41
      1. 01.01.01SupportingDeepEval: Build a Reproducible Eval SuiteBuild a small DeepEval suite from explicit cases and metrics, then pin, run and inspect it as normal test code.deepevalScheduled2,400 US/mo · KD 36
      2. 01.01.02SupportingLangfuse Evals: Connect Traces to TestsLearn how to derive and run evaluations from Langfuse traces. Evidence artifact, failure checks, verification, and decision rule included.langfuse evalsScheduled1,300 US/mo · KD 32
    2. 01.02ClusterAgent Evals: Trajectories, Tools, and OutcomesEvaluate agents across outcomes, trajectories, tool use, state and side effects rather than final-answer accuracy alone.agent evalsResearchIntent mapped
      1. 01.02.01SupportingCoding Agent Evals: Repository Tasks That ReproduceLearn how to evaluate coding agents on reproducible repository tasks. Evidence artifact, failure checks, verification, and decision rule included.coding agent evalsResearchIntent mapped
      2. 01.02.02SupportingTool-Use Evals for LLM AgentsLearn how to measure tool selection, arguments, and recovery behavior. Evidence artifact, failure checks, verification, and decision rule included.tool use evaluation llmResearchIntent mapped
    3. 01.03ClusterLLM Evaluation Design: From Failure to Test CaseTurn production LLM failures into scoped evaluation cases, explicit criteria and leakage-resistant datasets.llm evaluation designResearchIntent mapped
      1. 01.03.01SupportingBuild an LLM Evaluation DatasetLearn how to sample, label, and version an evaluation dataset. Evidence artifact, failure checks, verification, and decision rule included.llm evaluation datasetResearchIntent mapped
      2. 01.03.02SupportingLLM as a Judge: Calibration Before ScaleLearn how to calibrate an LLM judge against human labels. Evidence artifact, failure checks, verification, and decision rule included.llm as a judgeResearchIntent mapped
      3. 01.03.03SupportingLLM Eval Error AnalysisLearn how to cluster failures before adding evaluation metrics. Evidence artifact, failure checks, verification, and decision rule included.llm eval error analysisResearchIntent mapped