Evaluating LLM applications

Why vibes-based iteration fails

A pass rate from five hand-checked examples carries a 95% confidence interval of roughly ±45 percentage points. For a binary metric the half-width is about 1 / sqrt(n · reps): 25 cases at 2 reps is ±14 points, 100 at 2 reps ±7. Most prompt edits move real quality by less than 10 points. Reps on the same case are correlated, so n · reps flatters that resolution; scoring both variants on the same cases claws some back. At default sampling settings the same prompt will not reproduce its own output.

The second failure is asymmetric memory: you remember the cases you were staring at, not the adjacent behaviours you broke. Each edit is locally justified; the aggregate degrades. An eval surfaces the regression when it is introduced, not three weeks later in a support ticket.

Building an eval set from real traffic

Use the first source you actually have:

1. Production transcripts. Highest fidelity. Settle retention and PII constraints first; if the rows cannot live in a repo, store record identifiers and fetch inputs at eval time, or rewrite each case synthetically, preserving shape and difficulty.
2. Bug reports and escalations. Highest value per unit of effort.
3. Hand-written by the domain owner. Skewed toward the salient, not the frequent — a seed, not a set.
4. Synthesized from the system prompt and tools. Lowest fidelity, and never cold: anchor on three to five real examples plus a sentence on what makes a case hard here.

Fifteen to a hundred cases is right for a first eval: below fifteen one flaky case swings the headline; past a hundred nobody reads them, and the reading is the point.

Two properties matter more than volume. Both-directions coverage: an eval for "does the agent search when it should" also needs cases where it should not, or an always-search policy scores perfectly; same for refusals, escalation, tool use. Provenance: record whether each expected output was human-written, human-verified, or model-generated, and by which model. Gold from the incumbent makes reference-match scoring reward imitating it — backwards in a migration; never use a compared model's outputs as ground truth.

The grading spectrum

Pick the cheapest method that genuinely measures the property, without letting cost push a programmatic check onto something that needs judgment.

For agents that act, grade the end state, not the transcript: run each case in a throwaway checkout and check what it left behind — hidden tests pass, the diff touches only intended files, the typechecker is clean — plus a no-op detector for "claimed success, workspace unchanged." A judge reading a coding transcript grades the narration; reserve it for diff minimality and the PR text.

Score independent properties as separate metrics ({correct, formatted, grounded}) rather than one blend: atomic checks are more reproducible and diagnostic. With positives and negatives both present, report precision, recall, and specificity — an accuracy "win" is usually recall traded for precision.

LLM-as-judge: the four biases that matter

• Position bias. Judges favour one slot in pairwise comparison. Randomize which candidate is A, or score both orders and average.
• Length bias. Judges prefer longer answers absent instruction otherwise. Say that length is not a merit, and log output length to see whether a "win" is just more words.
• Self-preference. A judge prefers outputs resembling its own. Never use the model under test as its own judge; on close calls prefer a different family, or a jury.
• Label deference. Never tell the judge which output is the "reference", "baseline", or "human" one.

Treat candidate text as untrusted data — under optimization pressure a model will emit text that reads as a directive to the grader. Constrain the verdict with a JSON schema; "respond with only JSON" fails on unescaped quotes often enough to corrupt a run.

Before a judge steers any decision, calibrate it against human labels on a few dozen independently labelled cases and report agreement; well below ~90% on clear-cut cases means the rubric needs another pass. Then test it on known negatives — an empty string, "I don't know", a confident answer to a different question. It must fail all three.

Train, validation, test

The moment you iterate against the eval you are fitting to it: whoever proposes changes reads failing transcripts and fixes the cases they saw. Train is what the proposer may read; validation picks between rounds; test is scored every round and reported, and its transcripts stay closed.

Draw the split at random, stratified by category, never by score. Selecting worst-scoring cases into train guarantees regression to the mean — they improve on re-run by chance alone — and aims fixes at the tail. The signature of an overfit round is a healthy train gain with a flat held-out number. Confirm at baseline that the splits agree within noise, then freeze it and record the case IDs.

A proposed fix should describe the failure behaviour, not its content: pasting nouns from train cases into the prompt helps train and does nothing in production. Keep ground truth structurally unreachable: not in a mounted fixture, not in a file the agent's bash tool can cat, not on the open web if it has search. "Don't look at the answers" is not a defence.

Cost and latency are eval metrics

Read token counts from the response's usage block per row: input, output, cache-read, cache-write. Character heuristics and another vendor's tokenizer are off by enough to reverse a cost comparison, worse on code and non-Latin text; for pre-flight numbers use the provider's token-counting endpoint. Derive cost from recorded tokens and the row's actual served model, including cache rates, never a flat assumed rate. Use the cost calculator and compare models for current rates; the durable relationships are that output tokens cost several times input, cache reads roughly an order of magnitude less than uncached input, and batch processing typically half the price of synchronous calls — the right mode for offline runs.

Record the judge's cost separately, or grading spend hides differences between variants. Time only the final successful request; retries, backoff sleeps, and semaphore queueing go in a separate wall-clock field, or whichever variant hit more transient errors looks slower. For agentic flows, break tokens and timing down per model call and per tool call, or a slow tool is indistinguishable from a slow model. Report absolutes before deltas ("19.8 s/turn, $0.031/call, 480 output tokens"), and judge cost per completed task: a cheaper request needing three more turns is not cheaper.

Regression testing on model upgrades

Pin an exact model identifier, then read the model field back from each response and assert it matches, allowing for documented alias-to-snapshot resolution; a fallback or capacity reroute produces a score that measures nothing and may surface nowhere else. Check the live model index for current identifiers and context limits before pinning.

Run baseline and candidate against the same cases, reps, and scaffold, in one session where possible — cache warmth differing between variants is a cost and latency gap that is an artifact of run order. Record an environment fingerprint: repo commit, lockfile hash, disabled tools, model id. If the baseline came from a different day, score a no-change control alongside the candidate and gate against that, not the stale number.

Expect prompt-level work, not a string swap. Prompts accumulate instructions written for older models — verbose formatting rules, step-by-step recipes, "think carefully before answering" preambles — that a newer model does not need and may be harmed by. Reasoning and effort settings differ across generations too, and a parameter valid on the previous model is often rejected outright rather than ignored: exercise every configuration flag your app sets.

Production monitoring

An offline eval measures the distribution you captured; production is the distribution you get. Persist per request the message list, every tool call and result, the usage block, served model id, stop reason, latency, and a trace ID into your application logs: a surprising result should be explainable without re-running it. Alert on what an offline eval cannot see:

• Truncation rate — responses ending at the output cap rather than naturally. A clipped answer is not a wrong answer; never average it in as one.
• Refusal rate, as its own metric, so refusal-zeros and capability-zeros are not summed.
• Tool error rate and retry counts, so a broken downstream service does not read as model degradation.
• p50/p95 latency per completed task, not per call.
• Input distribution shift — prompt lengths and category mix drifting from what the eval covers.

Every production failure becomes a case, saturated items get hardened, the judge is re-calibrated when it drifts. An eval that has not gained a case in six months measures last quarter's product.

What people get wrong

• Reimplementing the app inside the runner. It must call the real entry point, or you measure a system nobody ships.
• Scoring infra failures as model failures. Timeouts, rate-limit errors after retries, unparseable output, and crashed graders belong in an errors sidecar with a failure class — not the results file, where they take a case slot and score the plumbing.
• Conflating "no answer" with "negative answer." Where "none" is valid, a runner that errors on every input scores identically to one that carefully found nothing.
• Trusting a zero. A cost of $0.00, a latency of 0.0, or a metric constant across all cases is a field-name mismatch or a swallowed exception, not a measurement.
• Never running the oracle and the null. Push the reference answers through the pipeline (should score near 100%) and an empty output (near 0%). Two runs, minutes, and most wiring bugs surface before a paid pass.
• Hill-climbing a saturated eval. Above roughly 95%, remaining variance is format quirks and grader tie-breaking, not capability; you improve something real only by adding harder cases.
• Reporting a mean when the question is worst-case. For rare high-stakes behaviours — data deletion, irreversible actions, PII leakage — fail-on-any is the honest aggregation; a mean diluted by easy cases hides the one result that matters.