How to read AI benchmarks

What the benchmark families actually measure

Every launch chart mixes benchmarks that test unrelated things. Sort them by family first.

A model that leads on GPQA and trails on Terminal-Bench is not "worse." Knowledge benchmarks are single-turn and closed-book; agentic benchmarks reward instruction adherence, tool-call formatting, and knowing when to stop. Different skills — and the second set is what production agents run on.

Pay attention to what a benchmark cannot see. SWE-bench Verified is 500 human-filtered issues from 12 Python repositories, graded by whether hidden tests pass. It says nothing about whether the patch is readable, whether it broke an untested path, or whether the model can work in Go, Terraform, or a monorepo with no test suite.

Contamination and training on the test distribution

Public benchmarks leak. Items appear in GitHub issues, blog posts, Stack Overflow answers, and scraped PDFs, and end up in pretraining data. No deliberate cheating is required; the leak is the default.

The subtler version is legal and just as distorting: training on the test distribution. A lab that generates a million synthetic competition-math problems in AIME format, or mines GitHub for issue-plus-test-patch pairs shaped like SWE-bench, never touches the test set and still posts a score that overstates transfer to your work.

Signals that a number is contamination-resistant:

• The test set is private or held out — ARC-AGI's private eval, FrontierMath's held-out tier, benchmarks refreshed from commits dated after the model's cutoff.
• The benchmark rotates items on a schedule rather than freezing them.
• Someone reports paired public/private scores. A double-digit drop from public to private split on the same distribution is memorization.
• The dataset ships a canary string and the lab claims to have filtered on it. Weak evidence, but non-zero.

A check you can run yourself: take a benchmark item, delete the final clause, and ask the model to complete it. Verbatim reconstruction of the original problem and its answer choices is close to proof of exposure.

Saturation

When the leaders on a benchmark cluster above roughly 85%, it has stopped measuring capability and started measuring tolerance for broken items. The remaining failures are disproportionately mislabeled answers, ambiguous phrasing, and grader bugs. A manual re-annotation of MMLU found errors in about 6.5% of questions overall and in 57% of the Virology subset — that puts a hard ceiling well below 100% on the honest maximum, and the ceiling differs by subject. HumanEval has been saturated for years; its residual movement is noise.

Saturation also compresses ranking. On a set where every model scores 88-93%, the entire visible spread sits inside the label-error band.

pass@1, best-of-n, and the harness

The same model on the same benchmark can move 10-20 points on settings that vendors disclose in footnotes, if at all.

• pass@1 — one sample, scored. The only number that resembles a single production request.
• pass@k — k samples, credit if any passes. Requires an oracle you do not have at inference time. Useful for research, not for capacity planning.
• cons@n / maj@n — n samples, majority vote on the final answer. Costs n times the tokens. Legitimate, but it is a system score, not a model score.
• best-of-n with a verifier — n samples ranked by a reward model or by running tests. Same caveat, higher ceiling.

Even inside "pass@1" there is wide latitude: temperature and top-p, reasoning budget, system prompt, agent turns allowed, whether the model may run tests and retry, retries on malformed tool calls, answer-extraction regex, few-shot count. On agentic benchmarks the scaffold contributes more variance than the model. A vendor comparing its own agent harness against a competitor's bare API is comparing two different systems.

If a competitor's bar carries no footnote and the vendor's own carries three, the comparison is not apples to apples.

Arena Elo is preference, not capability

Pairwise arena rankings measure which response a volunteer prefers, on prompts volunteers happen to submit. That distribution skews short, casual, and non-verifiable — closer to "write me a limerick" than "refactor this migration." Elo rewards formatting, length, and confident tone. Style-control variants regress out length and markdown and reshuffle the leaderboard, which is itself evidence that raw Elo was partly measuring layout.

Arena scores are a real signal for chat-facing consumer products and a weak one for extraction pipelines, code review bots, or anything graded programmatically. Confidence intervals there are wider than the rank order implies: models whose intervals overlap are tied, whatever their row numbers.

When is a gap real?

Benchmark scores are proportions from finite samples. The 95% interval around a single score:

A 2-point gap on any of these is inside the noise, before adding run-to-run sampling variance, which on agentic sets contributes several more points across identical repeated runs. A single AIME paper cannot separate two models at all: one problem is 6.7 points.

Paired analysis is far more sensitive, and that is the lever for your own testing. Run both models on the same items and count only the items where they disagree, and the variance collapses. A 2-point difference driven by 15 flips one way and 5 the other is measurable on a few hundred items; the same 2 points read off two independent leaderboard rows is not.

What people get wrong

• Treating a leaderboard row as a ranking. Rows with overlapping intervals are ties. Sorting by point estimate manufactures an order the data does not support.
• Comparing scores across harnesses. A SWE-bench number from a vendor's agent and one from a bare-API scaffold are different experiments with the same name.
• Assuming coding benchmarks predict coding. SWE-bench measures patching Python repos that already have tests. Not greenfield design, not unfamiliar frameworks, not debugging without a reproduction.
• Ignoring cost per solved item. Agentic runs consume orders of magnitude more tokens than a QA item, and reasoning tokens are billed whether or not the response displays them. Two models within a point of each other can differ several-fold in spend — check the cost calculator and compare models before treating them as interchangeable.
• Believing the multimodal bar. A large share of items on some multimodal sets are answerable from the text alone; text-only ablations expose this.
• Trusting an LLM judge without calibration. Judges reward length and show self-preference toward outputs from their own family. Measure judge-human agreement before you trust judge-model rankings.
• Reusing an eval set after tuning on it. Prompt iteration overfits a dev set quickly; after that the set reports your tuning, not capability.

A protocol for your own task

1. Collect 100-300 real examples from production traffic or real tickets, stratified across the input types you actually see, including the hard tail. Under ~100 items you cannot resolve anything smaller than about 10 points.
2. Split into dev and held-out halves. Iterate on dev. Touch the held-out set once per candidate decision.
3. Define grading before you look at outputs. Prefer programmatic checks — exact match, schema validation, tests pass, retrieval hit — over a rubric judge. Use a judge only where output is genuinely open-ended, and validate it against ~30 human-labeled items first.
4. Fix the harness across candidates. Same prompt, tools, retries, token budget. Then run a second round with per-model prompt tuning; the gap between the two rounds tells you how much of the difference is the model versus your prompt.
5. Run each candidate 3-5 times at your production temperature. Report mean and spread. If the spread exceeds the gap, you have no result.
6. Score paired, not marginal. Tabulate per-item wins, losses, and ties; a McNemar test on the discordant pairs is the right significance check.
7. Record cost and latency per solved item, not per call. Start candidate selection from the model index, then confirm on your own numbers.
8. Refresh the set quarterly from new traffic. Your input distribution drifts faster than the models do.

If your own evaluation disagrees with a published benchmark, your evaluation is right. It is measuring the thing you are paying for.