Users type a prompt, get two anonymous model responses, vote for one, and the votes are fit into a Bradley-Terry (Elo-style) rating with confidence intervals. Originally LMArena out of UC Berkeley's LMSYS, it now runs as the company Arena Intelligence and splits into 13 boards: Agent, Text, WebDev, Image-to-WebDev, Vision, Document, Search, Text-to-Image, Image Edit, Text-to-Video, Image-to-Video and Video Edit. Unlike every other entry here it has no ground truth — it measures which answer a self-selected voter preferred, which is a real signal about deployment appeal and a poor one about correctness. Style Control is an offered adjustment that regresses out response length and markdown formatting; the default board and the style-controlled board disagree, and which one gets screenshotted is a rhetorical choice.
| Measures | Human pairwise preference, aggregated to a Bradley-Terry rating, across text, code, vision, document, search, image and video |
|---|
| Format | Live blind A/B voting; ratings with CIs, optional Style Control adjustment; 13 category leaderboards |
|---|
| Saturation | Never saturates by construction (relative ranking), but top-10 confidence intervals overlap heavily, making rank order largely meaningless |
|---|
| Contamination risk | Low in the training-leak sense; high in the gaming sense — prompts are public and models can be tuned for voter-pleasing style |
|---|
| Maintainer | Arena Intelligence (commercial spinout of LMSYS / UC Berkeley) |
|---|
Watch out: No ground truth: a confidently wrong, nicely formatted answer beats a correct terse one, and votes come from an unrepresentative population of AI enthusiasts. The 2025 'Leaderboard Illusion' paper alleged that large labs test many private variants and retire the losers, and that sampling rates favour incumbents — the operator disputed parts of this and the underlying practice of private pre-testing persists. Ratings for new models are unstable until vote volume accumulates, and the rebrand from lmarena.ai to arena.ai as a commercial entity introduces a conflict of interest that a university project did not have. Wrong choice when you need to know whether output is factually correct or whether code runs.
not applicable
An agent gets a real repository at a specific commit plus an issue description, edits the code, and the patch is scored by running the maintainers' actual test suite. Verified is the 500-instance subset that OpenAI paid annotators to filter out under-specified issues and broken tests from the original 2,294-task set, and it has become the default coding-agent number in launch posts. The site now hosts six splits — Full, Verified, Lite, Multimodal, Multilingual and Bash Only — which matter because a 'SWE-bench' claim with no split named is unreadable. Bash Only in particular constrains the agent to shell commands and typically scores well below Verified for the same model.
| Measures | Repo-level bug fixing: does the generated patch make the hidden maintainer tests pass |
|---|
| Format | 500 human-validated instances from 12 Python repos; agentic, execution-graded, % resolved. Sibling splits: Full, Lite, Multimodal, Multilingual, Bash Only |
|---|
| Saturation | Heavily saturated — frontier agents report 70-80%+, and the residual is dominated by ambiguous or environment-flaky instances |
|---|
| Contamination risk | Very high — public repos, public issues, public fixing commits, all pre-cutoff for every current model |
|---|
| Maintainer | SWE-bench team (Princeton/Stanford); Verified subset curated with OpenAI |
|---|
Watch out: Twelve Python repositories (django, sympy, sphinx, astropy and friends) whose entire history is in every model's training data — the issues, the discussion threads and often the exact fixing commit. Scores are dominated by the scaffold: the same model varies by double digits between a bare loop and a tuned agent, so cross-lab comparisons on the public leaderboard compare harnesses, not models. Reported figures are frequently self-submitted and unaudited, and Epoch AI's independent runs land below lab claims. The tasks are localized single-issue fixes with tests already written — nothing about design, migrations, or multi-service work. Wrong choice for anything non-Python or for greenfield code.
not applicable
· open source
Scale's answer to SWE-bench Verified's saturation and contamination: 1,865 tasks across 41 repositories, split into 731 public instances from copyleft (GPL) open-source repos, 276 private instances taken from proprietary startup codebases, and 858 held out entirely. The GPL choice is deliberate — those repos are less likely to have been scraped into training sets — and the commercial split is genuinely unseen code. Tasks require multi-file changes rather than the single-hunk fixes that dominate Verified. The difficulty gap is the point: when top models sat around 23% on the public set they were clearing 70%+ on Verified, which is the clearest single demonstration that Verified's numbers are inflated by familiarity.
| Measures | Multi-file software engineering tasks on less-contaminated and proprietary codebases |
|---|
| Format | 1,865 tasks / 41 repos: 731 public (GPL), 276 private commercial, 858 held out; agentic, test-verified % resolved |
|---|
| Saturation | Low to moderate — early frontier scores ~23%, since climbing past 60% on the public set |
|---|
| Contamination risk | Low for private and held-out splits; rising for the public GPL split now that it is published |
|---|
| Maintainer | Scale Labs (Scale AI) |
|---|
Watch out: The private commercial split is scored by Scale on models Scale chooses to run, which is a single-vendor bottleneck and not independently reproducible — you are trusting a company that also sells evaluation services. The public GPL split will contaminate over time exactly like Verified did. Leaderboard coverage is thinner than SWE-bench's, so the model you care about may simply be absent, and results appear in batches rather than at launch. Still Python/JS-centric despite the wider repo count.
not applicable
· open source
GPQA is 448 multiple-choice questions in biology, physics and chemistry written by PhD holders and filtered so that skilled non-experts with 30+ minutes of unrestricted web search still fail most of them — hence 'Google-proof'. Diamond is the 198-question subset where two experts agreed and the majority of non-experts failed, and it is the subset everybody quotes. It was built for scalable-oversight research, not for model ranking, and its adoption as a headline capability metric is an accident of it being the hardest thing available in 2024. The dataset is gated on Hugging Face with a canary string precisely to slow contamination, and that has plainly failed.
| Measures | Graduate-level knowledge and reasoning in biology, physics, chemistry, resistant to quick web lookup |
|---|
| Format | 198 four-option MCQs in the Diamond subset (448 total in GPQA); single-turn, exact-match, no tools |
|---|
| Saturation | Saturated — frontier models exceed the 65% PhD-expert baseline; remaining headroom is label noise |
|---|
| Contamination risk | Very high despite gating and a canary string; widely mirrored and used in training mixes |
|---|
| Maintainer | David Rein and co-authors; effectively unmaintained as a leaderboard, re-run by Epoch AI and Artificial Analysis |
|---|
Watch out: At 198 questions, one item is 0.5 percentage points, so the 1-3 point gaps labs advertise are inside the noise band before you even account for sampling temperature. Frontier models are at or above the 65% human-expert bar, so the benchmark no longer ranks the models anyone is choosing between. Contamination is effectively certain: the dataset has six-figure monthly downloads and Hugging Face lists models trained on it, and the canary string is unenforceable. Multiple-choice format rewards elimination strategies that have nothing to do with the underlying science. Wrong choice for any decision about frontier models.
not applicable
· open source
AIME is the American Invitational Mathematics Examination — 15 problems, integer answers 0-999, three hours, run twice a year by the MAA — and HMMT is the Harvard-MIT Math Tournament. Neither was built as an AI benchmark; labs adopted them because they are hard, unambiguous to grade, and freshly generated each year. MathArena is the credible way to consume them: the ETH Zurich SRI Lab and INSAIT run models on each paper within days of the human contest, before solutions circulate widely, and publish standardised, multi-sample results across AIME 2025/2026, HMMT Feb and Nov, BRUMO, SMT, CMIMC, USAMO and IMO. That timing discipline is the whole value — an AIME 2024 score quoted in 2026 tells you nothing.
| Measures | Competition-level symbolic mathematics under short horizons; answer correctness, not proof quality |
|---|
| Format | 15 integer-answer problems per AIME paper (0-999, 3 hours); MathArena runs multiple samples per model across AIME, HMMT, BRUMO, SMT, CMIMC, USAMO, IMO |
|---|
| Saturation | AIME largely saturated at the frontier; HMMT Feb and olympiad proof papers still discriminate |
|---|
| Contamination risk | Near-zero for a paper evaluated within days of release; near-total for any prior-year paper |
|---|
| Maintainer | SRI Lab (ETH Zurich) and INSAIT for MathArena; MAA and HMMT for the underlying contests |
|---|
Watch out: Fifteen problems per paper means each one is 6.7 percentage points; MathArena mitigates with repeated sampling but the underlying instrument is tiny and variance is large. Any AIME paper more than a few months old is thoroughly contaminated — the problems, solutions and dozens of forum threads are indexed — so labs quoting AIME 2024/2025 in 2026 are quoting a memorisation test. Scores are saturating fast on AIME while remaining lower on olympiad proof-based papers, so 'math ability' collapses to which paper you picked. Answer-only grading gives full credit for a lucky integer with a broken derivation. Wrong choice as a proxy for research mathematics — use FrontierMath for that.
not applicable
· open source
Built to fix three concrete problems with MMLU: guessing was worth 25%, many gold labels were wrong, and scores swung 4-5 points on prompt phrasing alone. MMLU-Pro raises the option count from 4 to 10 (random guessing drops to 10%), strips trivial items, adds reasoning-heavy questions, and reports prompt sensitivity down to about 2%. It covers 14 categories — math, physics, chemistry, law, engineering, economics, health, psychology, business, biology, philosophy, computer science, history and other — with 12,000 test items plus a 70-item validation split. Chain-of-thought is effectively mandatory; direct-answer evaluation drops models by roughly 19 points, so any quoted score is meaningless without the prompting mode.
| Measures | Broad multi-domain knowledge plus reasoning across 14 academic and professional categories |
|---|
| Format | 12,032 questions (12,000 test / 70 val), 10 answer options each, CoT prompting, exact-match accuracy |
|---|
| Saturation | Approaching saturation at the frontier; still discriminates among mid-size and open-weight models |
|---|
| Contamination risk | High — fully public dataset, heavily downloaded, widely present in evaluation and training mixes |
|---|
| Maintainer | TIGER-Lab, University of Waterloo (last dataset update March 2026) |
|---|
Watch out: Still multiple choice, so it measures recognition rather than generation and rewards option-elimination heuristics that have nothing to do with knowing the answer. Public and fully downloadable since 2024, with millions of downloads — treat contamination as a given. Frontier proprietary models cluster in a narrow band at the top, so it no longer separates the models most teams are actually choosing between. The dataset has been silently corrected multiple times (including 15 medical answers revised on specialist advice), which means scores from different dates are not strictly comparable. Wrong choice for anything agentic, long-context, or generative.
not applicable
· open source
Nearly 1,000 subject experts from 500+ institutions in 50 countries submitted questions specifically designed to be unanswerable by then-current models; the public set was frozen at 2,500 items on 3 April 2025 and the work was published in Nature in January 2026. Crucially, CAIS and Scale keep a private held-out set of additional questions and compare public against private performance to detect overfitting — one of the few frontier benchmarks with that safeguard built in. It mixes short-answer and multiple-choice items across mathematics, humanities, natural sciences and more, and includes a multimodal fraction. An HLE-Rolling fork launched in October 2025 to accept continuing contributions as the static set ages.
| Measures | Frontier academic knowledge and reasoning at the edge of human expertise, 100+ subjects, partly multimodal |
|---|
| Format | 2,500 public questions (short-answer and MCQ) frozen April 2025, plus a private held-out set; accuracy and calibration error |
|---|
| Saturation | Not saturated — Gemini 3 Pro leads around 38.3%, with GPT-5 at 25.3% and Grok 4 at 24.5%; but climbing fast |
|---|
| Contamination risk | Moderate — public set is downloadable, mitigated by the private held-out set used to measure overfitting |
|---|
| Maintainer | Center for AI Safety (CAIS) and Scale AI; published in Nature 649:1139-1146 (Jan 2026) |
|---|
Watch out: Low absolute scores make it look discriminating, but the questions are deliberately adversarial trivia at the tail of human expertise — a model can score 38% on HLE and still be useless at your job, because nothing here resembles work. The published error rate in the gold answers is non-trivial, so the effective ceiling is below 100% and nobody knows exactly where. Scores are extremely sensitive to tool access and search: a browsing-enabled run is not comparable to a closed-book run, and headline numbers rarely say which was used. It is expensive to run at high reasoning effort. Wrong choice for evaluating agents, code, or long-horizon tasks.
not applicable
· open source
Visual grid-transformation puzzles where the model infers a rule from a handful of input-output examples and applies it to a new grid. ARC-AGI-2 raises the bar over v1 with symbolic interpretation, compositional rules that interact, and rules whose application depends on context rather than surface pattern. Its calibration is the distinguishing feature: every task in every split has been solved by at least two humans in under two attempts, so a low model score is unambiguously a model failure rather than an unfair item. Three parallel 120-task sets exist — public eval, semi-private (used for verified leaderboard runs) and private (final scoring) — and the leaderboard plots score against cost per task, refusing to display runs over $10,000 total. ARC-AGI-3 has since moved to interactive agent environments, with a 2026 Kaggle competition running under strict compute limits.
| Measures | Fluid rule induction from few examples: symbolic interpretation, compositional and context-dependent rules |
|---|
| Format | 120 tasks each in public eval, semi-private and private sets; grid-to-grid, exact match, 2 attempts; scored against cost per task with a $10k display cap |
|---|
| Saturation | Far from saturated — the stated 85% target remains open and log-linear compute scaling is documented as insufficient |
|---|
| Contamination risk | Low — semi-private and private sets are never published; public eval set is contaminated and used only for development |
|---|
| Maintainer | ARC Prize Foundation; ARC Prize 2026 competition on Kaggle |
|---|
Watch out: The task format — coloured grids — is narrow and artificial, and it is genuinely unclear how much grid-puzzle skill transfers to anything a practitioner ships. Scores are dominated by test-time compute: the same model swings enormously with sampling budget, which is why the cost axis exists and why any bare percentage is unreadable. Verified leaderboard placement requires running against the semi-private set on ARC Prize's terms, so the public numbers you see are often marked 'preview' and unofficial. The 85% grand-prize threshold is far away, so for most model-selection decisions the benchmark returns 'everything is bad' and does not discriminate. Wrong choice for anything language-, code- or knowledge-related.
not applicable
· open source
Agents get a containerised shell and a task — compile something, debug a broken build, recover data, configure a service — and are graded on the resulting system state rather than on their transcript. It runs inside Harbor, a separate agent-evaluation and optimisation framework, which means the same task set can be run across many agent scaffolds and the leaderboard reports model, agent, resolution rate, cost and token count side by side. That cost column is the reason to prefer it to SWE-bench for agent selection. Versioning is aggressive: 2.0 shipped November 2025, 2.1 fixed 28 tasks in May 2026, 3.0 pushed difficulty to the frontier in July 2026, and 4.0 landed August 2026 recalibrating task resources and removing saturated tasks.
| Measures | Agentic command-line competence: real terminal tasks graded on final system state |
|---|
| Format | Containerised shell tasks run under the Harbor harness; resolution rate reported alongside cost and token usage |
|---|
| Saturation | Deliberately reset each major version — 4.0 removed saturated tasks; 2.x is largely saturated at the frontier |
|---|
| Contamination risk | Moderate — tasks are public with a do-not-train canary string; mitigated in practice by frequent task rotation |
|---|
| Maintainer | Hosted by Stanford / Harbor / Laude Institute (4.0 released 28 Aug 2026) |
|---|
Watch out: The version churn is a real hazard — 2.0, 2.1, 3.0 and 4.0 all shipped within ten months and scores are not comparable across them, yet vendors and even Artificial Analysis's index quote different versions concurrently. Tasks are synthetic scenarios authored for the benchmark, not sampled from real operations, so they reward a particular authored style of problem. Containerised Linux only: nothing about Windows, GUIs, or long-lived stateful production systems. Removing 'saturated tasks' in 4.0 keeps the benchmark hard but breaks longitudinal tracking by construction. Results depend as much on the Harbor agent configuration as on the model.
not applicable
· open source
The design idea is temporal: problems are scraped continuously from three competitive-programming sites and tagged with a publication date, so you can evaluate any model only on problems that appeared after its training cutoff. v5 (January 2025) held 880 problems spanning May 2023 to January 2025; v6 (April 2025) expanded to 1,055 through April 2025. Beyond plain code generation it scores three other scenarios — self-repair, test output prediction and code execution — which separate 'can write a solution' from 'can predict what code does'. Grading is pass@1 / pass@5 against the contest test cases, so it is fully execution-verified with no judge model.
| Measures | Competitive-programming code generation, plus self-repair, test output prediction and code execution |
|---|
| Format | 1,055 problems in v6 (May 2023-Apr 2025) from LeetCode, AtCoder and Codeforces; pass@1 / pass@5 against contest tests, filterable by release date |
|---|
| Saturation | Easy and medium tiers saturated at the frontier; hard tier still separates models |
|---|
| Contamination risk | Low if you restrict to post-cutoff problems, which is the benchmark's core mechanism; high if you use the full set |
|---|
| Maintainer | UC Berkeley LiveCodeBench team |
|---|
Watch out: Competitive programming is not software engineering — a model that tops this can still be poor at reading an unfamiliar codebase, and vendors sometimes cite it as a proxy for engineering ability, which it is not. The date-filter defence only works if you know the true cutoff and if the problem was not discussed on forums before the stated date; both assumptions leak. Windows are short (v6 ended April 2025), so for a 2026 model the uncontaminated slice is small and the sub-sample noise is correspondingly large. Different published scores use different date windows and different versions, making cross-report comparison unreliable unless the window is stated. Python-dominant.
not applicable
· open source
Distinct from other coding benchmarks because it grades two things at once: whether the solution is right, and whether the model produced a syntactically valid edit that the tool could apply without human intervention. It runs 225 of the hardest Exercism exercises across C++, Go, Java, JavaScript, Python and Rust, reporting pass rate, edit-format compliance, token usage and dollar cost per run. That edit-format column is the unique signal — a model that solves the problem but mangles its search/replace blocks is useless inside a coding tool, and no other public benchmark isolates that. Note the leaderboard's last update was 20 November 2025.
| Measures | Code editing under instruction: solution correctness plus valid, applyable edit-format compliance |
|---|
| Format | 225 Exercism exercises in C++, Go, Java, JavaScript, Python, Rust; % correct, % correct edit format, tokens and cost |
|---|
| Saturation | Approaching saturation — top reported result was GPT-5 (high) at 88.0% before updates stopped |
|---|
| Contamination risk | High — Exercism exercises and community solutions are public and long-indexed |
|---|
| Maintainer | Paul Gauthier / aider project; leaderboard last updated 20 Nov 2025 |
|---|
Watch out: The leaderboard has not been refreshed since November 2025, so it is roughly ten months stale as of September 2026 and omits every model released since — treat it as a historical reference, not a current shortlist. Exercism exercises are self-contained puzzles with existing tests, nothing like navigating a real repository. Results are partly a measure of how well a model matches aider's specific diff format, which does not generalise to tools that use other edit protocols. Small sample (225 items) and single-maintainer governance mean it can go quiet without notice. Wrong choice for agentic or multi-file work.
not applicable
· open source
An agent handles a simulated customer (played by an LLM) in a domain with real tools and a real database, and is scored on whether the database ends in the correct state and the right tool calls were made — not on whether the transcript reads nicely. Its signature metric is pass^k: the probability that the agent succeeds on all k independent attempts at the same task, which punishes flakiness in a way pass@k explicitly rewards. The line has evolved through three generations: τ-bench (June 2024, retail and airline), τ²-bench (June 2025, adds telecom and a dual-control setting where the user also acts), and τ³-bench (March 2026), which adds a banking knowledge domain with ~700 documents and RAG retrieval, plus full-duplex real-time voice evaluation with interruptions, accents and background noise.
| Measures | Tool-using conversational agents under domain policy: database end state, correct tool calls, reliability across repeats |
|---|
| Format | Simulated user + tool APIs in retail, airline, telecom and banking-knowledge domains; pass^k reliability metric; text and full-duplex voice tracks |
|---|
| Saturation | Mixed by design — τ²-bench text is near saturation (~88%), while τ³-Banking sits near 55% and remains open |
|---|
| Contamination risk | Moderate — domains and policies are public on GitHub, though outcome-based grading limits the value of memorising transcripts |
|---|
| Maintainer | Sierra (sierra.ai); τ³-bench released March 2026 |
|---|
Watch out: The simulated user is itself an LLM, so a chunk of the variance is the simulator's behaviour rather than the agent's — change the user model and the scores move. Domains are small and hand-authored (retail, airline, telecom, banking), and the policies are Sierra's inventions, so a high score means 'follows these fictional policies', not 'follows yours'. Sierra sells customer-service agents, which is a direct commercial interest in the benchmark that defines the category. Task-level bugs have been material: the v1.0.x releases shipped 75+ task fixes for wrong actions, ambiguous instructions and impossible constraints, so older published scores are not comparable. Voice evaluation adds provider-specific latency effects that confound model quality.
not applicable
· open source
The reference benchmark for whether a model emits well-formed, correct tool calls. It grades two ways: abstract-syntax-tree comparison against a reference call (checking function name, parameter names and value types without needing to execute), and real execution against live APIs. V4 extends beyond single calls into holistic agentic evaluation, adding web-search tasks, memory, multi-turn interaction, enterprise and community-contributed function sets, and a format-sensitivity category that measures how much a prompt-based model's tool-calling degrades when the schema is presented differently. Everything is reproducible: results are pinned to a commit, and a versioned pip package (bfcl-eval) lets you re-run the exact harness yourself.
| Measures | Function/tool calling: syntax and semantic correctness by AST match plus real execution, multi-turn, agentic, memory, web search, format sensitivity |
|---|
| Format | V4 category suite over enterprise and community-contributed functions; accuracy per category with an overall aggregate; reproducible via bfcl-eval pip package |
|---|
| Saturation | Simple single-call categories are saturated; multi-turn, agentic and memory categories remain well below ceiling |
|---|
| Contamination risk | High for the static AST categories (public dataset, widely used in tool-calling fine-tuning); lower for live-execution and agentic categories |
|---|
| Maintainer | UC Berkeley Gorilla team (Shishir Patil et al.); leaderboard updated April 2026 |
|---|
Watch out: AST matching accepts only calls that resemble the reference, so a legitimately different-but-correct call can be scored wrong, and it says nothing about whether calling that function was a good idea. Live-execution categories depend on third-party APIs that change and break, making historical scores non-reproducible in practice despite the pinned commits. The functions are toy or synthetic relative to a real enterprise tool surface with dozens of overlapping endpoints. Native tool-calling models and prompt-based models are evaluated under different conditions, so cross-category comparison is misleading. Leaderboard refreshes are periodic rather than at-launch, so brand-new models may be missing.
not applicable
· open source
The default multimodal knowledge benchmark: 11,500 college-exam questions across 30 subjects in 6 disciplines and 183 subfields, paired with 30 heterogeneous image types — diagrams, charts, chemical structures, medical images, music scores, circuits, maps. MMMU-Pro (September 2024) is the hardened variant that filters out questions answerable from text alone, expands the candidate options, and adds a vision-only setting where the question itself is embedded in a screenshot, forcing the model to actually read the image rather than shortcut through the caption. That text-only-solvable filtering is important, because a meaningful fraction of original MMMU questions could be answered with no image at all.
| Measures | College-level multimodal understanding and reasoning over diagrams, charts, technical and scientific imagery |
|---|
| Format | 11.5K questions, 30 subjects / 6 disciplines / 183 subfields, 30 image types; MCQ and open-ended accuracy; MMMU-Pro adds vision-only and expanded-option settings |
|---|
| Saturation | Original MMMU val approaching saturation at the frontier; MMMU-Pro vision-only still leaves substantial headroom |
|---|
| Contamination risk | Very high since test-set answers were published in February 2026; also sourced from public textbooks and papers |
|---|
| Maintainer | MMMU team, Ohio State University and University of Waterloo (Xiang Yue et al.) |
|---|
Watch out: The test-set answers were released publicly on 12 February 2026 to allow local evaluation, which destroys the held-out property — any test-split score published after that date should be treated as self-reported and contamination-exposed. A slice of original MMMU is solvable without looking at the image; use MMMU-Pro if you care about that, and note the two are frequently conflated in launch charts. It is an exam benchmark, so it measures academic recall with pictures, not document processing, OCR quality, or visual grounding for agents. Image quality is uneven since figures were sourced from textbooks and papers. Wrong choice for evaluating screenshot-driven computer-use agents.
not applicable
· open source
The most-cited general video understanding benchmark. It splits 900 videos into short (under 2 minutes), medium (4-15 minutes) and long (30-60 minutes) buckets so you can see exactly where a model's temporal reasoning falls apart — the long bucket is where most video models collapse and where the useful signal lives. All 2,700 QA pairs are human-annotated rather than generated. It supports separate evaluation with and without subtitles (744 subtitle tracks provided) and with audio, which is diagnostically important: a large with-subtitles gain means the model is reading text, not watching video.
| Measures | Video understanding across short, medium and long durations, with optional subtitle and audio channels |
|---|
| Format | 900 videos / 254 hours / 2,700 human-annotated MCQ pairs; short (<2min), medium (4-15min), long (30-60min); accuracy with and without subtitles |
|---|
| Saturation | Short subset largely saturated (frontier models ~85%); long subset still well below human performance |
|---|
| Contamination risk | High — sourced from public YouTube video that is likely in pretraining corpora |
|---|
| Maintainer | MME-Benchmarks team (github.com/BradyFU/Video-MME); dataset largely unchanged since 2024 |
|---|
Watch out: The academic-only licence is a genuine blocker for commercial evaluation — you cannot legally use it to qualify a vendor for a product, unlike almost everything else in this chapter. Multiple choice, so it rewards elimination and does not test description, grounding or temporal localisation. Frame-sampling strategy dominates results: the same model scores very differently at 8 vs 64 vs 256 frames, and published numbers rarely state the frame budget, which makes cross-report comparison close to meaningless. YouTube-sourced videos are inherently contaminated for any model trained on public web video. The benchmark has not been substantially revised since 2024 while video models have advanced considerably.
not applicable
· open source
The most-used meta-tracker. It maintains internal copies of every evaluation dataset and runs them on its own infrastructure, so its numbers are not vendor self-reports — the deltas against launch-post claims are frequently the most useful thing on the page. Intelligence Index v4.2 aggregates ten evaluations in four weighted groups: Agents at 30% (AA-Briefcase, GDPval-AA v2, τ³-Banking), General at 30% (AA-Omniscience, GDP.pdf, AA-LCR v1.1), Coding at 20% (Terminal-Bench v2.1, SciCode) and Scientific Reasoning at 20% (Humanity's Last Exam, CritPt). Alongside intelligence it publishes measured throughput, time-to-first-token, end-to-end latency under load and per-token pricing across serving providers, and it extends to speech, image, video, music and coding agents. It also sells commercial products (Optima, MicroEvals).
| Measures | Composite intelligence (10 evals, weighted), plus measured output speed, TTFT, latency and price across providers and modalities |
|---|
| Format | Intelligence Index v4.2: Agents 30% (AA-Briefcase, GDPval-AA v2, τ³-Banking), General 30% (AA-Omniscience, GDP.pdf, AA-LCR v1.1), Coding 20% (Terminal-Bench v2.1, SciCode), Scientific Reasoning 20% (HLE, CritPt) |
|---|
| Saturation | Managed by rebasing — the index is re-versioned as components saturate; top models cluster around 54-57 on v4.2 |
|---|
| Contamination risk | Inherits the contamination of its public components (HLE, Terminal-Bench, SciCode); lower for its proprietary in-house evals |
|---|
| Maintainer | Artificial Analysis (independent commercial benchmarking company) |
|---|
Watch out: A single weighted composite hides everything that matters: two models can tie at 54 with completely different profiles, and the 30/30/20/20 weighting is Artificial Analysis's editorial judgement, not a fact about intelligence. The index rebases with each version, so v4.2 scores are not comparable to earlier versions and historical charts quietly break. Component versions lag — v4.2 uses Terminal-Bench v2.1 while the benchmark itself is on 4.0 — so a component can be saturated or superseded while still carrying 20% weight. Half the components are proprietary AA-built evals (AA-Briefcase, AA-Omniscience, AA-LCR) that nobody can independently reproduce or audit. Published cost figures exclude judge calls and preprocessing, so the real cost of a run is higher than shown. It is a commercial company selling evaluation products to the same industry it ranks.
not applicable
Epoch AI runs an independent benchmarking hub covering FrontierMath (Tiers 1-3 v2, Tier 4 v2, and the Lean-formalised Erdős set), SWE-bench Verified, GPQA Diamond, MATH Level 5, OTIS Mock AIME 2024-2025, SimpleQA Verified, MirrorCode, chess puzzles, Earthborne Rangers (EBR-bench) and mystery-game puzzles. The value is methodological transparency rather than novelty: they publish harness details, per-model run configurations and the raw result data alongside the leaderboards, which makes their numbers auditable in a way vendor blog posts are not. They also maintain the wider Epoch datasets on training compute, model releases and hardware trends, so a benchmark result sits next to the compute that produced it. The FrontierMath Open Problems expansion adds 50 genuinely unsolved research questions, of which AI has solved three so far.
| Measures | Independently re-run frontier evals: FrontierMath, SWE-bench Verified, GPQA Diamond, MATH Level 5, OTIS Mock AIME, SimpleQA Verified, MirrorCode and game-based reasoning |
|---|
| Format | Interactive dashboards with downloadable underlying data and documented harness configuration; multiple benchmarks per model |
|---|
| Saturation | Varies by benchmark — GPQA Diamond and MATH L5 saturated, FrontierMath Tier 4 and Open Problems far from it |
|---|
| Contamination risk | Low for FrontierMath (unpublished problems, held-out set) and Open Problems; inherited-high for the public benchmarks they re-run |
|---|
| Maintainer | Epoch AI (independent research organisation) |
|---|
Watch out: Coverage is deliberately narrow and updates lag launches by days to weeks, so it is a verification layer rather than a first look. Because they use their own harness and prompting, their figures are systematically lower than vendor claims — correct, but it means you cannot mix an Epoch number and a lab number in the same chart. FrontierMath's OpenAI funding, and OpenAI's access to problem data, drew sustained criticism in 2025 over whether an independent benchmark can be commissioned by an evaluated party; disclosure improved but the structural conflict is real. Several of their benchmarks (chess puzzles, board-game and mystery-game evals) are research instruments with little bearing on production model selection. Small team, so continuity depends on funding.
not applicable
· open source
Commissioned from professional mathematicians and never published, which makes it one of the few genuinely uncontaminated hard-reasoning benchmarks available. Problems span Tiers 1-3 (undergraduate through graduate difficulty) and Tier 4 (research level), and every problem has a computationally verifiable answer so grading needs no human and no LLM judge. Two extensions matter: Open Problems is 50 significant unsolved research questions where AI has so far cracked three, and FrontierMath Erdős formalises problems studied by Paul Erdős in the Lean proof assistant, requiring a complete machine-checked proof or disproof rather than an answer. That Lean track is the strongest available evidence about whether a model can do mathematics rather than pattern-match to it.
| Measures | Research-level mathematical reasoning with computationally verifiable answers; Erdős track requires formal Lean proofs |
|---|
| Format | Several hundred unpublished problems in Tiers 1-3 and Tier 4 (v2), plus 50 Open Problems and the Lean-formalised Erdős set; accuracy, held-out subset |
|---|
| Saturation | Not saturated — Tier 4 and Open Problems remain largely unsolved (3 of 50 Open Problems solved by AI) |
|---|
| Contamination risk | Very low — problems are unpublished and commissioned specifically for the benchmark, with a held-out subset |
|---|
| Maintainer | Epoch AI (creation funded by OpenAI, which drew independence criticism in 2025) |
|---|
Watch out: You cannot run it yourself — the problems are held by Epoch, so every number is Epoch's number and there is no way to reproduce, audit the items, or evaluate your own fine-tune. OpenAI funded the benchmark's creation and had access to problem data, which triggered a substantial 2025 controversy about independence; Epoch improved disclosure and maintains a held-out set, but a funder-evaluated-by-the-funded arrangement remains structurally compromised. Answer-verifiable problems are a biased sample of mathematics — they exclude most of what research mathematicians actually do, which is why the Lean Erdős track exists. Absolute scores are low and move in lumps, so it discriminates poorly among mid-tier models. Wrong choice for any practical model-selection decision.
not applicable