How to choose a model

Start from the shape of the task

Start with the job, not a shortlist of models. Four shapes cover most production work, each stressing a different capability:

Classification and extraction. Bounded output — a label, or JSON against a fixed schema — measurable against ground truth. The hard part is edge cases and schema adherence, not reasoning depth.
Generation. Open-ended text, graded against a rubric or a human. Style, instruction-following, and tone consistency dominate.
Agentic work. Multi-step tool-calling loops where the model decides what to do next. Per-step accuracy compounds: 95% per step over 20 steps is a 36% task success rate. This shape punishes weak models far harder than single-turn benchmarks suggest.
Long-context work. A large corpus — a codebase, a contract set, a transcript archive — queried across. Retrieving one fact is nearly solved; synthesising several scattered facts is not.

A model excellent at one shape can be mediocre at another: one tuned for chat helpfulness may be worse at strict JSON than a cheaper model with constrained decoding. The shape determines what you measure.

Tiers, not products

Names and version numbers churn quarterly; the tier structure has been stable for years, and it is what determines your architecture.

Tier	What it is	Good at	Breaks on
Frontier	Newest flagship from a major lab, usually with an extended-reasoning mode	Novel reasoning, long agentic chains, ambiguous specs, code across many files	Latency budgets, high-volume per-item work
Mid	The workhorse from the same labs, often a distilled sibling of the frontier model	Most production generation, RAG answers, moderate tool use	Long multi-step plans, adversarial or unusual inputs
Small / fast	Compact hosted models optimised for throughput	Classification, routing, extraction, reranking, guardrails	Multi-hop reasoning, many-tool schemas, deep nesting
Open-weight	Weights you host or rent, spanning small to near-frontier	Data residency, fine-tuning, fixed-cost high volume, no vendor lock	Ops burden: you own uptime, scaling, upgrades

Price falls monotonically from frontier to small, but do not assume an order of magnitude between adjacent tiers: each step is often a factor of two to five — narrow enough that a cached frontier model can undercut an uncached mid-tier one on the same workload. Read live rates from the cost calculator instead of treating tier as a proxy for cost.

Capability floor versus cost ceiling

The capability floor is the weakest model that produces acceptable output; it is a property of the task and can only be found empirically. The cost ceiling is the most you can spend per request while the feature still makes economic sense, set by volume, tokens per request, and margin.

Compute the ceiling in cost per completed task, not per million tokens. Three things make the sticker price misleading:

Output tokens bill at a multiple of input tokens — around 5x on current flagship lineups. A short prompt with a long answer inverts the intuition you get from input prices.
Reasoning tokens bill as output, whether or not you are shown them. Extended thinking can multiply the output bill several-fold on one call.
Agentic loops resend the whole transcript every turn. Cost grows with roughly the square of turn count: a 40-turn task sends its first turn 40 times.

Two mechanisms move real spend more than switching models does.

Prompt caching reprices repeated input to roughly a tenth of standard, which matters most for the agentic and long-context shapes that generate the most tokens. Writing an entry costs a premium over standard input — larger for a longer time-to-live — so caching pays from about the second read on, not the first. Caches key on a model and an exact prefix: switching models forfeits reuse, and any byte that changes early in the prompt invalidates everything after it.

Batch APIs commonly give about 50% off for asynchronous delivery — hours, not seconds — and stack with caching. Free money for backfills, evals, and nightly enrichment.

If the floor sits above the ceiling, the answer is not a worse model. Reshape the task: split it, add retrieval, constrain the output schema, cache aggressively.

Benchmark on your data, not on leaderboards

Leaderboards narrow dozens of candidates to three or four. They cannot pick between those three or four, for structural reasons:

Contamination. Published test sets leak into training corpora; scores drift upward without matching capability gains.
Aggregate scores hide your distribution. One percentage averages over subjects, languages, and formats you may not care about.
Prompt sensitivity. Harnesses use tuned prompts and answer parsers. A few points between models is often smaller than the gap between two prompt formats for the same model.
Preference arenas measure preference. Human-vote rankings reward length, formatting, and confident tone — actively harmful traits for extraction or classification.

Build a private eval instead. 100-300 real, hand-labelled examples covering the input distribution, over-sampling the cases you know are hard, separates tiers reliably. Use exact-match or F1 where ground truth exists; a rubric with a model-as-judge where it does not, hand-checking a sample of the grades. Version it in the repo and re-run on every model and prompt change. It is a regression test, not a leaderboard of your own — it makes model swaps a decision rather than a leap.

A model-as-judge from the same family as the candidate tends to prefer its own output style. Use a different family from the model under test, and validate the judge against human labels before trusting its verdicts.

Start high, then step down

Prototype on a frontier model. You are testing whether the task is achievable at all. If the best available model fails, no prompt tuning on a cheaper one will save you — change the decomposition instead.
Freeze an eval set from that working prototype, including the failures you had to fix.
Step down one tier, re-run the eval, and compare against your acceptance threshold — not the frontier model's absolute score.
Repeat until quality drops below the threshold, then go back up one step. That is your production model.
Split the pipeline. Most systems are not one task. Route the easy 80% to a small model and escalate the rest, or extract with a small model and synthesise with a larger one. Price the split honestly: two models means two cache namespaces, so a cascade forfeits reuse across the boundary.

Starting at the bottom inverts the problem: when a small model fails you cannot tell whether the task is too hard, the prompt is wrong, or the model is too weak. Redo the step-down when prompts change materially and re-check every few months — the mid tier absorbs frontier capability on a rolling basis, so the same eval often passes a cheaper model a year later. Use compare models within a tier and the model index for current specifications.

Latency is a separate axis

Cost and quality do not predict speed. Measure two numbers independently:

Time to first token — perceived responsiveness in streaming interfaces.
Output tokens per second — total completion time for long answers.

A model can have a fast first token and slow generation, or the reverse. Extended reasoning adds seconds to minutes before any visible output, and streaming does not hide it — several models suppress the reasoning stream by default, so the user sees a blank pause unless you enable summarised thinking.

Interactive UI generally needs first token under about a second, so streaming a small or mid model usually beats a frontier model that thinks first. Voice is stricter: round-trip budgets in the low hundreds of milliseconds rule out the frontier tier for the conversational turn. Background and batch work has no latency constraint — where the frontier tier is easiest to justify.

When a small model genuinely wins

Not as a compromise — as the correct choice:

High-volume classification and routing, where the label space is small and you have training data. A tuned small model can match frontier accuracy at a fraction of the cost.
Extraction against a fixed schema, especially with constrained decoding.
Reranking and filtering in front of a larger model, where recall matters more than nuance.
Guardrails, which run on every request and must be fast.
Deterministic-adjacent work where the real problem is parsing and the model is glue.
After distillation. Label a few thousand examples with a frontier model, fine-tune a small or open-weight model on them, and get frontier-adjacent quality on that narrow task at small-model economics.

Decision table

Task	Start at	Step down to	Primary metric
Intent routing, spam and topic classification	Mid	Small, then fine-tuned small	F1 on a labelled set
Structured extraction from documents	Mid	Small with constrained decoding	Field-level exact match
RAG question answering	Mid	Small for short factual answers	Groundedness plus answer accuracy
Summarisation of short inputs	Mid	Small	Rubric score, hallucination rate
Long-document synthesis across sources	Frontier	Mid once the prompt is stable	Multi-fact recall, not needle retrieval
Code generation in an existing codebase	Frontier	Mid for single-file edits	Tests passing, diff review rate
Multi-step agents with tool use	Frontier	Mid only after per-step accuracy is measured	End-to-end success over full runs
Real-time voice or sub-second UI	Small or fast mid	—	Time to first token at p95
Bulk offline enrichment	Frontier via batch API	Mid via batch API	Cost per thousand items
Regulated data, no third-party processing	Open-weight	Smaller open-weight	Same eval, plus hosting cost per request

What people get wrong

Choosing on price per million tokens. A cheap model that needs three retries, longer few-shot prompts, and a repair step costs more than one that answers correctly the first time.
Assuming prompts transfer. Prompts overfit to a model's quirks, and instructions tuned for an older generation can actively degrade a newer one. Moving a tuned prompt across tiers and concluding the model is worse is a common, expensive error — re-tune before you judge.
Treating window size as usable context. The advertised window bounds what the model accepts, not what it reasons over reliably. Multi-fact accuracy degrades well before the limit, and some providers price tokens above a threshold at a premium.
Testing only the happy path before stepping down. Tier differences show up in the tail: malformed inputs, unusual languages, adversarial phrasing, long tool chains. A step-down that passes on clean examples fails in production.
Using a reasoning model for non-reasoning work. Extended thinking on a classification task burns output tokens and latency for accuracy a small model already had. Where the tier is needed, turn effort or thinking depth down before changing models.
Never revisiting. Choices made 12 months ago are almost always leaving money on the table; re-running a stored eval takes an afternoon.
Ignoring the cost of switching later. Keep the provider call behind one interface, prompts in version control, and the eval runnable against any endpoint. Cheap up front, painful to retrofit.