Cutting API costs

Measure before you cut

You cannot rank levers without per-path numbers. Log four, tagged by code path — production, evals, and CI all count:

• Requests per day.
• Input tokens, split into uncached, cache writes, and cache reads.
• Output tokens, including reasoning/thinking tokens, which bill as output.
• Attempts per completed task, including retries your SDK performs silently.

Anthropic's usage object breaks out cache_creation_input_tokens and cache_read_input_tokens from input_tokens; log all three. A total-only log cannot produce a cache hit rate — the cost metric most teams are missing.

Two relationships shape everything below. Output costs several times input at list price (5x on current Claude models), so a 200-token output cut beats a 200-token prompt cut. Cache reads cost roughly a tenth of uncached input, less on some newer models, which makes a long stable prefix nearly free at steady state and expensive only on the first call. Current multipliers live in the cost calculator and the model index, not in your head.

Rank spend by requests × (input_tokens + 5 × output_tokens), substituting your provider's output ratio. Fix the top three paths; ignore the rest.

Free wins: no quality cost

Do all of these before touching model selection.

1. Prompt caching

Caching stores the computed prefix so repeat requests skip re-processing it. On the Claude API a cache write costs 1.25x normal input at the default five-minute TTL, 2x at the one-hour TTL; a read costs ~0.1x. Break-even depends on TTL: at five minutes, two requests clear it (1.25 + 0.1 vs 2.0); at one hour you need three (2.0 + 0.2 vs 3.0). A read refreshes the timer for free, so requests less than five minutes apart keep the default TTL warm indefinitely — the one-hour TTL then buys only the doubled write.

This is the highest-leverage lever in agentic workloads, where a large system prompt plus tool definitions is resent every turn. A 20-turn session on a cached prefix pays roughly 1.25 + 19 × 0.1 ≈ 3.2 prefix-equivalents instead of 20; Anthropic measures agent-loop savings at a factor of 2.5-3.7 at realistic hit rates. Opt-in or implicit by provider, what decides whether caching works is prefix stability.

2. Cache-prefix hygiene

Caching matches an exact byte prefix. The prompt renders in a fixed order — tools, then system, then messages — and a change at one level invalidates that level and everything after it. Change one tool definition and you lose the system prompt and conversation cache too.

What silently destroys hit rates:

• A timestamp, request ID, or "today's date" near the top of the system prompt. The most common one by far. Every request writes a new entry and never reads one, so the bill goes up by the write premium.
• Nondeterministic serialization. A config dict, tool schema, or retrieved-document set assembled from an unordered collection gives a different prefix per process. Sort keys and documents explicitly.
• A tool list that varies per user. Filtering by permission or feature flag gives every cohort its own entry. Ship the full list; enforce permissions at execution time.
• Switching models mid-workload. Caches are model-scoped, with no escape hatch; this forces a full rebuild.
• Varying thinking or reasoning-effort config per request. It always invalidates the message cache, and the tools and system caches above it on some models. Pin it per route. (tool_choice leaves tools and system caches intact; max_tokens and temperature are not part of the key at all.)
• A prefix below the minimum cacheable length. The floor is model-dependent and not monotonic across generations — some newer models cache shorter prefixes than their predecessors. Below it nothing caches and no error is returned.

The fix is mechanical: static content first, volatile content after the last breakpoint, and a test that hashes the serialized prefix across two synthetic requests. Where the API offers a cache-preserving channel for late instructions — a system message inside messages[] rather than an edit to the top-level system prompt — use it.

3. Trim the system prompt and few-shot examples

Few-shot blocks accumulate — one example per failure mode, none ever removed. Ablate: drop each, measure on your eval set, keep what moves a metric. Verbose descriptions across twenty tool schemas can outweigh the system prompt, and past roughly 10K tokens of schemas, deferred tool loading beats hand-trimming. Do this after caching: a cached prefix bills at a tenth, so trimming it saves a tenth of what it looks like.

4. Control output tokens

• Ask for less. "Answer in at most three sentences," or a strict output schema with no prose wrapper. Lower reasoning effort cuts thinking tokens, preamble, and tool-call chatter, but imposes no length bound on the answer — prompt for that separately.
• Stop paying for restated context. A prompt ending "explain your reasoning, then give the answer" doubles output cost where nobody reads the reasoning.
• Treat max_tokens as a backstop, not a knob. The model cannot see it; hitting it truncates mid-thought. In Anthropic's coding runs a 16K cap ended a sixth of one model's attempts and a third of another's, none of them solved: less spent per attempt, proportionally fewer solves. Set it as a runaway guard, stream anything large, and treat stop_reason: max_tokens as a failed attempt rather than a retry at the same cap.

5. Batch API for non-urgent work

Async batch endpoints run 50% cheaper across major providers, with a 24-hour completion window. Anything off the user-facing path belongs here: backfills, evals, classification sweeps, nightly summarization. Batch and cache discounts stack, but cache hits inside a concurrent batch are best-effort — use the longer TTL when items share a prefix, and don't read those misses as a broken prefix.

6. Deduplicate retries and wasted calls

A failed attempt still bills the tokens it produced; the retry bills them again.

• Retry on 429 and 5xx; never retry a 400 — it fails identically. Schema and history-binding errors are 400s.
• Know your SDK's defaults before adding your own. The Anthropic SDKs already retry twice with backoff, so an application-level loop stacks multiplicatively — and timeouts are retried too, putting worst-case wall clock at timeout × (max_retries + 1).
• Cache deterministic results in your own store. A hash-keyed result cache is cheaper than any model-side optimization.
• Audit agent loops for duplicate work: re-reading a file or re-issuing a search is a flat multiplier on spend.

Levers that trade quality for cost

Each needs an eval set with a pass threshold before you touch it.

Reasoning effort

The effort or thinking-budget control governs how many tokens go into reasoning. It is a one-parameter change, so reach for it first. The curve is workload-shaped. Anthropic measured near-flat curves on research and knowledge work — a mid setting matching default accuracy at 70-85% of the cost — while long-horizon coding is a real tradeoff, roughly two points of pass rate for half the cost. Effort is part of the cache key on most models, so vary it across routes, not mid-conversation.

Re-run failures instead of routing

Where the workload has a cheap failure signal — tests, a schema check, a validator — run everything at low effort and re-run only failures higher. In Anthropic's coding runs this matched the always-default pass rate at about half the cost, counting the wasted cheap attempts. It beats a model cascade because it keeps one model, and therefore one cache namespace.

Model step-down

Down a tier is often an order of magnitude per token — the largest single lever, the riskiest, and it forfeits every warmed cache entry. Step one tier at a time, re-run evals, and re-tune the prompt: prompts written for a stronger model are underspecified for a weaker one, and the missing explicitness recovers much of the gap. Price candidates on the hardest tenth of your traffic — on typical tasks every model looks the same. Use compare models to shortlist.

Routing and cascades

A classifier picks the tier up front (low latency, silent misroutes); a cascade answers cheap and verifies after (more accurate, but escalations pay twice). The cascade pays only when cheap + verifier + escalation_rate × strong < strong. Solve it with your own numbers, counting the verifier and the cache reuse lost across two models: a wide price gap tolerates a lot of escalation, two adjacent tiers almost none.

Distillation

Fine-tune a small model on frontier-model outputs for one task. Highest ceiling, highest maintenance cost, justified only at sustained volume on a stable task. Task drift kills these; budget for retraining.

Lever table

What people get wrong

Treating a cheaper per-token price as a cheaper system. A model that needs two attempts, writes longer output, or triggers a human escalation is not cheaper. Compute (price × tokens × attempts) + escalation_cost per successful task.

Optimizing the prompt before checking the cache hit rate. A week spent compressing a system prompt that already billed at a tenth, while the leak was a per-request timestamp above the breakpoint.

Counting characters instead of tokens. Use the provider's token-counting endpoint, not a tokenizer borrowed from another vendor's model. Non-English text, code, and JSON tokenize far worse than English prose.

Ignoring reasoning tokens. They bill as output and can dwarf the visible response. A "short answer" endpoint can be the most expensive one you run.

Mistaking context management for cost management. Context editing and compaction make room in the window; they do not save money. Every clearing pass rewrites the cached conversation and works against caching.

Assuming a longer context window is free capacity. Stuffing in a corpus because it fits bills on every uncached call. Retrieval that sends 4k relevant tokens beats sending 200k, on cost and on accuracy.

Working order

Instrument usage → check the cache hit rate and fix the prefix → move offline work to batch → fix retries and dedup → shorten output → build an eval set → step down effort → step down model tier → route or distill only at sustained volume. Change one lever per deploy so a revert is clean, and stop once the remaining spend is below the engineering cost of the next step.