Understanding token pricing

What you are actually buying

Every LLM API meters text in both directions, in tokens. A token is a subword unit from the model's tokenizer — roughly 4 characters of ordinary English prose, or about 0.75 words. That ratio is a planning heuristic, not a rule: minified JSON, deeply indented code, UUIDs, base64, and non-Latin scripts tokenize worse — commonly 2-3 characters per token, close to 1 for CJK. Images and PDFs are converted to a token count too, so they land on the same meter as prose.

Do not estimate a bill from a character count when precision matters, and do not use a third-party BPE library built for another vendor's model. Tokenizers differ across vendors and change between generations of the same family — a tokenizer change can move the same prompt by roughly 30% with no text change, which shifts your bill and your context-window headroom together. Use the provider's own token-counting endpoint: it returns a count without running inference, is free, and is rate-limited separately from message creation. Count against the exact model ID you will send to.

Input and output are not the same price

Output tokens cost a multiple of input tokens — 5x across the current Claude lineup, and 3-5x is typical of frontier vendors generally. The gap reflects how inference runs, not margin policy. Input is processed in a single parallel prefill pass over the whole prompt. Output is generated one token at a time, each token requiring a full forward pass, and the decode phase is memory-bandwidth bound and batches far less densely.

Consequence: you cannot rank two workloads by prompt size. A retrieval-heavy request with 50,000 input and 300 output tokens is input-dominated; an agent turn with 3,000 input and 8,000 output tokens is output-dominated. Run the arithmetic per route with current numbers from the cost calculator.

You pay for the entire conversation on every turn

Messages-style APIs are stateless. There is no server-side conversation; you resend the full history each turn, and every resent token bills as fresh input. Cost per turn grows linearly with turn count, and cumulative cost grows quadratically.

Example: a chat app with a 1,500-token system prompt, ~150-token user messages, and ~400-token replies. Turn sizes never change. The last column expresses total cost in input-token-equivalents, pricing output at 5x input.

Turn 40 costs about 7x what turn 1 cost. The 40-turn session costs about 157x a single turn, not 40x. Only 23,100 tokens of that 495,000-token input bill are text the API has not already been sent — roughly 95% of it is retransmission. This is why "average cost per message" is a misleading planning number: your bill is driven by the tail of long sessions, and a p99 session can cost two orders of magnitude more than the median.

The same mechanic drives agent-loop cost, with bulkier fuel. Tool results accumulate in the history and get resent every iteration, so a loop that reads three large files early pays for those files on every subsequent turn.

System prompts and tool schemas bill per call

Prompts render in a fixed order — tool definitions, then system, then messages — and all of it is input on every request. A 12,000-token tool schema block attached to a 200-token classification request means 98% of the uncached input bill is boilerplate that never varies.

Two fixes. Cache the prefix (below). Or defer tool loading: the model searches a catalog and only the schemas it discovers enter context. The published threshold for deferring is 10 or more tools, or more than ~10,000 tokens of definitions; below that, load everything up front. Deferred definitions are excluded from the cached prefix and expand inline when discovered, so deferring does not break caching — and it protects tool-selection accuracy, which degrades once more than roughly 30-50 tools are loaded at once.

Reasoning tokens are billed and usually invisible

Models with extended reasoning generate internal thinking tokens before the visible answer. They are billed as output tokens, and current models do not return the raw chain of thought — you get a summary, or nothing, depending on a display setting. Billing is identical either way.

So output_tokens in the usage object can be several times what the rendered response would suggest. Any cost model built by counting characters in the visible reply will under-forecast on reasoning-heavy routes. Read the meter, not the text. Reasoning depth is controlled by an effort or budget parameter, and dropping it a notch is usually the first quality-trading lever worth testing — after the free ones.

The four input meters

Prompt caching splits input into three rates, so a single request can bill four different meters:

Caching is a prefix match on exact bytes. One changed byte anywhere in the prefix invalidates everything after it. Break-even is two requests on a short TTL (1.25 + 0.1 = 1.35 versus 2.0 uncached) and three on a long TTL (2.0 + 0.2 = 2.2 versus 3.0). Applied to the chat curve above, caching cuts the 40-turn input bill by roughly 6x and total session cost by roughly 3.5x — consistent with the 2.5-3.7x Anthropic has published for agent loops at 81-90% hit rates.

There is a floor, a ceiling, and a budget. Prefixes below a model-dependent minimum (commonly 512-4,096 tokens) silently do not cache — no error, just a zero in the write meter. Input that is genuinely unique per request can never cache, so a workload that is mostly payload has a small caching ceiling however you place breakpoints. And you get only a handful of breakpoints per request (four on the Claude API), so they have to sit on real stability boundaries.

Batch and the gap between list price and your bill

Asynchronous batch processing bills every token in the request at 50% of standard rates, and the batch and caching discounts stack. The cost is latency: results arrive within a 24-hour window — an expiry, not an SLA — and requests that expire unprocessed are not billed. Cache hits inside a batch are best-effort, since requests are processed concurrently — pair batch with the longer cache TTL when requests share a prefix. Server-side tools do run inside a batch request; a client-side tool call cannot, because the batch has to return before you can answer it. Evaluation runs, backfills, and scheduled jobs belong here; anything with a user waiting does not.

List price is a starting point, not a forecast. What you actually pay is moved by: caching hit rate, batch share, committed-use or enterprise discounts, premium tiers (priority routing and faster-decode modes bill more per token than standard), and reseller pricing on cloud marketplaces, where partner-operated deployments have their own rate cards. Two teams on the identical model can differ several-fold in effective cost per task. Compare current rates across models and tiers on the model index and compare models.

Cost per completed task, not cost per request

The only number that ranks options correctly is cost per completed task: total spend across all attempts, divided by tasks finished to your quality bar. Per-request and per-token prices routinely rank options backwards, because failure is not free.

• A truncated response that hit max_tokens bills every token it generated and delivers nothing. Capping output to save money buys proportionally fewer completions, not a lower cost per completion.
• A retry after a malformed tool call bills the whole prompt again — and by then the prompt includes the failed attempt.
• A cheaper model that needs three loop iterations where a stronger one needs one is not cheaper; each iteration resends the whole growing history.
• Vendor benchmarks put frontier models at reduced reasoning effort ahead of cheaper mid-tier models on cost per completed task, at equal or better accuracy. Test the strong model at low effort before building a multi-model cascade — and note that a cascade forfeits cache reuse, since caches are model-scoped.

Log the usage object on every call, tag it with a task ID, and roll up per task rather than per request.

What people get wrong

• Assuming the API remembers the conversation. It does not. Every turn resends everything.
• Budgeting from the average message. Cost is concentrated in long sessions; model the distribution, not the mean.
• Interpolating a timestamp, user name, or session ID into the top of the system prompt. It sits at prefix position zero and makes every downstream token uncacheable, per user, forever.
• Enabling caching on prompts that never repeat. With no reads, the write premium is a pure surcharge.
• Reading input_tokens as prompt size. It is only the uncached remainder — the tail after the last breakpoint. Total prompt = uncached + cache writes + cache reads. An agent that ran for hours can show 4K there.
• Estimating output cost from the visible reply. Reasoning tokens are billed and usually hidden.
• Reusing token counts across a model change. Recount on the model ID you will actually send to; a tokenizer swap moves the number by tens of percent.
• Comparing vendors on input price alone when the workload is output-heavy, or on output price alone when it is a long-context RAG pipeline.
• Treating context-window management as a cost lever. Clearing old tool results rewrites the cached prefix; in the runs vendors have measured, it cost more than it saved. It buys window space, not money.