Context windows in practice

Tokens, not words

A context window is a token count, not a character or word count. English prose averages roughly four characters per token, so 1,000 tokens is about 750 words. Everything else is worse: JSON with verbose keys, minified payloads, base64 blobs, and non-Latin scripts all tokenise poorly, and CJK or Indic text can cost more than one token per character. Source code sits between the two.

Two operational consequences. First, tokenisers differ between vendors and between model generations from the same vendor, so the same bytes can cost materially more or fewer tokens after a model change — within a single vendor's lineup, a generation change has moved counts by a third. Re-baseline when you migrate; see compare models for what is actually equivalent. Second, do not estimate with a local tokeniser library built for a different provider. Use the provider's own token-counting endpoint. An estimate 20% low is the difference between a request that runs and a 400 at the edge of the window.

Input and output limits are separate

The advertised context window is an input ceiling. Maximum output is a distinct, much smaller number set by its own request parameter, and it has grown far more slowly than input windows — the ratio is model-specific and worth looking up rather than inferring. Read the input ceiling and the output ceiling as separate fields from the provider's models endpoint or the model index. Note that the output field is frequently named max_tokens, which is easy to misread as the window.

Four failure modes follow:

• You cannot round-trip a large document. A window that swallows a book will not emit a rewritten book. Long transformations must be chunked on the output side regardless of how much input fits.
• Reasoning tokens are output tokens. On models with extended or adaptive thinking, internal reasoning is billed as output and drawn from the same output budget — including on models that summarise or hide the reasoning, where you pay for tokens you never see. A generous cap can be eaten entirely by thinking on a hard problem.
• The output cap is enforced but invisible to the model. It truncates mid-sentence; it does not cause the model to plan a shorter answer. Some providers now expose a separate advisory token budget that the model can see and pace itself against — that is the parameter that changes behaviour, not the cap.
• Large output caps require streaming. Above a certain value a non-streaming request hits the client or gateway timeout before it hits the model limit, and SDKs increasingly refuse the combination outright. Always check the response's stop reason: hitting the cap is a truncation, not an error, and it is silent unless you look.

Advertised length is not useful length

Retrieval accuracy is not uniform across the window. Performance on retrieving a fact from a long context is roughly U-shaped in the fact's position — best at the beginning and end, worst in the middle ("Lost in the Middle", Liu et al., TACL 2024). Separately, accuracy on a fixed task degrades as input length grows even when the task itself does not get harder.

Two cautions when reading long-context benchmark numbers:

• Needle-in-a-haystack is the easy test. Retrieving a single lexically distinctive string from filler is close to a string search, and models score near-perfectly on it well past the length where they start failing real work. Degradation shows up first on multi-hop questions, aggregation ("how many of these records match?"), and anything requiring the model to notice an absence.
• Distractors dominate. Accuracy falls much faster when the haystack contains plausible near-misses than when it contains unrelated filler. Production context is almost entirely near-misses: prior versions of the same file, similar tickets, an earlier draft of the same paragraph.

Treat the effective window as a fraction of the advertised one, established by measurement on your own task, not by the spec sheet.

What context costs

The relationships that stay true across price changes — run your own traffic shape through the cost calculator:

• Output tokens cost several times input tokens per token: commonly 5x, and as much as 8x on some providers.
• Cache reads cost roughly an order of magnitude less than fresh input; cache writes carry a modest premium over fresh input, and nothing at all on providers that cache automatically. Caching is the only lever that reduces cost without trading away quality, so it comes first.
• Batch or asynchronous processing typically halves the rate for work that tolerates delay.
• Caches are scoped to a model. A cost cascade that routes between two models forfeits cache reuse across them, which frequently costs more than the cheaper model saves. Measure the capable model at a lower reasoning-effort setting before building one.

Latency scales with input length: time-to-first-token is dominated by prefill over the whole prompt, so a full window adds seconds before the first output token.

The trap specific to agents is that context cost is quadratic in turns. Every turn resends the entire history, so an N-turn loop with a growing transcript pays roughly N²/2 times the average turn's tokens. A 60-turn run with a 50k-token working context can bill several million input tokens, most of it re-reading. Prompt caching flattens the constant factor here more than any other change.

Context rot in long runs

Distinct from position-based degradation: over a long agent run, the transcript accumulates content that is wrong now. Superseded file contents, tool calls that failed and were retried differently, an abandoned plan, an instruction the user revised three turns later. The model has no reliable way to know which of two contradictory statements in its context is current, and will sometimes act on the stale one.

Symptoms worth alerting on: the agent re-reads a file it already read, reverts a change it already made, cites a value that was corrected earlier, or restates a plan you rejected. These are context hygiene problems, not capability problems, and a bigger window makes them worse rather than better.

A large window is not a substitute for retrieval

They compose: retrieve to select, then use the large window to give the selected material generous room, alongside stable instructions and examples that cache well.

Strategies and when to use them

Order of application for an agent running out of room: cache first (free), then externalise and retrieve (cheap, reversible), then clear dead tool results, and only then compact.

What people get wrong

• Estimating tokens by dividing characters by four, on code. The ratio holds for English prose and is badly wrong for source, JSON, and non-Latin text. Count, don't divide.
• Assuming the output limit equals the context window. It never does, and the error surfaces as truncated output rather than a rejected request.
• Reading needle-in-a-haystack scores as long-context competence. They measure lexical lookup, not reasoning over the same span.
• Putting a timestamp, request ID, or unsorted JSON near the top of the prompt. This invalidates the cache on every request. Verify by checking that cache-read token counts are non-zero across repeated calls; if they are zero, something in the prefix is varying — or the prefix is below the minimum cacheable length.
• Bolting a bigger window onto an agent that is failing at 60 turns. If the failure is contradictory or stale context, more room extends the run without fixing the accuracy problem, at higher cost.
• Compacting on token count alone. Compact at semantic boundaries — after a sub-task completes — rather than when a threshold trips mid-operation.
• Skipping the measurement. Effective context length is a property of your task, prompt, and data. Build a small eval that varies input length while holding task difficulty fixed; the length at which accuracy starts sliding is your real budget.