Retrieval, long context, or fine-tuning

Four techniques, four different problems

Retrieval, long context, fine-tuning, and tool access are not competitors on a single axis. Each solves a different problem:

Retrieval (RAG) — select a small relevant subset of a large corpus at request time and put it in the prompt. Solves: the corpus is far larger than any context window, and changes.
Long context — put the whole relevant body of text in the prompt every time. Solves: the corpus is small, bounded, and needed in full.
Fine-tuning — adjust model weights on example inputs and outputs. Solves: the model can already do the task but does it in the wrong shape, tone, format, or with the wrong default judgments.
Tool / API access — give the model functions that query your systems of record. Solves: the answer is computed or transactional, not written down anywhere.

The expensive mistake is using the third to solve the first. Fine-tuning teaches form, not facts.

Why fine-tuning does not install knowledge

Fine-tuning updates weights by gradient descent over your examples. A fact appearing a handful of times is a weak signal against a pretraining corpus of trillions of tokens. Models absorb examples that conflict with or extend pretrained knowledge much more slowly than examples consistent with it, and training long enough to actually learn the new facts tends to increase hallucination on adjacent questions: you have taught the model to answer confidently in a domain without teaching it the distribution.

The failure is silent. A model fine-tuned on your product docs answers in the right voice with the right terminology, and gets version numbers, prices, and limits wrong. Retrieval fails loudly ("no relevant documents"); fine-tuned recall fails fluently.

Fine-tuning is the right answer when the target is a behavior: emitting a strict schema without a validator retry loop, classifying into your taxonomy, matching a house style, adopting a specialist's default judgment on ambiguous cases, or letting a small fast model imitate a frontier model on one narrow task at a fraction of the cost. Parameter-efficient methods (LoRA-style adapters) make this cheap enough that a few hundred to a few thousand well-labeled examples is a normal budget — the labeled examples, not the GPU time, are the real cost.

Long context: better than it was, still not free

Stuffing everything into the prompt is the highest-quality baseline for small corpora: nothing is lost to a bad retrieval, and the model sees cross-document relationships that chunk-level retrieval destroys. The costs:

Prefill is not free. Attention cost grows quadratically with sequence length, so time-to-first-token grows faster than linearly with prompt size — a 10x longer prompt costs more than 10x the prefill latency.
Input tokens dominate the bill. Output tokens price several times higher than input tokens per token, but a long-context design sends tens of thousands of input tokens per turn and receives hundreds, so input is where the money goes. Run your own token mix through the cost calculator.
Prompt caching changes the arithmetic more than model choice does. Major APIs will serve a stable prompt prefix from cache — some automatically, some only when you mark the prefix — at a fraction of the normal input rate (roughly a tenth is typical), against a write premium on the first call and a time-to-live measured in minutes by default, with hour-scale options at a higher write premium. Read the exact multipliers off the cost calculator; they differ by provider and by TTL. A fixed corpus queried repeatedly inside the TTL window is the case where long context beats retrieval on cost outright. A corpus queried once an hour is not.
Recall degrades unevenly with length. An advertised context window is capacity, not uniform quality. Accuracy tends to sag for material buried mid-prompt, and multi-hop reasoning over facts spread across a very long input is worse than over the same facts concentrated — but how much varies sharply by model and has been moving. Compare window sizes across providers in the model index, then measure recall on your own data instead of trusting the number.

Heuristic: if the corpus fits comfortably inside the window with room for the conversation, and changes rarely, stuff it. Beyond that, retrieve.

Retrieval, conceptually

Chunking. Documents are split into passages, typically 200–1000 tokens with 10–20% overlap. Chunk boundaries are the highest-leverage knob most teams never touch: split on document structure (headings, sections, function definitions) rather than fixed character counts, and prepend each chunk with its document title and section path so an isolated passage stays interpretable. Retrieve small chunks for precision, then expand to the surrounding section before sending to the model.

Embeddings. Each chunk becomes a vector; the query becomes a vector; nearest neighbors are candidates. This is semantic matching, so it finds paraphrases and synonyms that keyword search misses. It also fails on exact identifiers — error codes, SKUs, part numbers, surnames — and handles negation poorly.

Hybrid search. Run lexical (BM25) and dense retrieval in parallel and fuse the ranked lists. Reciprocal rank fusion is the standard combiner, needs no score calibration, and beats either leg alone on mixed query workloads. If you build one thing beyond naive vector search, build this.

Reranking. Retrieve 50–100 candidates cheaply, then score each query–document pair with a cross-encoder that reads both together. Cross-encoders are far more accurate than embedding similarity and far too slow to run over a whole corpus, which is exactly why the two-stage shape exists. Reranking a candidate set adds tens to low hundreds of milliseconds and is usually the largest quality gain per unit of engineering effort in a RAG pipeline.

Evaluation. A retrieval system without a fixed golden set of query/expected-document pairs is not tunable; you will change chunk size, feel that it got better, and ship a regression. Measure retrieval (recall@k, hit rate) separately from generation (faithfulness, correctness). Most "the model is hallucinating" tickets are retrieval misses.

Comparison

	Retrieval (RAG)	Long context	Fine-tuning	Tools / API
Freshness	Minutes — re-embed changed docs only	Immediate — swap the text	Stale from the day training ends	Real-time by construction
Update cost	Incremental re-index of the delta	Zero, but re-pay prefill or bust the cache	Full retrain + eval cycle	Zero
Marginal cost / query	Low — small retrieved payload	High uncached, low on a cache hit	Lowest per token; large fixed training cost	Low, plus your backend's cost
Latency	+retrieval hop (typically 50–300 ms)	Prefill grows superlinearly with length	Fastest — no extra input tokens	+round-trip per tool call, often several
Attribution	Native — return document IDs and spans	Possible; model may misattribute across sources	None. Weights carry no provenance	Native — the record is the citation
Access control	Enforceable per user at query time	Enforceable per prompt assembly	Not enforceable — baked into weights	Enforced by the underlying API
Build effort	Medium-high: ingest, index, eval harness	Low	High: labeled data, training, eval, versioning	Medium: schemas, auth, error handling
Best at	Large, changing, citable corpora	Small bounded corpora needing whole-document reasoning	Format, style, taxonomy, cost reduction	Live state, transactions, computation

Anything in a fine-tuned model's weights cannot be deleted, scoped, or redacted per user. If your corpus contains customer records subject to deletion requests, or documents whose visibility differs by role, fine-tuning on it creates a compliance problem no amount of prompting will fix — the only remedy is retraining. Retrieval keeps the permission check at query time, where it belongs.

What people get wrong

Fine-tuning to add facts. If you cannot state the target as "the output should be shaped like this", fine-tuning is the wrong tool.
Treating "retrieval or long context" as the decision. They compose. Retrieve aggressively, then send a generous amount of surrounding context — the modern design is fewer, larger, better-ranked passages, not twenty 200-token fragments.
Pure vector search, no lexical leg. A support engineer searches for error code E4412, the embedding model has no representation of that string, and retrieval returns thematically similar prose about errors in general.
No reranker. Teams tune embedding models for weeks to recover quality a reranker delivers in an afternoon.
Chunking by character count. Splitting mid-table, mid-function, or mid-sentence poisons the index; every downstream fix compensates for it.
Ignoring prompt caching in the cost model. Estimates built on uncached input pricing overstate long-context designs by roughly the cache-read discount — and understate them when the traffic pattern never hits the TTL.
Fine-tuning a frontier model when the goal was cost. The savings come from distilling a large model's behavior into a small one. Fine-tuning the large model and still paying frontier prices captures none of it. See compare models for the tier gaps.
Retrieving when the answer is in a database. "What is this customer's current balance" is a tool call. Embedding nightly account snapshots is a slow, stale, expensive way to build a worse SELECT.

A default path

Start with retrieval over hybrid search plus a reranker, with tools for anything transactional or computed. Move to long context for the subset of documents small enough to include whole, and use prompt caching to make that affordable. Reach for fine-tuning last, against a specific measured gap in output form that prompting and few-shot examples could not close, with a golden set already in place to prove it helped.