Glossary

The vocabulary you meet in provider documentation, pricing pages and API errors, defined as the terms are actually used in 2026 rather than as they were coined.

Models and training

• Open-weight model — a model whose trained parameters are downloadable under some licence. It is not the same as open source: the training data, training code and licence restrictions usually remain closed, and "open source model" is widely misused for this.
• Pretraining — the first and most expensive training stage, where the model learns next-token prediction over a very large corpus. It fixes the knowledge cutoff and most of the model's raw capability.
• Post-training — everything done after pretraining to make a model useful and safe: supervised fine-tuning on demonstrations, preference optimisation such as RLHF or DPO, and reinforcement learning on tasks. Differences between two models on the same base are usually post-training differences.
• RLVR — reinforcement learning from verifiable rewards: post-training on tasks whose answers can be checked automatically, such as unit tests passing or a maths answer matching. It is the main engine behind the jump in coding and reasoning scores since 2024.
• Mixture-of-experts (MoE) — an architecture where each token is routed through a few of many parallel expert sub-networks, so total parameters greatly exceed the parameters used per token. Quoting the total count as if it were a dense model is the standard misuse; the number that predicts serving cost and speed is the active parameter count.
• Distillation — training a smaller student model on a larger teacher model's outputs. Vendors use it to ship cheap fast models in a family; note that a distilled model carries the teacher's style far better than its reliability.
• Fine-tuning — further training a model's weights on your own examples. It is not the fix for missing facts (that is retrieval) and not the fix for wrong format (that is structured outputs); it is for teaching consistent behaviour, tone or a task-specific mapping that prompting cannot hold.
• LoRA — low-rank adaptation: fine-tuning that freezes the base weights and trains small low-rank matrices alongside them. The result is a few megabytes rather than a full model copy, so many adapters can be hot-swapped on one served base model.
• Quantisation — storing weights and activations at lower numeric precision (FP8, INT8, INT4) to cut memory and increase throughput. Two hosts serving "the same" open-weight model can differ measurably in quality because one quantised more aggressively; the model name alone does not tell you what you are getting.
• Knowledge cutoff — the date after which the training data contains nothing, so the model has no reliable knowledge of later events. It is not the release date, and a model released in 2026 routinely has a cutoff many months earlier.
• Model snapshot — a dated, immutable model id (for example -2026-05-14) as opposed to a floating alias that silently points at whatever is current. Pin snapshots in production; aliases move under you and change output.
• Deprecation — a vendor's announcement that a model id will stop serving on a stated date, after which requests fail rather than fall back. Deprecation notices are the single most common cause of an application that worked for a year suddenly returning errors.

Tokens, context and cost

• Token — the unit a model reads and writes, produced by the tokeniser: roughly a short word or word fragment in English, and far less efficient for code, non-Latin scripts and long numbers. The "one token ≈ 0.75 words" rule of thumb is English prose only and misleads badly on JSON and CJK text.
• Tokeniser — the model-specific component that maps text to integer token ids, usually by byte-pair encoding. Token counts are not portable between providers, so the same prompt costs a different number of tokens on each.
• Context window — the maximum number of tokens the model can attend to in one request, counting the system prompt, all prior turns, tool definitions, tool results and the tokens it generates. It is a total budget, not an input allowance, and most vendors cap output separately and much lower.
• Max output tokens — the ceiling you set on generated tokens for a single response, called max_tokens, max_completion_tokens or maxOutputTokens depending on the provider. Hitting it truncates mid-sentence and is reported as a stop reason, not an error.
• Blended price — a single cost figure combining the separately metered input and output prices at an assumed traffic ratio, commonly 3:1 input to output. Output is almost always several times dearer, so ranking on input price alone flatters models with cheap input.
• Prompt caching — server-side reuse of the computed state for a repeated prompt prefix, so a long unchanged system prompt or document is not reprocessed on every call. Reads are billed at a steep discount and cache writes sometimes at a premium; entries expire in minutes, and any change to the prefix invalidates everything after it.
• Reasoning tokens — tokens a reasoning model generates while working before it produces the visible answer, also called thinking tokens. They are billed as output tokens and count against the context window even when the provider does not return them, which is why reasoning models cost more per answer than their per-token price suggests.
• Batch API — an asynchronous endpoint that accepts many requests and returns results within a window, typically 24 hours, at around half price. The right default for evaluation runs, backfills and any workload with no user waiting on it.
• Rate limits — per-key ceilings expressed as requests per minute and tokens per minute, sometimes with a separate output-token limit. Exceeding one returns HTTP 429 with a retry hint; production clients need exponential backoff with jitter, not a fixed retry.
• Context rot — the observed decline in accuracy and instruction-following as the used portion of a context window grows, well before the advertised limit is reached. A million-token window is a capacity claim, not a promise that attention at 900k tokens matches attention at 10k.
• Compaction — replacing an accumulated conversation or agent trajectory with a model-written summary plus a few retained artefacts when it approaches the context limit, so the session can continue. Cheaper than restarting and lossier than it looks: whatever the summariser omitted is gone.

Prompting and reasoning

• System prompt — the instruction block that sets role, constraints and output format, supplied separately from user turns and given more weight by the model. It is the natural cache prefix, so keeping it stable is both a quality and a cost decision.
• In-context learning — the model adapting its behaviour from examples placed in the prompt, with no weight change. "Few-shot" means supplying a handful of such examples; "zero-shot" means describing the task and supplying none.
• Chain of thought — prompting or training a model to produce intermediate steps before an answer. On modern reasoning models, asking for step-by-step working in the prompt is largely redundant and sometimes harmful, since the model already reasons in dedicated tokens.
• Reasoning model — a model post-trained to spend variable compute on internal reasoning before answering, trading latency and cost for accuracy on maths, code and multi-step planning. On short lookup or formatting tasks it is slower and dearer for no benefit.
• Effort level — a coarse control over how much reasoning a model does, exposed as values such as none, low, medium and high, or as an adaptive setting that lets the model decide. It is the main latency-versus-accuracy dial on current APIs.
• Thinking budget — an explicit token cap on reasoning for a request, where a provider offers one instead of, or alongside, effort levels. Setting it too low on a hard problem produces a confident, unreasoned answer rather than a refusal.
• Temperature — a sampling parameter that flattens or sharpens the model's output distribution; 0 is near-deterministic, higher values sample more unusual tokens. It is not a creativity dial and it is not a correctness dial, and on reasoning models several providers ignore or restrict it.
• Structured outputs — constrained decoding against a supplied JSON Schema so the response is guaranteed to parse and to match the schema. Distinct from function calling, which is about the model choosing a tool; structured outputs are about the shape of ordinary output.
• Assistant prefill — supplying the opening tokens of the assistant's reply so the model continues from them, used to force a format or skip a preamble. Supported by some providers and not others, and incompatible with some reasoning modes.

Tool use and agents

• Function calling — the model returning a structured request to invoke a named tool with arguments, instead of prose. The API does not execute anything: your code runs the tool, and the result goes back as another message.
• Tool definition — a name, description and JSON Schema for the arguments, supplied with the request. Descriptions are prompt text the model reads for every call, so they cost tokens and they are where tool-selection accuracy is won or lost.
• Tool loop — the cycle of model turn, tool call, tool result, model turn, repeated until the model answers without calling a tool. Every iteration resends the whole conversation, so cost grows quadratically with loop length unless prompt caching is in play.
• Parallel tool calls — the model emitting several independent tool calls in one turn, to be executed concurrently and returned together. Cuts wall-clock time; your code must be prepared for results arriving in any order and for the model calling the same tool twice.
• MCP — the Model Context Protocol: an open JSON-RPC protocol by which a host application connects to servers that expose tools, resources and prompts, over stdio or streamable HTTP. It standardises how a tool is offered to a model, not how models talk to each other; that is a different class of protocol.
• MCP server — a process implementing MCP that publishes a set of capabilities, for example a database, an issue tracker or a filesystem. A third-party server is untrusted code with a channel into your model's context, and should be scoped and reviewed as such.
• Agent — a system where a model chooses its own sequence of tool calls to reach a goal, rather than following a fixed script. The word is applied to almost anything with an LLM in it; the meaningful distinction is whether control flow is decided by the model or by your code.
• Harness — the surrounding program that runs an agent's loop: assembling context, executing tools, enforcing limits and deciding when to stop. Most observed differences between "agents" using the same model are harness differences.
• Computer use — a tool interface that gives the model screenshots and returns mouse and keyboard actions, letting it drive software with no API. Slow, token-heavy and error-prone relative to a real API; a last resort, not a default.
• Human in the loop — an explicit approval step before an agent takes a consequential or irreversible action. The practical unit of agent safety, and worth designing before autonomy is increased rather than after.

Retrieval and embeddings

• Embedding — a fixed-length vector representing a piece of text (or an image) such that semantically similar inputs land near each other. Vectors from different models, or different versions of one model, are not comparable, so changing embedding model means re-embedding the entire corpus.
• Chunking — splitting documents into passages small enough to embed and to fit usefully in a prompt. Chunk boundaries decide what can ever be retrieved together, and are usually the highest-leverage thing to tune in a retrieval system.
• Vector database — a store that indexes embeddings for approximate nearest-neighbour search, with metadata filtering. A dedicated one is not a requirement: Postgres with pgvector, or an existing search engine, is sufficient well past the scale most applications reach.
• Cosine similarity — the standard score for comparing embeddings, the cosine of the angle between two vectors. On normalised vectors it ranks identically to dot product, and absolute values are meaningless across models — only the ordering within one model matters.
• HNSW — hierarchical navigable small world: the dominant ANN index, a layered proximity graph traversed greedily from a sparse top layer down. M controls graph degree and build cost, ef_search trades query latency for recall.
• Hybrid search — combining lexical scoring (usually BM25) with vector similarity and fusing the two rankings, commonly by reciprocal rank fusion. It exists because embeddings are poor at exact identifiers, rare product codes and negation, which keyword search handles trivially.
• Reranking — rescoring the top candidates from a first-stage retrieval with a cross-encoder that reads query and passage together. Far more accurate than embedding similarity and far too slow to run over a whole corpus, which is why it runs second.
• Recall@k — the fraction of relevant documents that appear in the top k retrieved. The metric to measure before blaming the model: if the answer was never retrieved, no prompt change will produce it.
• RAG — retrieval-augmented generation: retrieving relevant text at query time and placing it in the prompt so the model answers from it. Now routinely used to mean any retrieval near an LLM, including agentic search where the model issues its own queries rather than being handed a fixed set of chunks.
• Grounding — constraining an answer to supplied source material, and usually requiring citations back to it. A grounded answer can still be wrong about what the source says; citations make that checkable, not impossible.

Serving and performance

• Prefill — processing the input tokens to build the model's internal state before the first output token. Compute-bound, parallel across tokens, and the phase prompt caching removes.
• Decode — generating output tokens one at a time, each pass reading the whole model and cache. Memory-bandwidth-bound, inherently sequential, and the reason output tokens cost more than input tokens.
• KV cache — the stored key and value tensors for every previous token, so each new token attends without recomputing the past. Its size grows with sequence length and concurrency and it dominates GPU memory at long context, which is why long-context serving is expensive.
• Time to first token (TTFT) — elapsed time from sending the request to receiving the first streamed token, covering queueing, network and prefill. The number users perceive as responsiveness, and the one prompt caching improves most.
• Tokens per second (TPS) — output token generation rate. Vendor figures are usually single-stream at low load; aggregate throughput across concurrent requests is a different and much larger number, and the two are frequently quoted interchangeably.
• Streaming — returning tokens incrementally as they are generated, typically as server-sent events. It changes perceived latency, not total latency, and complicates error handling because a request can fail after a successful HTTP status.
• Continuous batching — a scheduler that admits and retires sequences at each decode step rather than running fixed batches to completion. The main reason a busy inference server has far better throughput than naive per-request serving.
• Speculative decoding — a small draft model proposes several tokens which the target model verifies in a single pass, keeping the accepted prefix. It speeds up decoding without changing the output distribution, so it is a latency optimisation rather than a quality trade.
• Model router — a layer that decides which model serves each request, by classifier, rule or price and latency policy. It may be a gateway feature you configure, or internal to a single vendor model id that fronts several backing models — which makes output non-reproducible unless the vendor lets you pin.
• Provisioned throughput — reserved capacity billed by time rather than per token, giving predictable latency and no shared-pool rate limits. Worth it only above a utilisation level that is easy to overestimate.

Evaluation

• Eval — an automated test that scores model output on a fixed set of inputs, run in CI the way unit tests are. The unit of engineering discipline in LLM work: without one, a prompt change is a guess.
• Golden set — the curated inputs and expected outputs an eval runs against, ideally drawn from real traffic and including the failures that motivated each fix. It is a regression suite, so items get added when something breaks and are almost never removed.
• Benchmark — a published shared test set used to compare models, such as GPQA, MMLU-Pro or SWE-bench Verified. Benchmark scores predict very little about your task; they are a coarse filter before your own evals, not a substitute for them.
• Agentic benchmark — a benchmark scoring multi-step tool use in an environment on whether the end state is correct, rather than comparing text. SWE-bench Verified, τ-bench, OSWorld and WebArena are the common examples, and results depend heavily on the harness as well as the model.
• Contamination — benchmark data having appeared in training, inflating scores without any real capability gain. The reason held-out, private and freshly written evaluations are worth more than a leaderboard position.
• LLM-as-judge — using a model to grade outputs against a rubric or to compare two candidates. Cheap and correlates reasonably with human judgement when the rubric is specific, but carries known biases toward longer answers, toward the first option presented, and toward its own family's outputs.
• Pass@k — the probability that at least one of k sampled attempts is correct. A pass@10 figure describes a system that can retry and verify; comparing it against someone else's pass@1 is meaningless.
• Hallucination — a fluent, confidently stated output that is not supported by the input or by fact. Not a bug to be patched but a property of sampling from a probability distribution; it is reduced by grounding, verification and calibration, never eliminated by instructions not to do it.

Safety, privacy and governance

• Guardrails — the checks around a model rather than inside it: input and output classifiers, schema validation, allowlists, tool permissions and rate limits. A useful engineering category and a heavily abused marketing word; ask which specific checks run, where, and what happens when one fires.
• Moderation — classifying content against a policy taxonomy, usually via a dedicated cheap endpoint, before or after the main model call. Separate from the main model's own refusal behaviour and configurable independently.
• Jailbreak — a prompt from the user that induces the model to ignore its policy or system instructions. Contrast with prompt injection, where the attacker is not the user.
• Prompt injection — content that the model reads as instructions when it was supposed to be data. The defining vulnerability of tool-using systems and unsolved in the general case; it is mitigated by limiting what tools can do, not by telling the model to ignore instructions in documents.
• Indirect prompt injection — prompt injection delivered through material the model fetches itself: a web page, a document, a code comment, an MCP server response. Any agent with both untrusted input and consequential tools is exposed, and combining retrieval, private data and outbound network access is the dangerous configuration.
• Zero data retention (ZDR) — a contractual and technical mode in which prompts and outputs are not persisted after the response is returned. Read what it excludes: it usually disables abuse-monitoring logs, and it can rule out features that require storage, such as prompt caching, batch retrieval or server-side conversation state.
• Training on customer data — whether a provider may use your API inputs and outputs to train models. Business and enterprise API tiers now default to no; consumer chat products often default to yes, and the two are frequently confused when someone says "we use their model".
• Data residency — a commitment that data is processed and stored in a named region. Distinct from retention: data can be short-lived and still cross a border, and vice versa.
• Over-refusal — the model declining a benign request because it superficially resembles a prohibited one. The standard failure mode of aggressive safety tuning, and worth measuring in your evals alongside genuine unsafe completions.
• EU AI Act — the EU's risk-tiered regulation of AI systems, with obligations for general-purpose model providers already in force and high-risk system requirements phasing in through 2026 and 2027. If you deploy rather than build models you are a "deployer", with lighter but real duties around transparency, human oversight and logging.

Media models

• Multimodal — a model that accepts more than one input type, most often text and images. It rarely means the model can produce more than text; input and output modalities are advertised separately and should be checked separately.
• Image tokens — the unit images are billed in, derived from resolution by tiling the image and charging per tile. A handful of screenshots can outweigh a long text prompt, which is what makes vision-driven agent loops expensive.
• Diffusion model — an image or video generator trained to reverse a noising process, denoising from random noise toward a sample conditioned on the prompt. Step count trades quality for latency; most current systems operate in a compressed latent space rather than on pixels.
• Seed — the random number initialising generation. Fixing it makes a generation reproducible for a given model version and parameter set, and is the only practical way to iterate on a prompt while holding the image roughly constant.
• Text-to-video — generating a clip from a prompt, optionally conditioned on a starting image. Priced per second of output rather than per token, and constrained by clip length, resolution and how well identity and scene stay consistent across frames.
• ASR — automatic speech recognition, or speech-to-text: audio in, transcript out, usually with timestamps. Measured by word error rate, which varies sharply with accent, domain vocabulary and background noise.
• TTS — text-to-speech: synthesising audio from text in a selected or cloned voice, billed per character or per second. Voice cloning from a real person's recording carries consent and likeness obligations independent of the model's own terms.
• Speech-to-speech — a realtime API that takes audio in and returns audio out through one model, without a transcribe-then-generate pipeline. Lower latency and preserves tone and interruption handling; harder to log, moderate and evaluate because there is no text stage to inspect.
• C2PA — the provenance standard that attaches signed metadata recording how a piece of media was made or edited. Widely applied to generated images by major providers, and easily stripped, so it evidences provenance when present rather than proving absence of generation when missing.