Open weights or a hosted API

Three options, not two

Framing this as "run it ourselves or pay per token" skips the option most teams land on: open weights on someone else's serverless inference platform.

1. Hosted API from a model lab. Per-token pricing, no capacity to manage, access to the frontier tier.
2. Self-hosted open weights. Your GPUs (owned, reserved, or on-demand), your serving stack (vLLM, SGLang, TensorRT-LLM), your on-call rota.
3. Open weights on a serverless inference platform. Per-token pricing for open models, with dedicated-capacity options when you need them.

Options 1 and 3 share a cost shape. Option 2 has a different one, and that difference — not the headline price — is what the decision turns on.

Utilisation is the whole game

Per-token pricing is a pure variable cost: it goes to zero at 03:00 when nobody is using your product. A GPU reservation does not. Convert your GPU spend into a per-token figure before comparing anything:

cost per 1M tokens = node_$_per_hour ÷ (achieved_tokens_per_sec × 3600 ÷ 1e6)

achieved_tokens_per_sec = peak_throughput × utilisation

Both terms in the denominator are where people go wrong.

Peak throughput assumes saturation. Decode is memory-bandwidth bound: at batch size 1 you stream the entire weight matrix out of HBM to produce a single token, wasting nearly all the arithmetic capacity. Continuous batching amortises that read across concurrent sequences, so aggregate throughput on a well-fed server sits far above single-stream throughput — often an order of magnitude or more, until KV-cache capacity or bandwidth caps the batch. A server running a handful of concurrent requests therefore costs many times more per token than the same hardware at a full batch.

Utilisation is rarely what you assume. Product traffic is diurnal and spiky; you provision for p99 concurrency and pay for it at 04:00. Without global traffic or a batch workload to backfill the troughs, a first deployment commonly averages well under a third of what it provisions. Divide the quoted GPU price by measured utilisation, never by hoped-for utilisation.

Then add the costs that never reach the spreadsheet: serving-stack upgrades, quantisation and its regression testing, autoscaling, spare capacity for node failure and rolling upgrades, an eval harness to catch quality drift, observability, and on-call. This is sustained attention, not a project with an end date. One fully loaded infrastructure engineer costs more per year than a great deal of token spend — run that number before assuming otherwise.

Break-even, stated honestly

Break-even is not "a number of tokens". It is a surface with three axes: volume, utilisation, and the capability tier you actually need. For a mid-size open model on reserved capacity at high utilisation, the crossover generally lands in the billions of tokens per month — treat that as an order of magnitude, not a figure to quote. At low utilisation, the same workload may never cross at all.

Two corrections before you trust your own arithmetic:

• Compare like for like. Benchmarking a small open model on your GPUs against a frontier hosted model is not a cost comparison; it is a quality decision with a cost side-effect. Establish equivalence with compare models first, then run the cost calculator on your real input/output mix.
• Get the token mix right. Hosted providers bill output tokens at a multiple of input tokens — several times over, and the ratio differs by provider (current rates). Self-hosting inverts the intuition: prefill is a parallel, compute-bound pass, while decode is sequential and bandwidth-bound. Long-input/short-output workloads (classification, extraction, reranking, RAG with short answers) self-host far better than agent loops that emit thousands of tokens per request. Prompt caching skews it further — hosted cache reads bill at a fraction of base input, from roughly half down to a tenth depending on the provider, and cache writes usually carry a small premium. If your traffic shares a large system prompt, that discount can wipe out a self-hosting advantage on its own.

Decision table

Licences: open weights is not open source

Weights published under a permissive-sounding name are frequently not OSI-open. Read the licence file before the model reaches production, and check four things:

• Field-of-use and scale triggers. Some community licences flip to a negotiated commercial licence above a user or revenue threshold, measured at a fixed date. If you are near it, or plan to be, get legal sign-off now rather than after launch.
• Acceptable-use policies incorporated by reference. The AUP is often a separate document the licensor can revise unilaterally, and it binds your downstream users too.
• Naming and attribution obligations. Several licences require product-level attribution and require derivative models to carry the family name.
• Non-production and research-only tiers. Some labs ship a permissive licence for the small models in a family and a research or non-commercial licence for the large one. Same brand, different rights.

Separately: "open weights" means the weights — not the training data, not the data pipeline, and rarely the training code. You get reproducible inference, not reproducible training.

Compliance is usually the real driver

Most genuine self-hosting mandates come from data governance, not cost. Legitimate drivers: an air-gapped or classified environment, a regulator that requires data never leave a specific jurisdiction or your own VPC, contractual customer commitments you cannot subcontract, or an inability to accept any third-party processor.

Everything short of that deserves a check first. Hosted providers commonly offer regional processing, zero-retention arrangements, no-training-on-your-data terms, and sector-specific agreements. "Compliance requires us to self-host" is frequently an untested assumption that costs a team six months. Name the specific clause you cannot satisfy contractually — if there isn't one, the constraint is preference, not compliance.

Latency and cold starts

Self-hosting buys you control of the tail: no queueing behind other tenants, and the ability to trade throughput for time-to-first-token deliberately. It also buys you responsibility for the tail.

Serverless open-weight platforms invert this. Scale-to-zero eliminates idle cost, but the first request after a scale-up pays to load tens of gigabytes of weights into VRAM — seconds to tens of seconds, depending on model size and how the platform caches. Steady traffic hides this. Bursty traffic under a latency SLO does not, and the fix is a warm minimum replica count, which reintroduces idle cost. Measure the cold path, not just the warm one.

Fine-tuning and control

If you need more than supervised fine-tuning on chat data, open weights are the only route. Continued pretraining, distillation from a larger teacher, custom tokenisers, logit access, constrained decoding against a grammar, custom sampling, and speculative decoding with a matched draft model are all off the table on most hosted APIs.

Be clear about what you are buying. A LoRA adapter over a mid-size open model can beat a frontier model on a narrow, well-specified task given a good dataset. It will not close a general reasoning gap, and it creates a permanent obligation: every base-model upgrade means retraining and re-evaluating. Fine-tuning is a commitment to a maintenance cycle, not a one-time cost.

What people get wrong

• Comparing a saturated benchmark GPU to real traffic. The vendor's throughput figure comes from maximum batch on a synthetic load. Yours will not.
• Comparing a small open model to a frontier hosted model. The most common error by far. It makes self-hosting look wildly cheaper when it is really a downgrade with a discount.
• Forgetting the KV cache in VRAM sizing. The weights fit; then long contexts at real concurrency do not, effective batch size collapses, and cost per token climbs with it.
• Assuming quantisation is free. 8-bit is usually close to lossless on general tasks; aggressive 4-bit can degrade long-context, multilingual, and structured-output behaviour in ways general benchmarks miss. Run your own evals.
• Treating a serverless open-model endpoint as a compliance win. It is still a third-party processor. If your constraint was data residency, this does not solve it.
• Ignoring hosted deprecation risk. Providers retire model versions on their own schedule. If you need multi-year behavioural stability for an audited system, that is a genuine argument for holding the weights.
• Building the platform before proving the product. Start hosted, instrument token volume and mix, revisit when the bill is a real line item.

A procedure

Ship on a hosted API. Instrument tokens by input and output, cache-hit rate, concurrency, and time-of-day distribution. Once monthly spend is material, price the same capability tier three ways from the model index and the cost calculator: hosted frontier, open weights serverless, and open weights self-hosted at your measured utilisation with headcount included. Move to serverless open weights when the capability tier permits it, and to your own GPUs only when utilisation, licence terms, and a hard compliance constraint all point the same way.