vLLM v0.28.0 (released 2026-08-26) is the de facto reference engine for self-hosted open-weight serving: continuous batching, chunked prefill, PagedAttention, prefix caching, and speculative decoding via n-gram, suffix, EAGLE and DFlash. It serves an OpenAI-compatible HTTP API plus Anthropic Messages and gRPC endpoints, and supports tensor, pipeline, data, expert and context parallelism. Against SGLang, its differentiator is breadth rather than peak throughput — the widest model coverage, the widest hardware coverage (NVIDIA, AMD, x86/ARM/PowerPC CPUs, with plugins for Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX), and the largest contributor base. That is why Dynamo, llm-d, KServe, Ray Serve and BentoML all target it as their first backend.
| Hardware | NVIDIA, AMD, x86/ARM/PowerPC CPU; plugins for TPU, Gaudi, Ascend, Spyre, Apple Silicon, MetaX |
|---|
| Continuous batching + paged KV | Yes — continuous batching, chunked prefill, PagedAttention |
|---|
| Prefix caching | Yes, automatic prefix caching (on by default) |
|---|
| Tensor / pipeline parallelism | Tensor, pipeline, data, expert and context parallel |
|---|
| Speculative decoding | Yes — n-gram, suffix, EAGLE, DFlash |
|---|
Watch out: Still 0.x with no LTS and a roughly monthly cadence; engine internals (V1 engine, attention backends, scheduler) change between minor versions, so pinning a container digest and re-benchmarking on every upgrade is mandatory rather than optional. Non-NVIDIA backends are plugins with uneven kernel and quantization coverage and consistently lag the CUDA path. It is single-node-first: multi-node disaggregated prefill/decode and KV-aware routing require Dynamo, llm-d or the production-stack on top.
Free. Apache-2.0; your only cost is GPU/infrastructure.
· open source
SGLang v0.5.19 (2026-09-05) is the main performance-oriented alternative to vLLM, built around RadixAttention — a radix-tree KV cache that reuses prefixes across requests rather than only within a session, which is why it wins on multi-turn agents, few-shot prompts and shared system prompts. It supports continuous batching, speculative decoding, prefill-decode disaggregation, and tensor, pipeline, expert and data parallelism, with wide expert parallelism used for DeepSeek- and Kimi-class MoE models. Documentation claims deployments across more than 400,000 GPUs generating trillions of tokens per day. Hardware coverage per the docs: "Native support across Hardware Platforms including NVIDIA, AMD, Intel Xeon, Google TPU, and Ascend NPU accelerators."
| Hardware | NVIDIA, AMD, Intel Xeon, Google TPU, Ascend NPU |
|---|
| Continuous batching + paged KV | Yes — continuous batching with paged/radix-managed KV |
|---|
| Prefix caching | Yes — RadixAttention cross-request prefix cache (the headline feature) |
|---|
| Tensor / pipeline parallelism | Tensor, pipeline, expert and data parallel; prefill/decode disaggregation |
|---|
| Speculative decoding | Yes — EAGLE-family and MTP draft methods |
|---|
Watch out: Model coverage is narrower than vLLM's for long-tail and non-mainstream architectures, so a new open-weight release may land in vLLM days or weeks earlier. Getting the advertised throughput on wide expert parallelism or PD disaggregation is genuine expert tuning work, not a flag. AMD and non-NVIDIA paths have more sharp edges than the CUDA path, and the docs domain moved from docs.sglang.ai to docs.sglang.io, so older bookmarks and blog links break.
Free. Apache-2.0; your only cost is GPU/infrastructure.
· open source
TensorRT-LLM has been re-architected onto PyTorch with a high-level Python LLM API, moving away from the ahead-of-time engine-compilation workflow that made earlier versions painful to operate. It ships trtllm-serve for an OpenAI-compatible endpoint, and provides in-flight batching, paged KV cache with block reuse, EAGLE/MTP/NGram speculative decoding, and tensor, pipeline and expert parallelism across GPUs and nodes. Its reason to exist versus vLLM is quantization and kernel depth on NVIDIA's newest silicon: automatic FP8 on H100, native FP4/NVFP4 on B200. If you are NVIDIA-only and buying the newest GPUs, this is typically the fastest single-engine option.
| Hardware | NVIDIA GPUs only (best on H100/H200/B200) |
|---|
| Continuous batching + paged KV | Yes — in-flight batching, paged KV cache with block reuse |
|---|
| Prefix caching | Yes, via paged KV block reuse |
|---|
| Tensor / pipeline parallelism | Tensor, pipeline and expert parallel across GPUs and nodes |
|---|
| Speculative decoding | Yes — EAGLE, MTP, NGram |
|---|
Watch out: NVIDIA GPUs only — there is no portability story at all. The latest stable tag is v1.2.1 from 2026-04-20 while active development ships as weekly release candidates (v1.3.0rc25 as of 2026-08-31), so in practice production users are running RCs or NVIDIA's containers rather than a stable release. Kernel and quantization coverage is best on the newest data-center parts and thins out on older GPUs, and the PyTorch-backend rewrite still trails vLLM/SGLang on long-tail model coverage.
Free. Source is open on GitHub; your only cost is GPU/infrastructure. (Note: packaged as NIM containers, production use carries an NVIDIA AI Enterprise entitlement — see the NIM entry.)
· open source
Dynamo v1.4.2 (2026-08-29) is explicitly not an engine — its README states it "doesn't replace SGLang, TensorRT-LLM, or vLLM, it turns them into a coordinated multi-node inference system." Written in Rust with Python extensibility, it adds disaggregated prefill/decode (scale prefill and decode workers independently), KV-aware request routing to avoid redundant prefill, a multi-tier KV cache block manager (KVBM) that offloads to CPU and disk, and an SLA-based planner that autoscales to hit latency targets at minimum TCO. Disaggregated serving, KV-aware routing, the SLA planner and multimodal are supported across all three backends; KVBM is GA on TensorRT-LLM and vLLM and still in progress on SGLang.
| Hardware | Inherited from backend engine (NVIDIA-centric in practice) |
|---|
| Continuous batching + paged KV | Delegated to backend (vLLM, SGLang, TensorRT-LLM) |
|---|
| Prefix caching | Yes — KV-aware routing plus multi-tier KV offload to CPU/disk (KVBM) |
|---|
| Tensor / pipeline parallelism | Delegated to engine; Dynamo adds multi-node prefill/decode disaggregation |
|---|
| Speculative decoding | Delegated to backend engine |
|---|
Watch out: Its own README says it plainly: "If you're running a single model on a single GPU, your inference engine alone is probably sufficient." It introduces a distributed control plane (etcd/NATS plus Rust services) and a second system to operate, debug and upgrade. At v1.4.x it is young relative to Kubernetes-native alternatives, feature support is uneven across backends (KVBM on SGLang is marked in progress), and it overlaps heavily with llm-d and KServe — running two orchestrators is a common and costly mistake.
Free. Apache-2.0; your only cost is GPU/infrastructure.
· open source
llm-d v0.9.0 (2026-08-17) is a Kubernetes-native distributed serving stack that sits above vLLM and SGLang, and became a CNCF sandbox project in March 2026. It provides prefix-cache- and load-aware request routing (with experimental predicted-latency scheduling), tiered KV offload to CPU and disk with a global cache index, prefill/decode disaggregation, wide expert parallelism, SLO-aware autoscaling with scale-to-zero, and OpenAI-compatible batch APIs. It ships "well-lit path" Helm/kustomize recipes with published benchmarks — e.g. 3x output throughput and 2x faster TTFT from prefix-cache-aware routing vs round-robin on 4x AMD MI300X, and 13.9x throughput with hierarchical KV offloading at 250 concurrent users on 4x H100. Compared with Dynamo it is vendor-neutral and Kubernetes-first rather than a standalone Rust control plane.
| Hardware | Accelerator-neutral — tested on NVIDIA, AMD MI300X, Intel XPU, Google TPU |
|---|
| Continuous batching + paged KV | Delegated to vLLM / SGLang |
|---|
| Prefix caching | Yes — prefix-cache-aware routing plus tiered CPU/disk offload with global index |
|---|
| Tensor / pipeline parallelism | Delegated to engine; llm-d adds prefill/decode disaggregation and wide expert parallelism |
|---|
| Speculative decoding | Delegated to backend engine |
|---|
Watch out: Kubernetes-only, and it assumes Gateway API plus the inference gateway extension — there is no non-K8s deployment path, so a bare-metal or docker-compose shop cannot use it. Every headline performance number is vendor-published on a favourable prompt shape; reproduce on your own traffic before budgeting for it. It occupies the same slot as Dynamo and KServe's generative path, and at v0.x the guides shift between Helm and kustomize between releases.
Free. Apache-2.0; your only cost is GPU/infrastructure.
· open source
Triton v2.72.0 (2026-08-31) is NVIDIA's general-purpose inference server: one process, one model repository, and pluggable backends for TensorRT, ONNX Runtime, PyTorch, Python, FIL, DALI, plus vLLM and TensorRT-LLM backends for generative workloads. Its value is uniformity across model types — if the same cluster serves an LLM, an embedding model, a reranker, a CV model and a scikit-learn/XGBoost model, Triton gives them one deployment, metrics and versioning story. For LLM-only shops it is a layer above the engine rather than a replacement for one, and NVIDIA's newer datacenter-scale LLM investment is going into Dynamo.
| Hardware | NVIDIA GPU and CPU (x86, ARM); backend-dependent |
|---|
| Continuous batching + paged KV | Dynamic batching natively; continuous batching + paged KV via vLLM/TRT-LLM backends |
|---|
| Prefix caching | Only via the vLLM or TensorRT-LLM backend |
|---|
| Tensor / pipeline parallelism | Via TensorRT-LLM / vLLM backends; Triton adds multi-instance and ensemble scheduling |
|---|
| Speculative decoding | Only via the backend engine |
|---|
Watch out: For pure LLM serving it adds configuration surface (model repository layout, config.pbtxt, backend tuning) without adding LLM-specific capability — the batching and KV cache still come from the vLLM or TensorRT-LLM backend underneath, and there is no KV-aware routing or disaggregated serving. Documentation and examples skew heavily to NVIDIA hardware. NVIDIA's own LLM roadmap energy has visibly shifted to Dynamo, so treat Triton as the choice for multi-framework breadth, not for frontier LLM throughput.
Free. BSD-3-Clause; your only cost is GPU/infrastructure. (Enterprise support is sold via NVIDIA AI Enterprise — price not published.)
· open source
KServe v0.20.0 (2026-08-06) is a CNCF incubating project that defines an InferenceService custom resource covering both generative and predictive AI. On the generative side it supports vLLM and llm-d as optimized backends, an OpenAI-compatible inference protocol, model caching, KV cache offloading to CPU/disk, and request-based autoscaling. On the predictive side it keeps the older strengths — TensorFlow, PyTorch, scikit-learn, XGBoost, ONNX, canary rollouts, InferenceGraph pipelines, explainability, drift and outlier detection, and scale-to-zero. It is the choice when your platform team wants one Kubernetes API for every model an organization deploys, not just the LLMs.
| Hardware | Whatever the runtime supports (NVIDIA, AMD via vLLM/llm-d); CPU for predictive models |
|---|
| Continuous batching + paged KV | Delegated to vLLM / llm-d runtime |
|---|
| Prefix caching | Via runtime; KServe adds KV cache offloading to CPU/disk and model caching |
|---|
| Tensor / pipeline parallelism | Delegated to runtime; KServe handles replica and multi-node scheduling |
|---|
| Speculative decoding | Delegated to backend engine |
|---|
Watch out: Heavy: the serverless install pulls in Knative (and typically Istio) plus cert-manager and operators, and the raw-Kubernetes install drops canary and scale-to-zero. Scale-to-zero is a trap for large LLMs — cold start means pulling and loading tens or hundreds of GB of weights, so first-token latency after a scale-up is minutes, not seconds. Its generative features track upstream vLLM and llm-d with a lag, so bleeding-edge engine features arrive here last.
Free. Apache-2.0; your only cost is GPU/infrastructure.
· open source
Ray Serve LLM (part of Ray 2.58.0, released 2026-08-23) exposes vLLM through Ray's deployment model: you declare LLMConfig objects with vLLM engine kwargs and an autoscaling_config of min_replicas/max_replicas, and get an OpenAI-compatible ingress. Multiple models are served by passing a list of LLMConfigs to one OpenAiIngress deployment, and it supports multi-LoRA deployment, prefix-aware routing, prefill/decode disaggregation, KV cache offloading and data-parallel attention. Its real differentiator is composition in Python — chaining retrieval, a reranker, business logic and the LLM in one autoscaled application, on the same cluster that runs your batch inference and training.
| Hardware | Whatever vLLM supports (NVIDIA, AMD, TPU); Ray schedules heterogeneous resources |
|---|
| Continuous batching + paged KV | Delegated to vLLM |
|---|
| Prefix caching | Yes via vLLM, plus prefix-aware routing across replicas and KV cache offloading |
|---|
| Tensor / pipeline parallelism | vLLM TP/PP within replicas; Ray adds data-parallel attention and PD disaggregation |
|---|
| Speculative decoding | Delegated to vLLM |
|---|
Watch out: You inherit Ray cluster operations — head node, GCS, object store, worker autoscaler — which is a substantial and unfamiliar failure surface if Ray is not already in your stack. Functionally it is a thin layer over vLLM, so per-GPU throughput will not exceed vLLM's and each hop adds latency; if you only need one model behind one endpoint, run vLLM directly. Non-vLLM engines are not first-class.
Free. Ray is Apache-2.0; your only cost is GPU/infrastructure. (Anyscale's managed platform is priced separately.)
· open source
BentoML 1.4.x is a Python service framework: you write a Service class, declare dependencies and GPU requirements, and it builds a reproducible container with batching, job queues and multi-model pipelines. OpenLLM is its LLM-focused wrapper that starts an OpenAI-compatible API with a built-in chat UI in one command, using vLLM as the inference backend. The commercial Bento Inference Platform is a self-hostable control plane on top ("Self-host anywhere. Serve any model."). Pick BentoML when the deployable unit is not a bare model but a service — an LLM plus retrieval, guardrails, tokenizer quirks and custom pre/post-processing that has to ship and version as one artifact.
| Hardware | Whatever the embedded backend supports (typically NVIDIA via vLLM) |
|---|
| Continuous batching + paged KV | Adaptive request batching natively; continuous batching + paged KV via vLLM backend |
|---|
| Prefix caching | Only via the backend engine |
|---|
| Tensor / pipeline parallelism | Delegated to backend engine; BentoML handles replica scaling and pipelines |
|---|
| Speculative decoding | Only via the backend engine |
|---|
Watch out: BentoML is not an inference kernel: throughput is entirely whatever backend you embed (usually vLLM), and the framework adds Python-layer latency. OpenLLM looks less maintained than the marketing implies — its last tagged release is v0.6.30 from 2025-04-21 and its README still headlines Llama 3.3 and Qwen2.5, so verify current model support before depending on it. The commercial platform publishes no pricing at all, which makes budgeting a sales conversation.
BentoML OSS and OpenLLM are free (Apache-2.0). Bento Inference Platform pricing is unknown — bentoml.com/pricing returned a server error on 2026-09-06 and lists no figures; it is demo/sales-led.
· open source
LMDeploy v0.17.0 (2026-09-01) ships two engines: TurboMind, a C++/CUDA engine with paged attention, blocked KV cache, online INT8/INT4 KV cache quantization and W4A16 weight-only inference, and a pure-Python PyTorch engine that extends support to Huawei Ascend with graph mode. It is unusually strong on mixed-precision quantization — the team claims TurboMind MXFP4 on NVIDIA GPUs from V100 onward at 1.5x vLLM throughput on H800 for gpt-oss models, and has a EuroSys 2027 paper on the approach — and on vision-language serving for the InternVL family. It supports DeepSeek-style prefill/decode disaggregation via DLSlime and Mooncake, and exposes an OpenAI-compatible API server with tool calling.
| Hardware | NVIDIA (TurboMind, sm_75+); Huawei Ascend and others via the PyTorch engine |
|---|
| Continuous batching + paged KV | Yes — persistent/continuous batch with blocked paged KV cache |
|---|
| Prefix caching | Yes, prefix caching supported |
|---|
| Tensor / pipeline parallelism | Tensor parallel, multi-node serving; PD disaggregation via DLSlime/Mooncake |
|---|
| Speculative decoding | Limited — not a headline feature; verify per model |
|---|
Watch out: Much smaller English-language community and ecosystem than vLLM or SGLang, so troubleshooting means reading Chinese-language issues and the source. Model coverage is centered on InternLM, InternVL and Qwen; long-tail architectures land later than upstream. The two-engine split is a real operational cost — TurboMind is the fast NVIDIA path while Ascend and some newer features live in the PyTorch engine, and the two do not have identical feature sets, so switching hardware can mean losing capabilities.
Free. Apache-2.0; your only cost is GPU/infrastructure.
· open source
TGI is a Rust/Python/gRPC LLM server that powered HuggingChat and Hugging Face Inference Endpoints, with continuous batching, tensor parallelism, quantization and an OpenAI-compatible messages API. It is now archived on GitHub, and its README carries a caution: "text-generation-inference is now in maintenance mode. Going forward, we will accept pull requests for minor bug fixes, documentation improvements and lightweight maintenance tasks." Hugging Face explicitly redirects users onward: "downstream inference engines, which we contribute to and recommend using going forward: vllm, SGLang, as well as local engines with inter-compatibility such as llama.cpp or MLX." It is catalogued here so readers recognize it in old tutorials and Docker images rather than adopt it.
| Hardware | NVIDIA, AMD ROCm, Intel Gaudi, AWS Inferentia (as of last release) |
|---|
| Continuous batching + paged KV | Yes — continuous batching with paged attention |
|---|
| Prefix caching | Yes, prefix caching supported in v3.x |
|---|
| Tensor / pipeline parallelism | Tensor parallel; no active work on newer parallelism strategies |
|---|
| Speculative decoding | Yes (Medusa, n-gram) — frozen at last release |
|---|
Watch out: The repository is archived and the project accepts only minor bug fixes and documentation changes — no new model architectures, attention backends, quantization formats or speculative-decoding work will land. Any 2024-2025 tutorial that tells you to deploy TGI is stale. Existing deployments should plan a migration; the OpenAI-compatible surface makes vLLM or SGLang a mostly drop-in replacement.
Free. Apache-2.0; last release v3.3.7 on 2025-12-19.
· open source · deprecated
NIM packages a model together with a pre-tuned engine (TensorRT-LLM, vLLM or SGLang depending on the model and GPU), an OpenAI-compatible API, and NVIDIA-selected runtime configuration into a single container pulled from NGC. The pitch is that you skip the engine-selection and tuning work: NVIDIA has already picked the backend, quantization and parallelism for your specific GPU. It is the managed-experience end of self-hosting — the model still runs on your hardware, but the tuning decisions and the support contract belong to NVIDIA.
| Hardware | NVIDIA GPUs only; per-GPU optimized profiles |
|---|
| Continuous batching + paged KV | Yes — inherited from the bundled TensorRT-LLM / vLLM / SGLang engine |
|---|
| Prefix caching | Yes, via the bundled engine |
|---|
| Tensor / pipeline parallelism | Yes — preconfigured TP/PP profiles per GPU count |
|---|
| Speculative decoding | Model-dependent; configured by NVIDIA, not user-tunable |
|---|
Watch out: Pricing is entirely opaque — NVIDIA lists no figures publicly, so you cannot model cost per token without a sales conversation, which is a real problem for a category whose whole point is cost control. The containers are closed, so you cannot patch a kernel, swap an attention backend or apply a community fix. The catalog is NVIDIA-curated: day-0 support for a newly released open-weight model almost always appears in vLLM or SGLang first. NVIDIA GPUs only.
unknown — NVIDIA publishes no list price for NVIDIA AI Enterprise or NIM on nvidia.com (checked 2026-09-06). Free developer access and a 90-day trial license are offered; production entitlement is quoted by sales or bought per-GPU-hour through cloud marketplaces.
MAX (release max/v26.5.0, 2026-08-11) is an AI serving framework built on Mojo kernels rather than PyTorch, with OpenAI-compatible inference APIs and paged KV cache support. Its structural differentiator is dependency-free packaging: per the docs, "MAX doesn't depend on PyTorch, CUDA, or ROCm, so there's nothing to bundle, patch, or sync," giving containers under 700MB and materially faster cold starts than a multi-gigabyte vLLM+CUDA image — which matters for autoscaling and scale-to-zero. The same package targets NVIDIA, AMD and Apple silicon with no CUDA version matching, and the Python API, model pipelines and GPU kernels are open source.
| Hardware | NVIDIA, AMD and Apple silicon from the same package; CPU supported |
|---|
| Continuous batching + paged KV | Yes — continuous batching and paged KV cache (KVCacheStrategy.PAGED) |
|---|
| Prefix caching | Yes, via paged KV block reuse |
|---|
| Tensor / pipeline parallelism | Tensor parallel supported; multi-node coverage narrower than vLLM/SGLang |
|---|
| Speculative decoding | Supported but not a headline feature; verify per model |
|---|
Watch out: Far smaller ecosystem and model coverage than vLLM or SGLang — "hundreds of models" is a fraction of what vLLM supports, and new architectures arrive later. The headline benchmark ("171% of vLLM throughput") is vendor-published on specific hardware; reproduce it yourself. Writing custom kernels means learning Mojo, a language with a small talent pool, and the managed control plane and cloud rates are not published.
Self-hosted Community Edition is free and open source. Modular Cloud is per-token or per-GPU-hour (e.g. DeepSeek V4 at $1.74 in / $3.48 out per 1M tokens); BYOC is per-minute and Enterprise is custom — GPU-hour and per-minute rates are not published.
· open source
llama.cpp (v0.4.0, 2026-09-04, plus continuous b-numbered builds) is the reference CPU-and-anything runtime, using GGUF quantized weights and the ggml tensor library. Its llama-server binary is a genuine server, not a demo: OpenAI-compatible chat/completions, responses and embeddings routes, Anthropic Messages compatibility, a reranking endpoint, continuous batching with parallel multi-user decoding, schema-constrained JSON output, function calling, multimodal input and speculative decoding. It is the only entry here that runs well with no GPU at all, and the only one that covers Metal, Vulkan, SYCL, CUDA and ROCm from one codebase — which is why it is the substrate under Ollama, LM Studio and most consumer local-AI tooling.
| Hardware | CPU (x86/ARM), Apple Metal, NVIDIA CUDA, AMD ROCm/HIP, Vulkan, Intel SYCL |
|---|
| Continuous batching + paged KV | Continuous batching with parallel slots; slot-based KV, not paged attention |
|---|
| Prefix caching | Per-slot prompt cache reuse; no global cross-request radix cache |
|---|
| Tensor / pipeline parallelism | No true tensor parallel — layer/row split across GPUs; experimental RPC multi-node |
|---|
| Speculative decoding | Yes — draft-model speculative decoding (--model-draft) |
|---|
Watch out: Per-GPU throughput under real concurrency is well below vLLM or SGLang — there is no paged-attention-class scheduler, and multi-GPU is layer or row splitting rather than true tensor parallelism (multi-node RPC is experimental). GGUF requires a separate quantization step and lags day-0 architectures, so a new model may be days or weeks away. It uses rolling build tags rather than stable releases, and server flags and behavior change between builds, so pin an exact build. Serving a shared production endpoint on it is the classic self-hosting cost mistake.
Free. MIT license; no fees of any kind.
· open source
Ollama v0.33.3 (2026-09-02) wraps model download, quantization selection, GPU offload and serving behind `ollama run <model>`, with a Docker-Hub-style registry and Modelfiles for customization. It exposes OpenAI-compatible /v1 endpoints, so application code written against the OpenAI SDK works unchanged. It has also grown a hosted side — Ollama Cloud runs larger models than local hardware allows, on paid tiers. Its role in a self-hosting decision is as the fastest possible local evaluation loop before you commit engineering time to vLLM or SGLang, not as the production server itself.
| Hardware | CPU, NVIDIA, AMD, Apple Silicon (macOS, Linux, Windows) |
|---|
| Continuous batching + paged KV | Basic parallel request handling (OLLAMA_NUM_PARALLEL); no paged attention |
|---|
| Prefix caching | Per-session context reuse only |
|---|
| Tensor / pipeline parallelism | No — single-node multi-GPU layer offload only |
|---|
| Speculative decoding | Not exposed as a user-facing feature |
|---|
Watch out: The wrong tool for a shared production endpoint: concurrency handling is basic, there is no paged-attention-class scheduler, no tensor parallelism and no multi-node story, so tokens/sec/GPU at concurrency 32+ is a fraction of vLLM's. Its default quantization is chosen for convenience and can silently cost you quality versus the weights you thought you were serving. The paid tiers buy a hosted service, not better self-hosted capability — paying Ollama does not make your own GPUs faster.
Local use is free (MIT). Ollama Cloud: Free $0 with starter credits; Pro $20/mo or $200/yr ($60 of usage credits/mo); Max $100/mo ($300 credits/mo, 10 concurrent requests); Team $500/mo ($1,000 shared credits/mo); Enterprise custom.
· open source
LM Studio is a closed-source desktop application for macOS, Windows and Linux that handles model discovery, GGUF/MLX download, quantization choice and GPU offload through a GUI, then exposes a local OpenAI-compatible server plus an `lms` CLI and SDKs. It runs llama.cpp and Apple MLX engines underneath. Since 8 July 2025 it is free for work use with no form to fill in — "You and your team can just use LM Studio at work!" — removing the commercial-license friction it previously had. In a self-hosting workflow its job is comparing quantizations and prompt behavior interactively before you decide what to put on a server.
| Hardware | CPU, NVIDIA, AMD, Apple Silicon (MLX) on macOS/Windows/Linux desktops |
|---|
| Continuous batching + paged KV | Inherited from llama.cpp/MLX; not tuned for server concurrency |
|---|
| Prefix caching | Per-conversation context reuse only |
|---|
| Tensor / pipeline parallelism | No — single machine, multi-GPU layer offload |
|---|
| Speculative decoding | Yes — draft-model speculative decoding is exposed in the app |
|---|
Watch out: It is a desktop application, not server software: no clustering, no Kubernetes deployment, no multi-node, no headless-first operational model, and closed source so you cannot audit or patch it. Throughput ceilings are llama.cpp's, with the same concurrency limits. Treat the local server as a developer convenience on localhost; putting a laptop app in a production request path is not a supported architecture.
Free ($0) for local use, personal and commercial. Cloud is pay-as-you-go per token (e.g. DeepSeek V4 Flash at $0.13 input / $0.028 cached / $0.26 output per 1M tokens). Enterprise/Teams plans exist; "Bionic" plan pricing is listed as coming soon.
Xinference v3.3.0 (2026-08-30) is a model manager rather than an engine: one control plane that launches and supervises many models across a cluster, each on whichever backend suits it — vLLM, SGLang, llama.cpp or Transformers — all behind a single OpenAI-compatible API. Its practical value in a self-hosted RAG or agent stack is that the embedding model, the reranker, the speech model and the LLM all live behind one endpoint with one client library, instead of four separately deployed services. An Xinference Enterprise edition is sold on top of the Apache-2.0 open-source core.
| Hardware | NVIDIA, AMD, Apple Silicon and CPU, depending on the backend chosen per model |
|---|
| Continuous batching + paged KV | Delegated to backend (vLLM / SGLang / llama.cpp) |
|---|
| Prefix caching | Delegated to backend engine |
|---|
| Tensor / pipeline parallelism | Delegated to backend; Xinference schedules replicas across cluster workers |
|---|
| Speculative decoding | Delegated to backend engine |
|---|
Watch out: It is a layer over other engines, so peak per-GPU throughput is bounded by whichever backend it launches — never faster than running vLLM directly, and with an extra process and hop of latency. Documentation, issues and community skew Chinese-language, and README content lags the code. Enterprise pricing is unpublished, and the multi-backend abstraction means engine-specific tuning flags are exposed unevenly, so deep performance work usually means dropping down to the engine anyway.
Open-source edition is free (Apache-2.0). Xinference Enterprise pricing is unknown — not published; sales-led.
· open source