§6

LLM observability and evaluation

LLM observability and evaluation tools capture what an LLM app or agent actually did — the prompts, tool calls, retrieved documents, token counts, latency and cost of every span — and then let you score that behaviour, either offline against a dataset or online against live traffic. They exist because the usual software signals (exit codes, HTTP status, p99 latency) say nothing about whether an answer was correct, grounded, or safe, and because a one-line prompt change can silently degrade quality across thousands of requests.

Data checked 2026-09-06

The category has consolidated around OpenTelemetry as the wire format, which means instrumentation and backend are now largely separable. Most products bundle four things in varying proportions: tracing, evaluation (LLM-as-judge, code assertions, human review), dataset/experiment management, and prompt versioning. Pricing is the sharpest differentiator and the least comparable: vendors bill by trace, by span, by gigabyte ingested, by seat, or by some combination, and the same workload can differ 20x in cost between two tools.

A How to choose

Four axes actually separate these tools. (1) Billing unit versus your payload size — this decides cost more than anything on a feature matrix. Tools that bill per GB ingested (Braintrust, W&B Weave, Laminar, Confident AI) punish long RAG contexts and multimodal payloads; W&B Weave's $0.10/MB overage works out to roughly $100/GB, which is brutal if you trace full retrieved documents.

Tools that bill per trace or per span (Langfuse, Arize AX, Galileo, Datadog) are predictable but penalise deeply nested agent runs where one user turn is 200 spans. Estimate your average serialised trace size in KB before you look at any pricing page, and check whether the vendor lets you sample or truncate. (2) Self-host and data residency.

If traces contain regulated data or you cannot send prompts to a third party, the real shortlist is short: Langfuse (MIT core, genuinely usable self-hosted), Phoenix (ELv2, local-first), Opik and Helicone (Apache-2.0), and Logfire's Helm chart. Everyone else — LangSmith, Braintrust, Galileo, HoneyHive, PromptLayer, Traceloop, LangWatch — gates self-hosting behind an Enterprise contract, so "self-host: yes" on a comparison page usually means "yes, after a sales call". (3) Whether your centre of gravity is debugging or evaluation.

If you mostly need to see what went wrong in production, a tracing-first tool (Langfuse, Helicone, Logfire, Datadog) is enough and cheaper. If your team runs prompt/model changes through scored experiments in CI and argues about scorer quality, buy an eval-first product — Braintrust and Confident AI are built around that loop and the others treat it as a feature. (4) Who edits prompts.

Engineers versioning prompts in git do not need prompt management at all; if PMs or domain experts must change prompts without a deploy, that requirement narrows you to PromptLayer, Freeplay, LangSmith or Langfuse. Defaults worth stating plainly: for most teams shipping today, start with Langfuse — MIT, OTel-native, cloud from $29/mo, and you can move the whole thing in-house later. Choose Braintrust if evals are the product and $249/mo is noise.

Choose LangSmith mainly if you are deep in LangGraph, and price it carefully, since its LCU/LSU compute-and-storage units are hard to forecast. Choose Phoenix if you want zero vendor and are happy running a container. Choose Datadog Agent Observability (the renamed LLM Observability) only when correlating agent behaviour with the rest of your infra matters more than eval depth, and you already have the contract.

Do not start on Humanloop — it was acquired by Anthropic and is sunsetting. Whatever you pick, instrument with OpenTelemetry or OpenLLMetry rather than a vendor SDK: switching backends then costs a config change instead of a rewrite, and in this category vendors get renamed, repriced and acquired on a roughly annual cadence.

B At a glance

Name	Tracing	Evals	Prompt management	Self-host	Free tier	Pricing
Langfuse	Yes — OTel-native plus Python/JS SDKs	Yes — LLM-judge, custom scores, human annotation queues	Yes — versioned prompt registry with SDK caching	Yes — MIT, free and fully featured for core	50k units/mo cloud; unlimited self-hosted	Hobby $0 (50k units/mo, 30-day retention, 2 users); Core $29/mo; Pro $199/mo; Enterprise cloud $2,499/mo — each with 100k units included, then $8/100k units, graduating to $7/$6.50/$6 per 100k at 1M/10M/50M. Self-hosted open source free; self-hosted Enterprise is custom-priced.
LangSmith	Yes — native SDKs plus OTel ingest; LangGraph-aware views	Yes — datasets, experiments, LLM-judge, human review	Yes — prompt hub with versioning	Enterprise only (self-hosted and hybrid)	5k base traces/mo, 1 seat	Developer $0 (1 seat, up to 5k base traces/mo then pay-as-you-go); Plus $39/seat/mo (10k base traces/mo included, unlimited seats, includes one small serverless deployment); Enterprise custom. Overages billed as LCU $1.50/unit (compute) and LSU $1.00/unit (storage). Base traces have 14-day retention; 400-day extended traces cost extra.
Braintrust	Yes — spans/logs, though evals are the centre of gravity	Yes — strongest in class: datasets, scorers, experiment diffs, CI	Yes — versioned prompts and playground	Enterprise only — hybrid, data plane in your cloud	Starter: 1 GB and 10k scores/mo	Starter $0/mo ($10 model credits, 1 GB processed data, 10k scores, 14-day retention, unlimited seats); Pro $249/mo ($100 model credits, 5 GB then $3/GB, 50k scores then $1.50/1k, 30-day retention then $0.50/GB/mo); Enterprise custom. Starter overages: $4/GB and $2.50 per 1k scores. 6–12 months free for qualifying startups.
Weights & Biases Weave	Yes — Python/TS decorators, OTel ingest supported	Yes — Evaluations API, scorers, online monitors	Yes — versioned prompt objects and playground	Enterprise only for commercial use; dedicated cloud available	1 GB/mo ingestion, 5 seats	Free $0 (up to 5 Models seats, 5 GB storage, 1 GB/mo Weave ingestion, extra at $0.10/MB); Pro from $60/mo (up to 10 Models seats, 100 GB storage then $0.03/GB, 1.5 GB/mo Weave ingestion, extra at $0.10/MB); Enterprise custom with dedicated cloud or self-managed. Local self-hosted 'Personal' edition is free but non-commercial.
Arize AX	Yes — OpenTelemetry/OpenInference spans	Yes — online and offline evals, LLM-judge, experiments	Yes — prompt playground and versioning	Enterprise only	25k spans/mo, 15-day retention	AX Free $0 (25k spans/mo, 1 GB storage, 15-day retention, unlimited seats); AX Pro $50/mo (50k spans/mo, 10 GB, 30-day retention, unlimited seats); AX Enterprise custom (custom volume/retention, SaaS or self-hosted). Startup pricing available.
Arize Phoenix	Yes — OTel/OpenInference, local UI	Yes — eval library with prepackaged judge templates	Yes — playground and prompt versioning	Yes — pip, Docker, Helm; free	Unlimited self-hosted; hosted limits unknown	Self-hosted: free, no volume limits (infrastructure costs only). Hosted Phoenix Cloud free tier limits are not published on the pricing page — treat as unknown; paid production workloads are directed to Arize AX (Pro $50/mo).
Helicone	Yes — request/session logging via proxy or async	Basic — scores and online evaluators; weakest area	Yes — prompt versioning and experiments	Yes — Apache-2.0, Docker Compose or Helm	10k requests/mo, 7-day retention	Hobby $0 (10k requests/mo, 1 GB storage, 1 seat, 7-day retention, 10 logs/min); Pro $79/mo (10k free requests then usage-based, unlimited seats, 1-month retention, 1,000 logs/min); Team $799/mo (5 orgs, 3-month retention, 15,000 logs/min, SOC 2/HIPAA); Enterprise custom (on-prem, SAML SSO). Storage overage roughly $0.97/GB. 50% off first year for startups under 2 years and $5M raised; free for students.
Comet Opik	Yes — OTel-compatible, Python/JS SDKs	Yes — metrics SDK, LLM-judge, pytest-style test suites	Yes — prompt library with versioning, playground	Yes — Apache-2.0, same codebase as hosted	25k spans/mo cloud; unlimited self-hosted	Open source self-hosted: free, unlimited spans/users. Free Cloud $0 (25k spans/mo, 60-day retention, up to 10 users); Pro Cloud $19/mo (100k spans/mo, 60-day retention, up to 50 users); Enterprise custom. Overage $5 per 100k spans; extended retention 60→400 days at $29 per 100k spans.
Galileo	Yes — traces and agent-graph views	Yes — Luna evaluator models, custom metrics, runtime guardrails	Yes — prompt versioning and experiments	Enterprise only — VPC or on-prem	5,000 traces/mo	Free $0/mo (5,000 traces/mo, unlimited users, unlimited custom evals); Pro $100/mo billed yearly (50,000 traces/mo, standard RBAC, dedicated Slack support), scaling with trace volume; Enterprise custom (unlimited traces, hosted/VPC/on-prem, SSO, guardrails, 24/7 support). Retention is not published per tier.
Datadog Agent Observability	Yes — LLM spans correlated with APM/infra, OTel supported	Yes — datasets, automated evaluators, human review	Limited — experimentation over prompts, not a prompt CMS	No	None (14-day trial)	Billed annually: first 100k LLM spans $160/mo; each additional 10k LLM spans $3.50/mo. Retention add-ons per 10k spans/mo: 30-day $1.50, 60-day $3, 90-day $4. No free tier beyond Datadog's standard 14-day trial; AI Credits sold in 500-credit bundles at $500/mo or $1.30/credit.
Pydantic Logfire	Yes — OTel-native, SQL over spans, strong Python instrumentation	Partial — via separate Pydantic Evals library, not in-app	No	Enterprise only — Kubernetes/Helm	10M records/mo, 30-day retention	Personal $0 (10M records/mo, 30-day retention, 1 admin + 2 read-only guests, ingestion pauses at the limit); Team $49/mo (10M records included, $2 per additional million, 30-day retention, 5 seats, $25/extra seat); Growth $249/mo (10M included, $2/M, up to 90-day retention, unlimited seats); Enterprise custom with Kubernetes/Helm self-hosting.
Confident AI (DeepEval)	Yes — spans, though evals came first	Yes — DeepEval metrics, CI test runs, live-traffic evals, red teaming	Yes — prompt versioning	Enterprise only (framework itself runs anywhere)	1 GB-month spans, 5 test runs/week, 2 seats	Free $0 (2 seats, 1 project, 1 GB-month of trace spans, 5 test runs/week); Starter $200/mo (unlimited seats, 5 projects, 5 GB-months); Team $2,000/mo (unlimited projects, 75 GB-months, SSO, custom RBAC, SOC 2); Enterprise custom (red teaming, HIPAA, on-prem). Span overage $1 per GB-month ingested or retained; online eval tokens billed separately at roughly $0.05/M input and $0.40/M output.
HoneyHive	Yes — OTel-based, session and span views	Yes — automated evaluators plus human review workflows	Yes — prompt versioning and playground	Enterprise only — self-hosted, hybrid or single-tenant	10,000 events/mo, 5 users	Developer $0 (10,000 events/mo, up to 5 users, 30-day retention, no credit card). Enterprise: custom pricing, not published — custom event volume, unlimited users, custom retention, SSO, SLA, optional self-hosted/hybrid/single-tenant deployment.
Laminar	Yes — OTLP ingest, SQL queries, browser session replay	Yes — evaluations plus LLM-driven 'Signals' analysis	Limited — unknown/secondary to tracing	Yes — open-source repo, self-support	1 GB, 7-day retention, 1 seat	Free $0 (1 GB data, $5 Signals credit, 7-day retention, 1 seat, 1 project); Starter $30/mo (3 GB then $2/GB, $15 Signals credit, 30-day retention, unlimited seats/projects); Pro $150/mo (10 GB then $1.50/GB, $50 Signals credit, 6-month retention); Enterprise custom. Signals metered at $0.50/$3 per 1M tokens on Starter, $0.40/$2.50 on Pro.
Freeplay	Yes — observability with cost/latency analytics	Yes — model-graded, code-based and human review, batch testing	Yes — a core strength; prompts, models and params versioned together	unknown — not published	None published	unknown — no public pricing page or published tiers; contact sales. Deployment options (SaaS versus VPC) are likewise not published.
Traceloop (OpenLLMetry)	Yes — OpenLLMetry OTel instrumentation, exports to 25+ backends	Yes — evaluations and CI/CD checks in the hosted platform	Yes — included on the free tier	Enterprise only for the platform; library runs anywhere	50k spans/mo, 5 seats, 24-hour retention	Free Forever $0 (up to 50k spans/mo, up to 5 seats, 24-hour retention, includes monitoring, evaluation, CI/CD and prompt management); Enterprise custom for >50k spans/mo (unlimited seats, custom retention, SOC 2, on-prem/air-gapped, dedicated Slack). 14-day trial; available on AWS, GCP and Azure marketplaces. OpenLLMetry itself is free and Apache-2.0.
LangWatch	Yes — OTel-compatible event/trace capture	Yes — custom evaluators, annotations, scenario-based agent simulation	Yes — versioning and prompt optimisation	Enterprise only for supported deployments; OSS components Apache-2.0	50k events/mo, 14-day retention, 2 users	Developer €0 (50k events/mo, 14-day retention, 2 users, 3 scenarios / 3 simulations / 3 custom evals); Growth €29 per core seat/mo (200k events/mo then €5 per 100k, 30-day retention then €3/GB, unlimited lite users, unlimited simulations and evals, volume discounts above 20 users); Enterprise custom (negotiated volume, custom retention, SSO/RBAC, audit logs, ISO 27001, self-hosted/hybrid/on-prem).
PromptLayer	Partial — request/log-centric, weaker on deep agent span trees	Yes — evaluation pipelines and regression tests, basic depth	Yes — the core product; visual registry, versions, release labels	Enterprise only — GCP/AWS/Azure or single-tenant	2,500 requests/mo, 5 seats	Free $0 (2.5k requests/mo, 5 seats, 1 workspace); Pro $49/mo (2.5k+ requests, $0.003/transaction overage, 5 seats, unlimited workspaces); Team $500/mo (100k+ requests, $0.002/transaction overage, 25 seats); Enterprise custom (self-hosted on GCP/AWS/Azure, EU-hosted or single-tenant). Free/Pro/Team are US-hosted only.
Openlayer	Yes — inference tracing and production monitoring	Yes — test-suite framing, dev and production tests with alerts	Limited — version comparison rather than a prompt CMS	Enterprise only — on-premise available	20,000 inferences/mo, 1 member, 3-month retention	Basic $0 (20,000 inferences/mo, 1 member, 5 projects, 3-month retention, no on-premise). Enterprise: custom quote based on scale, deployment and support — no dollar figures published. There is no self-serve paid tier between the two.
Humanloop	Was yes — no longer viable	Was yes — no longer viable	Was yes (its main strength) — no longer viable	unknown / moot	Not applicable — sunsetting	unknown — pricing is no longer meaningful; the platform is being sunset following the Anthropic acquisition. No end-of-service date is published on the announcement page.

C Entries

Langfuse

Open-source platform covering traces, sessions, datasets, LLM-as-judge and human-annotation evals, and a prompt registry with versioning and caching SDK-side. The whole core product is MIT-licensed and self-hostable — unusual in this category, where OSS usually means a stripped client library. Cloud pricing is per 'unit' (roughly one ingested observation): Hobby free at 50k units/30-day retention, Core $29/mo, Pro $199/mo, Enterprise $2,499/mo, all with 100k units included and $8/100k after, stepping down to $6/100k above 50M. Integrates via OpenTelemetry as well as native Python/JS SDKs.

Tracing	Yes — OTel-native plus Python/JS SDKs
Evals	Yes — LLM-judge, custom scores, human annotation queues
Prompt management	Yes — versioned prompt registry with SDK caching
Self-host	Yes — MIT, free and fully featured for core
Free tier	50k units/mo cloud; unlimited self-hosted

Watch out: Self-hosting is not a single binary: production deployment needs Postgres, ClickHouse, Redis and S3-compatible storage, which is real ops work. Project-level RBAC, audit logs, data-retention policies, server-side masking and SCIM are Enterprise-licensed even when you self-host. The experiment/eval UI is less opinionated and less polished than Braintrust's, so teams whose whole workflow is scored experiments often find it thinner than they expected.

Hobby $0 (50k units/mo, 30-day retention, 2 users); Core $29/mo; Pro $199/mo; Enterprise cloud $2,499/mo — each with 100k units included, then $8/100k units, graduating to $7/$6.50/$6 per 100k at 1M/10M/50M. Self-hosted open source free; self-hosted Enterprise is custom-priced. · open source

LangSmith

Trace capture, dataset-backed experiments, LLM-judge and human-review evals, and a prompt hub, with the tightest integration into LangChain and LangGraph (agent runs render as graph steps rather than flat spans). It also accepts OpenTelemetry and non-LangChain SDK traffic. Billing has moved from a simple per-trace model to consumption units — LCUs for compute at $1.50 and LSUs for storage at $1.00 — layered on top of seat pricing, with 14-day retention on base traces and a paid upgrade to 400-day extended traces.

Tracing	Yes — native SDKs plus OTel ingest; LangGraph-aware views
Evals	Yes — datasets, experiments, LLM-judge, human review
Prompt management	Yes — prompt hub with versioning
Self-host	Enterprise only (self-hosted and hybrid)
Free tier	5k base traces/mo, 1 seat

Watch out: Self-hosted and hybrid deployment are Enterprise-only, so regulated teams face a sales cycle. The LCU/LSU consumption model makes forecasting hard — you cannot look at a trace count and predict the invoice — and 14-day default retention pushes you toward paid extended traces sooner than expected. Value drops noticeably if you are not on the LangChain stack; for plain OTel traffic other backends do the same job for less.

Developer $0 (1 seat, up to 5k base traces/mo then pay-as-you-go); Plus $39/seat/mo (10k base traces/mo included, unlimited seats, includes one small serverless deployment); Enterprise custom. Overages billed as LCU $1.50/unit (compute) and LSU $1.00/unit (storage). Base traces have 14-day retention; 400-day extended traces cost extra.

Braintrust

Built around the evaluation loop rather than around logging — datasets, scorers (code or LLM-judge), side-by-side experiment comparison, a prompt playground, and CI integration, with production logs feeding back into datasets. Billing is on processed data volume and score count, not traces: Starter is free with 1 GB/mo and 10k scores; Pro is $249/mo with 5 GB and 50k scores, then $3/GB and $1.50 per 1k scores. Enterprise offers a hybrid deployment where the data plane (API, Postgres, Redis, object storage and the Brainstore query engine) runs in your cloud while Braintrust hosts only the UI and auth metadata.

Tracing	Yes — spans/logs, though evals are the centre of gravity
Evals	Yes — strongest in class: datasets, scorers, experiment diffs, CI
Prompt management	Yes — versioned prompts and playground
Self-host	Enterprise only — hybrid, data plane in your cloud
Free tier	Starter: 1 GB and 10k scores/mo

Watch out: There is no free self-hosted option — hybrid deployment is Enterprise-only, and the control plane still lives at Braintrust even then. GB-based billing means verbose RAG contexts or image payloads can multiply your bill without any change in request volume, and default retention is short (14 days on Starter, 30 on Pro) with extra retention billed per GB-month. It is the wrong choice if you mainly want cheap high-volume request logging; a per-trace or OSS tool will be far cheaper for that.

Starter $0/mo ($10 model credits, 1 GB processed data, 10k scores, 14-day retention, unlimited seats); Pro $249/mo ($100 model credits, 5 GB then $3/GB, 50k scores then $1.50/1k, 30-day retention then $0.50/GB/mo); Enterprise custom. Starter overages: $4/GB and $2.50 per 1k scores. 6–12 months free for qualifying startups.

Weights & Biases Weave

The LLM-app layer of the W&B platform: decorator-based tracing for Python and TypeScript, an Evaluations API with programmatic and LLM-judge scorers, a playground for model comparison, online monitors that score production traffic, and saved prompt objects. It shares accounts, teams and RBAC with W&B Models, so teams that already train or fine-tune there get one login and one permissions model. Weave data ingestion is metered separately from Models storage: 1 GB/mo on Free, 1.5 GB/mo on Pro (from $60/mo), and $0.10 per additional MB.

Tracing	Yes — Python/TS decorators, OTel ingest supported
Evals	Yes — Evaluations API, scorers, online monitors
Prompt management	Yes — versioned prompt objects and playground
Self-host	Enterprise only for commercial use; dedicated cloud available
Free tier	1 GB/mo ingestion, 5 seats

Watch out: The ingestion overage is the headline risk: $0.10/MB is roughly $100/GB, one to two orders of magnitude above per-GB competitors, so tracing full documents or images gets expensive fast — plan on redaction and sampling. Self-hosting for commercial use requires an Advanced Enterprise agreement; the free local edition is non-commercial. Seat counts on Free/Pro are tied to the Models product, and if you do not otherwise use W&B the platform carries a lot of experiment-tracking surface you will never touch.

Free $0 (up to 5 Models seats, 5 GB storage, 1 GB/mo Weave ingestion, extra at $0.10/MB); Pro from $60/mo (up to 10 Models seats, 100 GB storage then $0.03/GB, 1.5 GB/mo Weave ingestion, extra at $0.10/MB); Enterprise custom with dedicated cloud or self-managed. Local self-hosted 'Personal' edition is free but non-commercial.

Arize AX

The commercial, production-facing counterpart to open-source Phoenix: OTel span ingestion, online and offline evaluations, dataset curation from live traffic, prompt experimentation, and the ML-monitoring heritage Arize brought from the pre-LLM era (drift, cohort analysis, embedding visualisation). Priced on span volume plus storage — Free at 25k spans/mo and 1 GB with 15-day retention, Pro at $50/mo for 50k spans and 10 GB with 30-day retention, Enterprise custom with SaaS or self-hosted deployment. Seats are unlimited on every tier.

Tracing	Yes — OpenTelemetry/OpenInference spans
Evals	Yes — online and offline evals, LLM-judge, experiments
Prompt management	Yes — prompt playground and versioning
Self-host	Enterprise only
Free tier	25k spans/mo, 15-day retention

Watch out: Span allowances are small relative to price — 50k spans/mo on Pro is a few thousand agent runs, and agentic apps burn spans quickly, so most real deployments land in a sales conversation. The Phoenix/AX split confuses people: features you saw in the OSS docs may sit behind the paid product or vice versa. Self-hosting is Enterprise-only, and the platform's breadth means more onboarding than a tracing-only tool.

AX Free $0 (25k spans/mo, 1 GB storage, 15-day retention, unlimited seats); AX Pro $50/mo (50k spans/mo, 10 GB, 30-day retention, unlimited seats); AX Enterprise custom (custom volume/retention, SaaS or self-hosted). Startup pricing available.

Arize Phoenix

A self-contained observability app you can start with pip install or a container: OpenTelemetry/OpenInference tracing, an eval library with prepackaged LLM-judge templates (hallucination, relevance, toxicity, QA correctness), versioned datasets and experiments, a prompt playground, and a remote MCP server so agents like Claude Code can query traces directly. Deployment targets include Docker Compose, a Helm chart, and one-click hosts like Railway and Cloud Run. Free with no volume metering when self-hosted; Arize also runs a hosted Phoenix, with heavier production workflows steered to the paid AX product.

Tracing	Yes — OTel/OpenInference, local UI
Evals	Yes — eval library with prepackaged judge templates
Prompt management	Yes — playground and prompt versioning
Self-host	Yes — pip, Docker, Helm; free
Free tier	Unlimited self-hosted; hosted limits unknown

Watch out: ELv2 is not an OSI-approved licence — you may not offer Phoenix as a managed service to third parties, which rules it out for some vendors and some corporate OSS policies. It is designed as a single-node app, so retention and query performance degrade well before the volumes AX handles, and there is no built-in alerting, on-call routing or SLA. Expect the upgrade path to be 'buy AX' rather than 'scale Phoenix'.

Self-hosted: free, no volume limits (infrastructure costs only). Hosted Phoenix Cloud free tier limits are not published on the pricing page — treat as unknown; paid production workloads are directed to Arize AX (Pro $50/mo). · open source

Helicone

Started as a one-line proxy (swap your base URL) that logs every LLM request with cost, latency, caching and rate limiting, and now also offers async logging so you need not route traffic through it, plus an AI gateway with fallbacks across 100+ models. Fully Apache-2.0 and self-hostable via Docker Compose or Helm. Cloud tiers are Hobby free (10k requests/mo, 7-day retention, 10 logs/min), Pro $79/mo with unlimited seats and 1-month retention, Team $799/mo with 3-month retention and SOC 2/HIPAA, plus Enterprise with on-prem.

Tracing	Yes — request/session logging via proxy or async
Evals	Basic — scores and online evaluators; weakest area
Prompt management	Yes — prompt versioning and experiments
Self-host	Yes — Apache-2.0, Docker Compose or Helm
Free tier	10k requests/mo, 7-day retention

Watch out: Its evaluation tooling is the thinnest of the serious contenders — this is a logging and gateway product first, so dataset-driven experiments and scorer management are not where it competes. The proxy mode puts a third party in your request path unless you use async logging or self-host. Free-tier limits are harsh in practice (7-day retention and a 10 logs/min ceiling), and self-hosting means running Postgres/Supabase, ClickHouse, MinIO and multiple services.

Hobby $0 (10k requests/mo, 1 GB storage, 1 seat, 7-day retention, 10 logs/min); Pro $79/mo (10k free requests then usage-based, unlimited seats, 1-month retention, 1,000 logs/min); Team $799/mo (5 orgs, 3-month retention, 15,000 logs/min, SOC 2/HIPAA); Enterprise custom (on-prem, SAML SSO). Storage overage roughly $0.97/GB. 50% off first year for startups under 2 years and $5M raised; free for students. · open source

Comet Opik

Comet's LLM product, shipped as an open-source codebase identical to the hosted versions: trace and span capture, an evaluation SDK with prepackaged metrics (hallucination, moderation, context precision/recall), experiment tracking, an agent playground, and pytest-style test suites for CI. Cloud pricing is the cheapest of the mainstream managed options — Free at 25k spans/mo for up to 10 users, Pro at $19/mo for 100k spans and up to 50 users, with additional spans at $5 per 100k and 400-day retention available at $29 per 100k spans.

Tracing	Yes — OTel-compatible, Python/JS SDKs
Evals	Yes — metrics SDK, LLM-judge, pytest-style test suites
Prompt management	Yes — prompt library with versioning, playground
Self-host	Yes — Apache-2.0, same codebase as hosted
Free tier	25k spans/mo cloud; unlimited self-hosted

Watch out: Cloud tiers are cheap because the span allowances are small — 100k spans/mo on Pro disappears quickly under agentic workloads, and the per-100k overage is where the real cost lands. Self-hosting the full stack (ClickHouse, MySQL, Redis, via Docker Compose or Kubernetes) is more involved than the quickstart suggests. Opik sits inside Comet's broader ML platform, which means extra product surface and, for buyers, a vendor whose main business is experiment tracking rather than LLM ops.

Open source self-hosted: free, unlimited spans/users. Free Cloud $0 (25k spans/mo, 60-day retention, up to 10 users); Pro Cloud $19/mo (100k spans/mo, 60-day retention, up to 50 users); Enterprise custom. Overage $5 per 100k spans; extended retention 60→400 days at $29 per 100k spans. · open source

Galileo

Positions itself around agent reliability rather than generic tracing: trace capture, agent-graph analysis, a metrics library run by small purpose-built evaluator models (the Luna family) instead of frontier-model judges for every score, and real-time guardrails that can block or flag responses in the request path. Free gives 5,000 traces/mo with unlimited users and unlimited custom evaluators; Pro is $100/mo billed yearly for 50,000 traces with RBAC and Slack support; Enterprise adds hosted, VPC or on-prem deployment, dedicated low-latency inference for evaluators, and 24/7 support.

Tracing	Yes — traces and agent-graph views
Evals	Yes — Luna evaluator models, custom metrics, runtime guardrails
Prompt management	Yes — prompt versioning and experiments
Self-host	Enterprise only — VPC or on-prem
Free tier	5,000 traces/mo

Watch out: Trace allowances are low for the price — 50k traces/mo at $100 is thin for anything agentic, and the pricing page publishes no retention figures, so total cost is hard to model without sales. The product has been repositioned and renamed repeatedly (the older Evaluate/Observe/Protect modules versus today's agent-reliability framing), so blog posts, tutorials and SDK docs drift out of sync. Self-hosting is Enterprise-only and the motion is clearly sales-led — not a tool you can fully evaluate on a credit card.

Free $0/mo (5,000 traces/mo, unlimited users, unlimited custom evals); Pro $100/mo billed yearly (50,000 traces/mo, standard RBAC, dedicated Slack support), scaling with trace volume; Enterprise custom (unlimited traces, hosted/VPC/on-prem, SSO, guardrails, 24/7 support). Retention is not published per tier.

Datadog Agent Observability

Datadog's LLM product, now marketed as Agent Observability: LLM span tracing correlated with APM traces, infrastructure metrics and RUM, plus datasets, automated evaluators and human review for offline experimentation. The pull is correlation — you can follow a slow agent turn into the backend service and database call that caused it in one product. Priced per LLM span with retention sold separately: $160/mo for the first 100k spans (annual billing), $3.50 per additional 10k, and 30/60/90-day retention at $1.50/$3/$4 per 10k spans per month.

Tracing	Yes — LLM spans correlated with APM/infra, OTel supported
Evals	Yes — datasets, automated evaluators, human review
Prompt management	Limited — experimentation over prompts, not a prompt CMS
Self-host	No
Free tier	None (14-day trial)

Watch out: Note the rename — 'LLM Observability' is now Agent Observability, and older docs, dashboards and blog links use the old name. It is the most expensive option per span at volume, and the retention surcharge means the sticker price understates the bill. Evaluation depth — scorer iteration, experiment diffing, dataset curation — is shallower than Braintrust or Langfuse, and there is no free tier or self-hosted option, so it only makes sense inside an existing Datadog contract.

Billed annually: first 100k LLM spans $160/mo; each additional 10k LLM spans $3.50/mo. Retention add-ons per 10k spans/mo: 30-day $1.50, 60-day $3, 90-day $4. No free tier beyond Datadog's standard 14-day trial; AI Credits sold in 500-credit bundles at $500/mo or $1.30/credit.

Pydantic Logfire

General-purpose OpenTelemetry observability from the Pydantic team, with first-class instrumentation for Pydantic AI, LLM SDKs, FastAPI, HTTPX and databases — LLM calls show up as spans alongside the rest of your service, queryable with SQL. Every plan includes 10 million spans/logs/metrics per month ($20 of usage); Team is $49/mo with 5 seats and $2 per additional million records, Growth is $249/mo with unlimited seats and up to 90-day retention, and Enterprise adds a Helm-chart self-hosted deployment. Evaluation lives in the separate open-source Pydantic Evals library rather than in the app UI.

Tracing	Yes — OTel-native, SQL over spans, strong Python instrumentation
Evals	Partial — via separate Pydantic Evals library, not in-app
Prompt management	No
Self-host	Enterprise only — Kubernetes/Helm
Free tier	10M records/mo, 30-day retention

Watch out: It is not an LLM eval platform: there is no in-app dataset/experiment/LLM-judge workflow, no scorer management and no prompt registry — you get spans and SQL, and run evals separately via the Pydantic Evals library. The UI is trace- and query-oriented rather than built around comparing prompt versions, so product managers will not use it. Self-hosting is Enterprise-only, and the LLM-specific views are less rich than a dedicated tool's.

Personal $0 (10M records/mo, 30-day retention, 1 admin + 2 read-only guests, ingestion pauses at the limit); Team $49/mo (10M records included, $2 per additional million, 30-day retention, 5 seats, $25/extra seat); Growth $249/mo (10M included, $2/M, up to 90-day retention, unlimited seats); Enterprise custom with Kubernetes/Helm self-hosting.

Confident AI (DeepEval)

The hosted companion to DeepEval, the widely used open-source LLM evaluation framework with 30+ single-turn and 15+ multi-turn metrics (faithfulness, answer relevancy, contextual precision, task completion) that run as pytest tests. The cloud product adds shared datasets, test-run history and comparison, tracing, live-traffic evals, prompt versioning, alerting and no-code metric configuration. Free covers 2 seats, 1 project and 5 test runs per week; Starter is $200/mo, Team $2,000/mo, with span storage billed at $1 per GB-month beyond the included allowance and Enterprise adding red teaming, governance and on-prem.

Tracing	Yes — spans, though evals came first
Evals	Yes — DeepEval metrics, CI test runs, live-traffic evals, red teaming
Prompt management	Yes — prompt versioning
Self-host	Enterprise only (framework itself runs anywhere)
Free tier	1 GB-month spans, 5 test runs/week, 2 seats

Watch out: The jump from free to $200/mo Starter is the steepest in the category — there is no $20–50 tier for a small team. Production tracing and monitoring are newer and less mature than the offline eval tooling; if live debugging is your main need, this is the wrong shape of product. DeepEval's metrics are LLM-judge-based, so evaluation itself burns tokens you pay for on top of the subscription, and judge cost scales with dataset size.

Free $0 (2 seats, 1 project, 1 GB-month of trace spans, 5 test runs/week); Starter $200/mo (unlimited seats, 5 projects, 5 GB-months); Team $2,000/mo (unlimited projects, 75 GB-months, SSO, custom RBAC, SOC 2); Enterprise custom (red teaming, HIPAA, on-prem). Span overage $1 per GB-month ingested or retained; online eval tokens billed separately at roughly $0.05/M input and $0.40/M output. · open source

HoneyHive

An LLM-ops platform combining OTel tracing, automated and human evaluations, prompt versioning, dataset curation and CI integration, aimed at enterprise teams — it publicises deployments at large financial institutions. The Developer tier is genuinely usable for prototyping: free, 10,000 events per month, up to 5 users and 30-day retention. Everything above that is Enterprise, which adds custom volumes, SAML/custom SSO, an uptime SLA, a dedicated account manager, and optional self-hosted, hybrid or single-tenant deployment.

Tracing	Yes — OTel-based, session and span views
Evals	Yes — automated evaluators plus human review workflows
Prompt management	Yes — prompt versioning and playground
Self-host	Enterprise only — self-hosted, hybrid or single-tenant
Free tier	10,000 events/mo, 5 users

Watch out: There is no published mid-tier price and no self-serve paid plan — you go from a free 10k-event account straight to a sales call, which makes budgeting and procurement slow. The community and integration ecosystem are much smaller than Langfuse's or LangSmith's, so you will find fewer worked examples and third-party recipes. Self-hosting is Enterprise-only.

Developer $0 (10,000 events/mo, up to 5 users, 30-day retention, no credit card). Enterprise: custom pricing, not published — custom event volume, unlimited users, custom retention, SSO, SLA, optional self-hosted/hybrid/single-tenant deployment.

Laminar

A tracing and evaluation backend for AI agents that ingests over OTLP, adds SQL querying and dashboards over spans, records browser sessions for web-automation agents, and offers 'Signals' — LLM-driven analysis of traces, metered in tokens. Billed on data volume rather than trace count: Free gives 1 GB, 7-day retention, 1 seat and 1 project; Starter is $30/mo for 3 GB, 30-day retention and unlimited seats; Pro is $150/mo for 10 GB and 6-month retention, with overage at $2/GB and $1.50/GB respectively. Note the product moved from lmnr.ai to laminar.sh — old links redirect.

Tracing	Yes — OTLP ingest, SQL queries, browser session replay
Evals	Yes — evaluations plus LLM-driven 'Signals' analysis
Prompt management	Limited — unknown/secondary to tracing
Self-host	Yes — open-source repo, self-support
Free tier	1 GB, 7-day retention, 1 seat

Watch out: Small team and young product: the brand moved domains from lmnr.ai to laminar.sh, and self-hosting details are not documented on the pricing page, so confirm the licence and support expectations before committing. GB-based billing means verbose agent traces (full page DOM, screenshots) consume the allowance fast. Free-tier limits of 1 seat and 1 project rule out even trial collaboration, and eval tooling is lighter than Braintrust's or Langfuse's.

Free $0 (1 GB data, $5 Signals credit, 7-day retention, 1 seat, 1 project); Starter $30/mo (3 GB then $2/GB, $15 Signals credit, 30-day retention, unlimited seats/projects); Pro $150/mo (10 GB then $1.50/GB, $50 Signals credit, 6-month retention); Enterprise custom. Signals metered at $0.50/$3 per 1M tokens on Starter, $0.40/$2.50 on Pro. · open source

Freeplay

An ops platform built around cross-functional workflow rather than developer tooling: prompts, models and hyperparameters versioned together so non-engineers can change and deploy configuration; a collaborative playground for side-by-side comparison against saved datasets; model-graded, code-based and human evaluations for both batch testing and production traffic; datasets built from production logs; and observability with cost and latency analytics. It also ships AI-assisted features for generating evaluators and optimising prompts from production data.

Tracing	Yes — observability with cost/latency analytics
Evals	Yes — model-graded, code-based and human review, batch testing
Prompt management	Yes — a core strength; prompts, models and params versioned together
Self-host	unknown — not published
Free tier	None published

Watch out: No public pricing, no published free tier and no open-source component — you cannot evaluate it without talking to sales, and its own site did not serve a pricing page when checked. It is a small vendor with a much thinner integration ecosystem than Langfuse or LangSmith, and its cross-functional framing is overkill for a solo developer or an engineering-only team that versions prompts in git. Confidence in the commercial details here is low.

unknown — no public pricing page or published tiers; contact sales. Deployment options (SaaS versus VPC) are likewise not published.

Traceloop (OpenLLMetry)

Two things under one name. OpenLLMetry is an Apache-2.0 set of OpenTelemetry instrumentations covering LLM SDKs, vector databases and frameworks, which exports to 25+ existing observability backends — it is the standard way to get LLM spans into a stack you already run. Traceloop is the hosted backend: monitoring, evaluation, CI/CD checks and prompt management, free forever up to 50,000 spans/month with 5 seats but only 24-hour retention, with Enterprise adding higher volumes, SOC 2, and on-premise or air-gapped Kubernetes deployment.

Tracing	Yes — OpenLLMetry OTel instrumentation, exports to 25+ backends
Evals	Yes — evaluations and CI/CD checks in the hosted platform
Prompt management	Yes — included on the free tier
Self-host	Enterprise only for the platform; library runs anywhere
Free tier	50k spans/mo, 5 seats, 24-hour retention

Watch out: The hosted free tier's 24-hour retention makes it a demo rather than a working plan — anything real requires an Enterprise quote, and there is no published mid-tier. The genuinely valuable, widely adopted piece is the OSS instrumentation library, which most teams use with a different backend, so evaluate Traceloop-the-platform on its own merits rather than assuming the library's popularity transfers. Self-hosting the platform is Enterprise-only.

Free Forever $0 (up to 50k spans/mo, up to 5 seats, 24-hour retention, includes monitoring, evaluation, CI/CD and prompt management); Enterprise custom for >50k spans/mo (unlimited seats, custom retention, SOC 2, on-prem/air-gapped, dedicated Slack). 14-day trial; available on AWS, GCP and Azure marketplaces. OpenLLMetry itself is free and Apache-2.0. · open source

LangWatch

European LLM-ops platform with Apache-2.0 open-source components, notable for agent simulation: you define scenarios and run repeated simulated conversations against your agent, rather than only scoring static datasets. Adds tracing, custom evaluators, annotation and prompt optimisation. The Developer tier is free with 50k events/month, 14-day retention and 2 users; Growth is €29 per 'core seat' per month with 200k events included, €5 per additional 100k, 30-day retention and €3/GB beyond it, with unlimited 'lite' users; Enterprise adds hybrid, self-hosted or on-premises deployment plus ISO 27001 reporting.

Tracing	Yes — OTel-compatible event/trace capture
Evals	Yes — custom evaluators, annotations, scenario-based agent simulation
Prompt management	Yes — versioning and prompt optimisation
Self-host	Enterprise only for supported deployments; OSS components Apache-2.0
Free tier	50k events/mo, 14-day retention, 2 users

Watch out: The billing model is the most complicated here — per-core-seat plus per-event plus per-GB retention, in euros — so comparing it against per-trace competitors takes a spreadsheet. Free-tier caps of 3 scenarios and 3 custom evals limit how far you can trial the differentiating feature. Ecosystem and community are smaller than Langfuse's, and supported self-hosting is Enterprise-only even though components are Apache-2.0.

Developer €0 (50k events/mo, 14-day retention, 2 users, 3 scenarios / 3 simulations / 3 custom evals); Growth €29 per core seat/mo (200k events/mo then €5 per 100k, 30-day retention then €3/GB, unlimited lite users, unlimited simulations and evals, volume discounts above 20 users); Enterprise custom (negotiated volume, custom retention, SSO/RBAC, audit logs, ISO 27001, self-hosted/hybrid/on-prem). · open source

PromptLayer

A prompt management system first and an observability tool second: a visual prompt registry with versioning, A/B release labels and a no-code editor so product and content people can change prompts without a deploy, backed by request logging, evaluation pipelines and analytics. Priced per request, not per span — Free covers 2.5k requests/month with 5 seats, Pro is $49/mo with overages at $0.003 per transaction, Team is $500/mo with 100k requests included and $0.002 overages plus 25 seats, and Enterprise adds self-hosting on GCP, AWS or Azure, EU hosting, or single-tenant cloud.

Tracing	Partial — request/log-centric, weaker on deep agent span trees
Evals	Yes — evaluation pipelines and regression tests, basic depth
Prompt management	Yes — the core product; visual registry, versions, release labels
Self-host	Enterprise only — GCP/AWS/Azure or single-tenant
Free tier	2,500 requests/mo, 5 seats

Watch out: Request-based pricing is unforgiving at volume — $0.002–0.003 per transaction means a million requests a month costs thousands, far above per-span competitors. Its tracing model is centred on LLM requests rather than deep OTel span trees, so multi-step agent debugging is weaker than in Langfuse, Phoenix or Braintrust. Self-hosting and EU data residency are Enterprise-only, and the eval tooling is basic next to eval-first products.

Free $0 (2.5k requests/mo, 5 seats, 1 workspace); Pro $49/mo (2.5k+ requests, $0.003/transaction overage, 5 seats, unlimited workspaces); Team $500/mo (100k+ requests, $0.002/transaction overage, 25 seats); Enterprise custom (self-hosted on GCP/AWS/Azure, EU-hosted or single-tenant). Free/Pro/Team are US-hosted only.

Openlayer

Frames evaluation as a test suite: you define tests and quality goals that run against development datasets and then continue to run against production inferences, with alerting when a test starts failing on live traffic. Covers tracing, monitoring, and regression detection across model and prompt versions. The Basic tier is free with 20,000 inferences per month, 1 member, 5 projects and 3-month data retention — the longest free retention among the tools here — with Enterprise adding unlimited members and projects, custom volumes and on-premise deployment.

Tracing	Yes — inference tracing and production monitoring
Evals	Yes — test-suite framing, dev and production tests with alerts
Prompt management	Limited — version comparison rather than a prompt CMS
Self-host	Enterprise only — on-premise available
Free tier	20,000 inferences/mo, 1 member, 3-month retention

Watch out: Pricing above the free tier is entirely unpublished, and there is no self-serve paid plan — a single free seat then a sales call. The single-member free tier makes team trials impossible without a contract. It is a smaller vendor with a modest ecosystem, and if you want exploratory trace debugging or prompt collaboration rather than a codified test suite, tools built around those workflows fit better.

Basic $0 (20,000 inferences/mo, 1 member, 5 projects, 3-month retention, no on-premise). Enterprise: custom quote based on scale, deployment and support — no dollar figures published. There is no self-serve paid tier between the two.

Humanloop

Formerly a prompt management, evaluation and observability platform aimed at enterprise teams, with a strong prompt-versioning and human-feedback workflow. Humanloop was acquired by Anthropic and the company is sunsetting the platform rather than continuing it: its own site states it will 'continue to work closely with our customers to make their transition as smooth as possible' and points users to a migration guide. Do not start new work here; if you are on it, plan a migration to Langfuse, Braintrust, LangSmith or Freeplay depending on whether prompt management or evals mattered most to you.

Tracing	Was yes — no longer viable
Evals	Was yes — no longer viable
Prompt management	Was yes (its main strength) — no longer viable
Self-host	unknown / moot
Free tier	Not applicable — sunsetting

Watch out: The platform is shutting down, so every limitation is fatal: no roadmap, no new integrations, and an end-of-service date that has not been publicly stated, which makes migration planning harder than it should be. Treat any tutorial, comparison article or vendor matrix that still lists Humanloop as an option as out of date. Confidence in the exact shutdown timeline is low; check the vendor's migration guide directly.

unknown — pricing is no longer meaningful; the platform is being sunset following the Anthropic acquisition. No end-of-service date is published on the announcement page. · discontinued