// category · 10 tools · 5 open source
LLM Observability Tools
Tracing, logging and monitoring for LLM apps and agents in production, so you can see what happened, what it cost and where quality slipped.
LLM observability tools record what a language model application does in production so you can debug it, measure cost and quality, and catch regressions. This category covers tracing platforms, instrumentation libraries and monitoring products for single model calls, retrieval pipelines and multistep agents.
At minimum, observability captures each prompt and completion, the model and parameters used, latency, token counts and cost. For retrieval apps it adds the documents fetched. For agents it adds the whole loop: which tools were offered and chosen, their arguments and results, retries, subagents and how many steps a task took. Quality signals such as user feedback and automated scores are attached to the same trace, which links monitoring to evaluation.
Open source platforms. Langfuse has an MIT licensed core and became part of ClickHouse in January 2026. Arize Phoenix is free to self host under the Elastic License 2.0, and Arize became part of Dynatrace on October 1, 2026. Opik and Laminar are Apache 2.0 options, with Laminar aimed at long running agents.
Instrumentation. OpenLLMetry is not a dashboard but a set of OpenTelemetry instrumentations that send traces to any compatible backend. OpenTelemetry's GenAI conventions are becoming the shared format, which keeps your code portable between vendors.
Hosted products. LangSmith fits teams already on LangChain and LangGraph. Helicone, HoneyHive, Galileo, W&B Weave and Braintrust cover hosted tracing, monitoring and evaluation with different emphases.
Agents raised the stakes in 2026. In October, Partnership on AI reported six telemetry gaps in widely used agent frameworks, including missing records of permission changes, memory changes and human approvals. A tracing platform cannot show data that the framework never emitted, so check what your stack actually records.
To choose, start with our roundup of the best LLM observability tools, then read AI agent observability for agent specific monitoring. Head to head comparisons: Langfuse vs LangSmith and Langfuse vs Arize Phoenix.
Langfuselangfuse.comOpen source LLM engineering platform (MIT core) for tracing, prompt management, datasets and evals, free to self host. Owned by ClickHouse since January 2026.ObservabilityOpen source★ 36.6k
Laminarlaminar.shOpen source observability platform for AI agents with tracing, evaluations and browser session replays.ObservabilityOpen source★ 3.5k
Opikcomet.comComet's open source platform for logging, evaluating and monitoring LLM apps, RAG systems and agents.ObservabilityOpen source★ 23k
Arize Phoenixarize.comFree, self hostable tracing and evaluation tool built on OpenTelemetry, source available under the Elastic License 2.0. Arize became part of Dynatrace in October 2026.ObservabilityFree★ 12k
OpenLLMetrytraceloop.comOpenTelemetry based instrumentation for LLM providers and frameworks, sending traces to any observability backend.ObservabilityOpen source★ 7.5k
Heliconehelicone.aiOpen source LLM gateway and request logger for tracking cost, latency and usage. Acquired by Mintlify in March 2026; the hosted service now runs in maintenance mode.ObservabilityOpen source★ 6.2k
W&B Weavewandb.aiWeights & Biases toolkit for tracing, evaluating and iterating on LLM applications.ObservabilityFreemium★ 1.1k
Datadog Agent Observabilitydatadoghq.comDatadog's hosted tracing and evaluation product for LLM apps and agents (formerly LLM Observability), billed per LLM call and linked to Datadog APM and infrastructure data.ObservabilityFreemium
HoneyHivehoneyhive.aiPlatform for testing, monitoring and evaluating AI agents across development and production.ObservabilitySee site
LangSmithlangchain.comLangChain's platform for tracing, evaluating and monitoring LLM apps and agents, framework agnostic.ObservabilityFreemium// guides
Guides to LLM observability tools
Best LLM Observability Tools in 2026, Compared
The best LLM observability tools in 2026 compared: Langfuse, LangSmith, Phoenix, Opik, Weave and more, with licenses, self hosting and who now owns what.
AI Agent Observability: What It Is and Top Tools
AI agent observability explained: trace tool loops, permissions and memory, the six PAI telemetry gaps, and which tools fit, from Langfuse to Datadog.
Is LangSmith API Free? Keys, Limits and Costs
Is LangSmith API free? Yes: API keys and the REST API cost nothing, and the Developer plan covers 5,000 traces a month. See key types, limits and overage costs.
Is Langfuse Free for Commercial Use? (2026)
Is Langfuse free for commercial use? Yes: the MIT core can be self hosted in paid products at no cost. See what needs a license key and when the cloud bills.
Datadog LLM Observability vs Langfuse (2026)
Datadog LLM Observability vs Langfuse: how each bills (LLM spans vs units), retention, self hosting, evals and setup, with worked cost examples for 2026.
Langfuse vs LangSmith: Which Should You Use?
Langfuse vs LangSmith in 2026: open source license, self hosting, pricing, retention, framework fit and migration tips, so you can pick the right LLM tracer.
Langfuse vs Arize Phoenix: Which to Self Host?
Langfuse vs Arize Phoenix in 2026: licenses, self hosting stacks, features, managed pricing, new owners and which open LLM tracing tool fits your team best.
How to Use LangSmith with LangGraph (2026)
How to use LangSmith with LangGraph: set four environment variables, read node traces and threads, then score your graph and single nodes on a dataset.
RAG Evaluation Metrics: Ragas vs DeepEval vs TruLens
RAG evaluation metrics explained: faithfulness, relevancy, context precision and recall, which need a reference, and how Ragas, DeepEval and TruLens differ.
LLM as a Judge: How It Works and When to Trust It
LLM as a judge explained: how model graders work, the biases that skew them, how to validate one against human labels, and which open source tools to use.
How to Evaluate LLM Applications (2026 Guide)
How to evaluate LLM applications step by step: error analysis, code checks, calibrated LLM judges, CI gates and production sampling, with tools for each stage.
What Is Answer Completeness in RAG Evaluation?
What is answer completeness in RAG evaluation? The share of key facts from a reference answer that a response covers, and how Azure, Bedrock and Ragas score it.
Artificial Hivemind: Why AI Models Sound Alike
The Artificial Hivemind paper explained: why AI models give the same answers, what Infinity-Chat and NoveltyBench measure, and how to test output diversity yourself.
LLM Confidence Scores: Can You Trust Them?
Can you trust an LLM confidence score? What verbalized scores, logprobs and sampling measure, what 2026 calibration research shows, and how to set safe thresholds.
What Is Faithfulness in RAG Evaluation?
What is faithfulness in RAG evaluation? The share of an answer's claims backed by the retrieved context, and how Ragas, DeepEval, TruLens and HHEM score it.
LLM Red Teaming Tools: Promptfoo, garak, PyRIT
The best LLM red teaming tools in 2026 compared: Promptfoo, garak, PyRIT, DeepTeam and Giskard, with licenses, first commands and OWASP LLM Top 10 coverage.