LLM Observability / Criteria
How we compare: the criteria
Every tool is judged on the same 11 questions, answered from the vendor's own public pages and linked to them. No scores, no ratings: the facts, side by side, and what each one means for your team.
1. What it is built around
Is the product centred on monitoring production, on running evals, or on both?
Why it matters
Every tool here traces LLM calls. What differs is the job the rest of the product is shaped for. A platform built around experiments assumes you know what good looks like before you ship; a monitoring product assumes you will learn it from production. Pick the one shaped like your biggest gap.
How to check it
- Read the product's own one-line description and its docs' introduction page.
- Look at which workflow the quickstart walks you through first: a trace, or an experiment.
How the tools answer
- Lemma: Production monitoring and observability for AI agents running with real users. Source
- Braintrust: Evals and observability in one platform: discover patterns in production, turn them into evals, improve with each release. Source
- HoneyHive: Agent observability and evaluation for teams running agents in production, aimed at enterprises. Source
- Laminar: Open-source agent observability that catches agent failures at scale and helps fix them. Source
2. Tracing depth
What does one trace hold: just the model call, or the whole agent run with tool calls, sub-agents and conversation turns?
Why it matters
An agent failure usually happens between model calls: a tool returned nothing, a sub-agent was handed the wrong context, a turn forgot what the user said three turns ago. If the trace stops at the prompt and the completion, you cannot see the step that went wrong.
How to check it
- Does a trace include tool calls with arguments and results?
- Are sub-agents and handoffs nested in one trace?
- Can multi-turn conversations be linked across traces (a thread or session ID)?
How the tools answer
- Lemma: One trace per agent execution: the conversation, the span tree, LLM generations, tool calls and errors; multi-turn conversations linked by a thread ID. Source
- Braintrust: Prompts, responses and tool calls per trace, with latency, token usage and cost; auto-instrumentation for supported providers. Source
- HoneyHive: Every prompt, tool call, output and handoff; long-running trajectories with retries, loops and sub-agent handoffs. Source
- Laminar: LLM calls, tool calls, sub-agents, costs and tokens in a transcript view; screen recording for browser agents; full-text search across spans. Source
3. Silent failure detection
How does the tool find a bad output that raised no error: scorers you write, thresholds you set, or analysis that finds failure modes on its own?
Why it matters
Most LLM failures are silent. The request returns 200, the agent sounds confident, and the answer is wrong. Error-rate alerts never fire. Detection that depends on checks you write upfront only catches failures you already imagined; detection that reads traces against what the agent was supposed to do can catch the ones you didn't.
How to check it
- Do you have to define every failure as a scorer or rule before it can be caught?
- Does the tool group repeated failures into one pattern, or show you each trace separately?
- What does it compare an output against: a rubric you wrote, a reference answer, or the agent's own instructions?
How the tools answer
- Lemma: Audits each completed trace against the agent's instructions, without hand-written assertions, and groups repeated evidence into issues. Behavior monitors you describe in words add targeted checks. Source
- Braintrust: Scorers (LLM, code or human) run on production traces as online scoring; Topics classifies logs by intent, sentiment and issues to surface recurring patterns. Source
- HoneyHive: Online evaluators (LLM judges, code or human) score sampled live traffic; alerts fire when a score crosses a threshold or drifts. Source
- Laminar: Signals analyze runs to surface failure modes you did not define, using Laminar's own trace-analysis model, and cluster similar failures. Source
4. Triage and workflow
Once something is flagged, how does it reach someone who can fix it, and how does it get to a fix?
Why it matters
A flagged trace no one looks at is noise. The useful unit is an issue: a recurring failure with its evidence, a status, an owner, and a path into the code, whether that is a ticket, a review queue or a coding agent with the context loaded.
How to check it
- Is there an issue or cluster object with status, or only individual flagged traces?
- Can a coding agent read traces and issues (an MCP server or CLI)?
- Does it connect to where your team already works (Slack, Linear, PagerDuty, webhooks)?
How the tools answer
- Lemma: Issues carry status (open, in progress, resolved, dismissed), priority and their supporting traces; Inspect answers questions about a trace or issue; Linear tickets can be created from issues. Source
- Braintrust: Custom trace views and annotation, review queues, one-click trace-to-dataset, and the Loop agent for natural-language trace investigation. Source
- HoneyHive: Flagged traces route to annotation queues for expert review; an MCP server, CLI and agent skills let coding agents root-cause alerts. Source
- Laminar: Signal clusters with written reports; Laminar Agent answers questions in Slack; CLI and MCP give coding agents SQL access to traces. Source
5. Evals
Can you run offline experiments on a dataset before release, score live traffic online, or both?
Why it matters
Offline evals answer "is this version better than the last one?" before users see it. Online evals answer "is production still behaving?" after. Teams iterating on prompts and models weekly lean on the first; teams whose failures only show up with real users lean on the second.
How to check it
- Are there versioned datasets and side-by-side experiment comparisons?
- Can a production failure become a test case or online eval?
- Can evals run in CI and block a release?
How the tools answer
- Lemma: No offline evaluation. After a fix ships, Lemma creates an online eval from the real failure so a regression is flagged. Source
- Braintrust: Offline experiments against versioned datasets, side-by-side prompt and model comparison, automated and human scoring. Source
- HoneyHive: Offline experiments on datasets, regression comparison between agent versions, datasets built from production failures, evals in CI. Source
- Laminar: Error clusters can become eval datasets; agent evaluations run in real environments with lmnr-cli. Source
6. Alerting
What triggers an alert, and where does it go?
Why it matters
An alert on a metric threshold tells you a number moved. An alert on a new issue tells you what broke. Both are useful; they need different setup, and the second needs detection that understands outputs, not just counts them.
How to check it
- Is the trigger a threshold you set, drift, or a newly detected failure pattern?
- Which channels are supported: Slack, email, PagerDuty, webhooks?
How the tools answer
- Lemma: Issue alerts and Issue Briefs in Slack; signed webhooks for issue lifecycle events. Source
- Braintrust: Log, time-window and environment alerts sent to Slack, PagerDuty or webhooks. Source
- HoneyHive: Aggregate thresholds and drift detection, notified by email or webhook. Source
- Laminar: New-event and new-cluster alerts per Signal, in-app and by Slack or email. Source
7. Integrations and framework support
Does it support the agent framework and language you already use, and does it accept OpenTelemetry?
Why it matters
Instrumentation is the cost you pay first and the one that decides trace quality. A native integration for your framework captures tool calls and sub-agents correctly; a generic wrapper often flattens them. OpenTelemetry support keeps you from rewriting instrumentation if you switch.
How to check it
- Is your framework listed in the integration docs (LangGraph, OpenAI Agents SDK, Vercel AI SDK, Mastra, and so on)?
- Is there an SDK in your language?
- Can existing OpenTelemetry or other tracing stay in place?
How the tools answer
- Lemma: OpenAI Agents SDK, Vercel AI SDK, LangChain, LangGraph and Mastra; existing Langfuse or OpenTelemetry instrumentation can stay alongside the Lemma SDK. MCP server for coding agents. Source
- Braintrust: SDKs for Python, TypeScript, Go, Ruby and C#; OpenTelemetry; LangChain, LangGraph, Vercel AI SDK and more; an MCP server for coding agents. Source
- HoneyHive: OpenTelemetry-native; SDK auto-instrumentation it puts at 100+ models and frameworks; coding agents and no-code platforms such as ServiceNow and Copilot Studio. Source
- Laminar: Vercel AI SDK, Claude Agent SDK, OpenAI Agents SDK, LangChain, Mastra, Pydantic AI, LiteLLM, Browser Use, Stagehand, Playwright, and OpenAI, Anthropic and Gemini SDKs. Source
8. Setup
What does it take to get from nothing to useful findings?
Why it matters
Getting traces in is usually quick. The real setup cost is what comes after: writing scorers, choosing thresholds, labelling data. Count those hours too, not just the install.
How to check it
- How many steps from install to the first trace?
- What must be configured before the tool flags anything useful?
How the tools answer
- Lemma: Install the SDK with an API key and project ID, or ask a coding agent to add tracing with Lemma's published skill (github.com/uselemma/skills). Source
- Braintrust: Add the SDK and enable auto-instrumentation; online scoring and Topics are configured as automations per project. Source
- HoneyHive: Install the SDK or send OpenTelemetry spans, then configure evaluators, sampling and alert thresholds. Source
- Laminar: Two lines to integrate with a supported framework, or a copyable setup prompt for a coding agent. Source
9. Cost tracking
Does it track token usage and spend per trace, model and feature?
Why it matters
Agents loop, retry and call expensive models. If finance is asking why the bill doubled, you need cost per trace and per model you can trust, not just an estimate on a dashboard.
How to check it
- Is cost computed from token counts and list prices, or reconciled with provider bills?
- Can you break cost down by model, agent or user?
How the tools answer
- Lemma: Analytics shows estimated cost from observed token counts and public list prices, which Lemma states are not invoice amounts. Source
- Braintrust: Tracks latency, cost and quality in real time; cost dashboards can be built across traces. Source
- HoneyHive: Monitoring tracks quality, cost, latency, usage and outcomes over time. Source
- Laminar: LLM cost tracking per span and trace, plus custom SQL dashboards. Source
10. Hosting and security
Where do traces live, and what compliance does the vendor list?
Why it matters
Traces hold your users' messages and your agents' outputs. Regulated teams may need hybrid or self-hosted deployment; most others need SOC 2 and clear data isolation.
How to check it
- Is self-hosting or hybrid deployment documented?
- Which certifications are listed in the vendor's trust center?
How the tools answer
- Lemma: Managed cloud, SOC 2 Type II, data isolated per organization. No self-hosted option is listed on its site; ask Lemma if you need one. Source
- Braintrust: Managed cloud; SOC 2 Type II, GDPR and HIPAA listed, with SSO and RBAC. For self-hosting or hybrid terms, check its pricing page or ask Braintrust. Source
- HoneyHive: Managed SaaS, hybrid (traces stored in your cloud) or fully self-hosted on Kubernetes; SOC 2 Type II, GDPR, HIPAA BAA. Source
- Laminar: Managed cloud, or self-host with Docker or Helm charts on AWS or GCP; HIPAA and SOC 2 Type II listed. Source
11. Pricing
Where are prices published?
Why it matters
Prices and plan limits change often. We don't repeat them here; we point you to where each vendor publishes them so you read the current terms.
How to check it
- Check the vendor's pricing page on the day you decide.
- Ask how trace volume, seats and retention are counted.
How the tools answer
- Lemma: No public price list on its site at the time of writing; Lemma offers a demo booking. Check uselemma.ai for current terms. Source
- Braintrust: Published on its pricing page; check there for current plans. Source
- HoneyHive: Published on its pricing page, with a free start; check there for current plans. Source
- Laminar: Published on its pricing page, with a free start; check there for current plans. Source