Two different starting points
Lemma starts from production. You instrument the agent, every execution becomes a trace, and Lemma reads completed traces for behavior that hurt the agent's task: an answer that broke the refund policy in its system prompt, a lookup that invented an order ID, a workflow that ended without the ticket it was meant to create. Repeated evidence becomes one issue with its supporting traces.
Braintrust starts from evaluation. Its core is experiments: a dataset, a task, scorers written as LLM judges, code or human review, and a comparison between runs. Production tracing feeds that loop. Online scoring runs your scorers on live traffic, Topics classifies logs to surface patterns, and a trace becomes a dataset row in one click.
Finding failures nobody wrote a check for
This is where the two differ most. Braintrust's online scoring catches what your scorers measure, and Topics helps you notice categories of behavior you then turn into scorers and datasets. Someone still decides what each pattern means and writes the check.
Lemma's detection compares each run with the agent's own instructions and the context you give it about that agent, so a failure can be found before anyone has named it. Lemma's docs say plainly that detection is probabilistic: an issue is a hypothesis backed by traces, and you open the evidence before calling it confirmed.
Evals and release gating
Braintrust is the stronger choice if you need offline evals. Versioned datasets, side-by-side prompt and model comparison, and scored experiments are the centre of the product.
Lemma does not offer offline evaluation. It evaluates production usage: after a fix ships, it creates an online eval from the failure so you know if the problem comes back. If your release process runs on pre-production experiments, that gap matters.
Triage and fixing
Both give a coding agent access through an MCP server. Lemma's unit of work is the issue, with status, priority, supporting traces, Slack alerts, Linear tickets and webhooks. Braintrust's is the trace, the review queue and the dataset, with Loop to investigate traces in natural language and alerts to Slack, PagerDuty or webhooks.