How it works
From a real conversation to a benchmark: traces, environments, tasks and benchmarks.
Every task starts as a conversation your agent really had. The console follows it from there in four steps, one page each: Traces, Environments, Tasks and Benchmarks. Your team and ours work in the same workspace and see the same things.
1. Traces
Add the SDK, send OpenTelemetry, or import conversations you already store. Every model call, tool call and reply is recorded, including what each tool returned, off your request path, so your agent never waits.
Reviewing a conversation gives it one verdict. Report a problem puts one at the front of the review queue.
| Verdict | Means |
|---|---|
| Went wrong | Marked at the first wrong step, with the failure mode, severity, what went wrong at each step and what the agent should have done |
| Fine | Nothing wrong |
| Good example | Handled well, and kept as a reference past the 90-day retention |
| Unsure | Needs a second look |
Each conversation is filed under a workflow, the kind of work it was. The Workflows tab shows each one's share of traffic, how often the agent gets it right, and the tasks guarding it. Failures that repeat are named as one failure mode, under the Failure modes tab, with the conversations that show it, most common first. Confirm whether each is a real problem, the agent was right, or known and accepted: a conversation marked as gone wrong counts as failed unless the agent was right.
2. Environments
An environment is the world the agent works in for a set of tasks: its tools, the records they return, and its instructions. Build one from traces with Add this trace on a conversation, then fill in what production never showed, such as a record no conversation looked up. The page lists what it cannot answer yet. Many tasks share one environment, so fixing a tool once fixes every task on it.
3. Tasks
A task does one of two jobs. It catches a failure: for a confirmed failure mode, say in a sentence what the agent should have done, and it becomes a task built from the conversations behind it. Or it keeps it right: for work that is common and matters, a conversation the agent handled well becomes a task, so a later change cannot quietly break it. Next up suggests the busy workflows that have nothing keeping them right.
- Environment: a shared environment, or the recordings of the task's own conversations. For an agent that looks things up and talks to people, a person writes in and shares what they know only when asked.
- Grader: what decides a run passed. By default, a rubric drafted from the review: what the agent should have done, and each step it got wrong (or, for a success, right) in production. Sharpen it with what you know of the work, or write it as code when the outcome can be checked directly.
- Reference run: the production run itself, which must fail a task built from a failure and pass one built from a success. It checks the grader before anything is spent.
An agent that does other work, such as writing code or building something, can use its own environment and grader instead, run by your code. Every real name, contact and account number is replaced before a task is ready. Runs, under Tasks, shows every run and the commands that make them. Tasks and environments covers what a task holds and how it is graded.
4. Benchmarks
A benchmark is sets of ready tasks, the agents compared on them, and the axes to read results by: workflow, origin, severity, tag or split. Freezing a version pins exactly which tasks a result was measured on, and flags any task edited since. About one task in five is held out: it only ever measures your agent and is never used to change it, so its score stays honest.