Get startedView as Markdown
Introduction
We turn your agent's production workflows into RL and eval tasks, and your tasks into a benchmark.
General Context reads your agent's production conversations and finds where it goes wrong. Each failure you confirm becomes a task, built from the conversation itself: a synthetic environment where your agent's tools answer with what they returned in production, a simulated person, and a grader drafted from what went wrong. Your tasks together are your benchmark. Run it on every model, prompt or code change, or train on it.
- Production workflowA real conversation your agent handled, with every model call, tool call and result.
- Synthetic environmentYour agent's tools, answering with what they returned in production, with every real name and number replaced.
- TaskThe situation, a simulated person, and a grader that returns a reward from 0 to 1.
- BenchmarkEvery task from your traffic, with a held-out share that only ever measures.
Who does what
You never read transcripts to get a benchmark: we do the reading. You make the calls only you can make.
| You | General Context | |
|---|---|---|
| Connect | Add the SDK or an OpenTelemetry exporter, in about five minutes | Record every call off your request path, so your agent never waits on us |
| Review | Confirm each failure mode we find: a real problem, the agent was right, or known and accepted | Read your traffic, mark the step each failure went wrong, and name the failure mode it shows |
| Tasks | Say what the agent should have done, or what it must keep doing | Build the environment, the person and the grader from the conversation |
| Benchmark | Run it in CI on every change, or train on it | Add tasks as new failures show up, and for busy work the agent gets right |
QuickstartRecord a Python or Node agent's OpenAI, Anthropic, Gemini or Bedrock calls in about five minutes.How it worksFrom one conversation to a task, step by step, and what you decide.A worked exampleOne support email, followed from production to a graded task.Tasks and environmentsWhat a task holds, how it is graded, and how to run or train on it.
What you need
- An invitation to your company's workspace. Your admin can invite the rest of your team.
- An agent that calls OpenAI, Anthropic, Google Gemini or AWS Bedrock from Python 3.9+ or Node 18+, or that already emits OpenTelemetry GenAI spans. Other stacks can import.