A worked example
One support email, followed from production to a graded task.
This example comes from a support agent for an online store that answers customers by email. The company, people and records are synthetic, made for these docs, but every step is one a real task goes through, and the run below is real.
1. What happened in production
A customer emailed support:
Your checkout glitched when I paid on March 14 and I was charged twice. Please refund the extra charge.
The agent found the order and its two $129 payments, 40 seconds apart, and the status page's incident for that evening's payment retries. It asked the billing team to refund the duplicate, with that evidence, which was right.
Then she asked whether billing could also look at the $18 express shipping fee on her February order. She had chosen express at checkout, and the order arrived on the promised date. The agent opened a second billing task to review that fee, and told her billing would.
2. What the review found
The February order and the shipping article, both in the agent's own tool results, showed nothing went wrong in February: express was a paid upgrade, and it arrived on time. We marked the step where the agent opened the February task as wrong: "Asked billing to review a fee nothing went wrong with." We named the failure mode Escalates fees nothing went wrong with, in the out of scope category.
3. What the company said
On the Failure modes tab, the company answered Yes, fix it and added: "Billing should only get requests where something actually went wrong. Support declines the rest." Then, under Make this a task, what the agent should have done:
Ask billing to refund only the duplicate March 14 charge, with the incident as evidence. Decline the February express fee plainly, because it was a paid upgrade that arrived on time, and open no task for it.
4. The task
| Part | In this task |
|---|---|
| Environment | The agent's 7 tools across the helpdesk, orders, the status page, the knowledge base and team tasks, answering with the 10 results production returned: both orders with their payments, the incident, the shipping article |
| Person | The customer. She volunteers that she wants the February fee looked at, and explains why she chose express only if asked. |
| Grader | The four criteria below, drafted from the review and sharpened from the tool results |
| Split | Held out, so it only ever measures |
| Criterion | Must be true at the end |
|---|---|
march-refund-routed | A billing task asks to refund the duplicate March 14 charge, citing the incident or the two payments. |
no-february-task | No task asks billing to refund or review the February express shipping fee. |
explains-february | The desk tells her the February fee is not refundable because express was her choice and it arrived on time. |
no-promise | The desk does not say any refund has already been made. |
evals check grades the production run first: it must fail, and it does, with a reward of 0.5.
5. Running it
We ran a reference model on the task three times. Every run made the same mistake production did:
$ generalcontext evals run benchmark.json --task shipping-fee-not-escalated \
--agent openai:gpt-6.1-terra@low
FAIL shipping-fee-not-escalated openai:gpt-6.1-terra@low #1
2/4 criteria; failed: no-february-task, explains-february;
2 lookups not in the recordingIts reward was 0.5. It routed the duplicate charge with the evidence and promised nothing, but again opened a billing task for the February fee. The judge's reason for no-february-task: "The agent created a billing task to review the $18 express shipping fee on February order 10318." Once the agent is fixed, the task stays in the benchmark, so the mistake cannot come back unnoticed.
"2 lookups not in the recording" means the model looked up something production never did. The environment answers those as unrecorded instead of inventing a result, and counts them, so a run that strays far from the recording says so.