# Tasks and environments > What a task holds, how it is graded, and how to run it as an eval or an RL environment. A task is one situation from your traffic, made runnable: a prompt, an environment the agent works in, and a grader that decides whether the run passed. Every task is an episode with a reward at the end, so the same tasks serve evaluation and RL training. Nothing in a task is a real customer record. A task either **catches a failure**, a confirmed failure mode the task catches if it comes back, or **keeps it right**, a conversation your agent handled well in work that is common and matters, which the task keeps that way. Tasks that keep it right teach what good looks like and stop a fix for one case from breaking another. ## Environment and grader kinds Every task names the kind of environment and grader it needs. Two are built in, for an agent that looks things up and talks to people. Any other kind is your own code. | Kind | What it is | | --- | --- | | Environment `replay` | Your agent's tools, answering with what they returned in production, and a model playing the person who wrote in | | Grader `rubric` | A judge model grades each criterion, with a reason, and the reward is the share met | | Your own | Any other name, such as `game-workspace` or `play-check`. Your code builds the environment or grades the run, and reads the fields the task gives it | ## The replay environment A tool call with the arguments production used gets the recorded result, and a near match gets the closest one. A lookup production never made is answered as unrecorded and counted, never invented, so a run that strays far from the recording says so. Writes, such as sending a message or opening a request, are logged for the grader. It runs anywhere, with no access to your systems: in CI, on a laptop, on every pull request. A model plays the person who wrote in, from the task's brief. It knows every fact, and shares the ones marked `when_asked` only when your agent asks, so your agent has to ask. ## What a task holds | Part | What it holds | | --- | --- | | Prompt and expected | What the agent is given, and what it should do (a failure) or did (a success) | | Environment | Its kind, and what it needs. For replay: your agent's tools and every result they returned, the person's brief and facts | | Grader | Its kind, and what it needs. For rubric: ground-truth facts and criteria on what the person was told and what your agent did | | Reference runs | Known runs, such as the production one, and whether each should pass | | Agent | What your agent had in production: its instructions, models and tools | | Origin, split, tags, workflow | Failure or success, train or held out, labels to slice by, and the kind of work it came from | ## Grading and reward The rubric grader has a judge model grade every criterion against the conversation, your agent's writes, and the task's facts, with a one-sentence reason for each. The reward is the share of criteria met, from 0 to 1. A run passes only when every criterion does. Your own grader returns the same: a reward from 0 to 1, and whether it passed. A rubric starts from the review, not from a rule book: what the agent should have done, and each step the review marked wrong (or right, for a success). Tighten it from there with what the conversation and its tool results show and what you know of the work. When the outcome can be checked directly, such as a record that must or must not exist, write the grader as code instead. **Tasks** shows each agent's runs: the share passed, and the tasks it passed every time, since a task passed four times in five still fails one customer in five. ## Check the graders first `evals check` grades every task's reference runs without running any agent. A failure's production run must not pass, and a success's must. If a grade disagrees, the grader is wrong, and the command says which task and exits 1. ``` generalcontext evals check benchmark.json OK shipping-fee-not-escalated production expect fail, got fail (reward 0.5) 1 reference run checked, 0 disagree with expect. ``` ## Run it as an eval Your agent: ```bash pip install generalcontext openai anthropic generalcontext benchmark pull --out benchmark.json generalcontext evals run benchmark.json --agent my_agent:handle --trials 3 ``` A reference model: ```bash generalcontext evals run benchmark.json --agent openai:gpt-6.1-terra --trials 3 ``` `--agent module:function` runs your own code. It is called as `handle(messages, toolbox)`: run your agent loop on `messages`, call tools through `toolbox.call(name, arguments)` instead of your real systems, and return the final text. `toolbox.openai_tools()` and `toolbox.anthropic_tools()` give the tools in each provider's format. `--agent openai:`, `claude-code:` (the Claude Code CLI on its signed-in plan) or `anthropic:` runs a model on the instructions recorded from your agent, a quick baseline. ```python import json from openai import OpenAI client = OpenAI() def handle(messages, toolbox): convo = [{"role": "system", "content": "You are our support assistant."}, *messages] for _ in range(8): reply = client.chat.completions.create(model="gpt-6.1-terra", reasoning_effort="none", messages=convo, tools=toolbox.openai_tools() or None).choices[0].message if not reply.tool_calls: return reply.content or "" convo.append(reply.model_dump(exclude_none=True)) for call in reply.tool_calls: result = toolbox.call(call.function.name, json.loads(call.function.arguments or "{}")) convo.append({"role": "tool", "tool_call_id": call.id, "content": result if isinstance(result, str) else json.dumps(result)}) return "" ``` The model playing the person and the judge run on OpenAI by default, so set `OPENAI_API_KEY`, or choose others with `--person` and `--judge` (`anthropic:`, `claude-code:`). Results go to `runs/runs.json` in the `generalcontext.runs/v1` format, one file per trial beside it. Add `--upload` to send them to your workspace with a key that can run the benchmark, or use **Import runs.json** under **Tasks**, **Runs**. Run it in [CI](https://generalcontextlabs.com/docs/cli) to catch a regression before it ships. ```bash generalcontext evals run benchmark.json --agent my_agent:handle --trials 3 --upload ``` ## Your own environment and grader For a task whose environment or grader is your own kind, name your code on the command line. A task file never names code to run, so a downloaded benchmark cannot run anything you did not choose. ```bash generalcontext evals run benchmark.json --agent my_agent:build \ --environment my_env:workspace --grader my_env:play_check ``` `--environment module:function` is called with the task and returns an environment with `tools`, `call(name, arguments)`, `start()` for the first message, `respond(said)` returning the next message and whether it is done, and optionally `state()` and `close()`. An environment that is not a conversation returns done at once and puts what the agent produced in its state. `--grader module:function` is called with the task and the episode (transcript, tool calls, output, state) and returns `{"reward": 0..1, "passed": true|false}`, with criteria and a note if you have them. A task whose kind has no code given is skipped with the reason, never guessed. For example, an agent that builds games can use an environment that runs the build in a fresh workspace, and a grader that opens the game in a browser, plays it with a few taps, and checks it boots, draws and responds. ## Use it as an RL environment Each task is an episode: it starts from the opening message, your policy acts through the tools and talks to the person, and the grader returns a reward at the end. The same harness runs from Python: ```python from generalcontext import evals tasks = [task for task in evals.load_tasks("benchmark.json") if task["split"] == "train"] person = evals.model("openai:gpt-6.1-terra@none") judge = evals.model("openai:gpt-6.1-sol@high") for task in tasks: result = evals.run_task(task, my_agent, person, judge) result["reward"] # 0 to 1, the share of rubric criteria met result["criteria"] # each criterion, passed or not, with the judge's reason result["transcript"] # the episode; result["tool_calls"] has every call ``` `my_agent` has the same shape as `--agent module:function`, such as `from my_agent import handle as my_agent`. If your agent raises, the episode ends there with reward 0 and the exception in `result["error"]`, so check it before you train on the reward. Training runs on your side, with your own stack. > **Warning:** Train only on the `train` split. Held-out tasks are for measuring: once they shape your agent, their scores stop meaning anything. ## The task file `benchmark pull` downloads your ready tasks as one `generalcontext.benchmark/v1` file of `generalcontext.task/v1` tasks. New fields may be added to the format, never renamed or removed: read the ones you need and ignore the rest. ```json { "format": "generalcontext.task/v1", "slug": "shipping-fee-not-escalated", "origin": "failure", "split": "held_out", "tags": ["scope"], "workflow": { "name": "Refund requests" }, "prompt": "Your checkout glitched when I paid on March 14…", "expected": "Refund only the duplicate charge; decline the February fee.", "agent": { "instructions": "…", "models": ["…"] }, "environment": { "kind": "replay", "person": { "brief": "…", "facts": [{ "text": "…", "disclose": "when_asked" }] }, "tools": [{ "name": "orders__get", … }], "observations": [ { "tool": "orders__get", "arguments": { … }, "result": { … } } ] }, "grader": { "kind": "rubric", "facts": ["…"], "criteria": [ { "id": "no-february-task", "claim": "No task asks billing to…" } ] }, "reference": [{ "label": "production", "expect": "fail", "transcript": […] }] } ``` ## Full simulations A record production never looked up can be added to a shared [environment](https://generalcontextlabs.com/docs/how-it-works) by hand. Some work needs more: a check on exactly what changed in your systems. For those, we can build a full simulation of the systems involved with your team, as an environment of its own kind. The same tasks run there, and the grader adds checks on the end state.