Methodology

Task design: Each task is a real engineering or research problem that would take an expert a day or more. There are no visible test suites and no way to verify a solution without building and running the whole thing. The model decides what to build, writes its own checks, and stays coherent for hours.
Harness: Every model runs in Horizon, our Pi-based harness, on the highest reasoning setting, with one instruction prompt and three tools: bash, read and submit. Like FrontierSWE's Proximus and Terminal-Bench's Terminus, it stays minimal and identical across models, with no vendor scaffolding or per-model tuning, so results reflect the model rather than the harness. Like Proximus, it adds what multi-hour runs need: it compacts context when it fills up, tells the agent how much time is left, and lets it read images such as plots and renders. A run ends only when the agent submits twice, and automatic checkpoints let an interrupted run resume.
Environment: Each task runs in an offline container on the same Debian base image (python:3.12-slim), with its own pinned toolchain and fixed CPU and memory, under a 4 to 12 hour limit.
Grading & scoring: A hidden verifier rebuilds the final artifact in a fresh environment and tests it on inputs the agent never saw. A run passes only if every required check passes; tasks with a performance target also get a 0 to 100 score. Each model runs every task up to 4 times (a few task and model pairs have fewer graded runs), and pass@1 is each task's pass rate averaged across tasks. Runs lost to infrastructure errors are rerun, not scored as fails.
Task QA: Each task undergoes reward-hacking audits before inclusion in the set. Tasks where the agent could game the verifier, such as editing tests, hard-coding outputs, or calling the reference, were revised or dropped.