Domains
Scientific Computing
Models are increasingly capable of scientific work, and benchmarks like Terminal-Bench Science are starting to measure it. These tasks push further, into problems where approximate answers don't pass: solving a stiff PDE to one part in a million, forecasting groundwater nitrate through decades of lag, and inferring matrix structure that linear-algebra rewrites can safely rely on.
| # | Model | Pass@1 | Avg tokens | Avg steps |
|---|---|---|---|---|
| 1 | GPT-6.1 Sol max | 44.8M | 145 | |
| 2 | GPT-6 Astra xhigh | 48.8M | 150 | |
| 3 | Claude Sonnet 5.5 max | 186.4M | 414 | |
| 4 | Claude Opus 5.5 max | 94.6M | 231 | |
| 5 | GPT-5.6 Sol max | 140.9M | 454 | |
| 6 | Qwen3.8 Max xhigh | 90.1M | 266 | |
| 7 | Muse Spark 1.3 max | 113.9M | 477 | |
| 8 | Kimi K3 max | 61.8M | 263 | |
| 9 | Gemini 3.8 Flash high | 92.3M | 325 | |
| 10 | GLM 5.3 Flash max | 117.0M | 447 | |
| 11 | Grok 4.7 xhigh | 35.7M | 158 |
Results by task
| Success % | Task | Astra | Sol 6.1 | Sonnet 5.5 | Opus 5.5 | Qwen3.8 | Sol 5.6 | Kimi K3 | Grok 4.7 | Muse 1.3 | GLM 5.3 | Gemini 3.8 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 45% | Cahn-Hilliard Solver | 3/4 | 4/4 | 4/4 | 4/4 | 1/4 | 2/4 | 1/4 | 0/4 | 1/4 | 0/4 | 0/4 |
| 18% | PyTensor Matrix Assumptions | 4/4 | 3/4 | 0/4 | 1/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 |
| 14% | SuperCollider Synth Resampling | 1/4 | 3/4 | 2/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 |
| 7% | Groundwater Nitrate Legacy Lag | 1/4 | 2/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 |
Example task · 12 hours · C++
Task overview. The Cahn-Hilliard equation describes how a mixture separates into two phases, like oil and water settling out. Here it is forced and posed on an eccentric annulus, a ring whose hole is off-center, which makes the geometry and the numerics hard. The agent gets the problem only through black-box data oracles and has to write a C++ solver that matches the true solution to one part in a million, within fixed CPU and time limits.
How it's graded. The sources are built with a fixed command and run once per hidden instance under CPU and wall-clock limits scaled to the grading machine's speed. The solver passes only if the worst relative L2 error over all graded instances is at most 1e-6.
- Toolchain
- g++ 12.2 with OpenMP, Eigen 3.4, CMake 3.25, gdb 13.1, Python 3.12 (standard library)
- Compute
- 4 CPUs · 8 GB RAM
- Time limit
- 12 hours
Other example tasks in Scientific Computing
- Groundwater Nitrate Legacy Lag
- Nitrate from fertilizer can take decades to reach groundwater, because it is stored in the soil above the aquifer and in the aquifer itself. Predict how groundwater nitrate responds to proposed fertilizer programs from monitoring data, accounting for that multi-decade lag. The task is complete when the predictions are correct on the hidden program cases, including every fertilizer-cut case.
- PyTensor Matrix Assumptions
- PyTensor is the tensor library behind PyMC. Build a system that infers structural facts about matrices (diagonal, triangular, symmetric, positive definite) across a computation graph and lets the linear-algebra rewrites use them. The task is complete when every inferred fact is sound on hidden graphs and the rewritten graphs still compute the right answer.
- SuperCollider Synth Resampling
- SuperCollider is a real-time audio synthesis platform. Let a single synth run at its own block size and sample rate, resampling audio at its boundaries, in both of SuperCollider's servers. The task is complete when audio converts exactly at every boundary and real-time scheduling stays intact under the hidden tests.