Domains
Engineering & Robotics
Models are starting to operate in the physical world, where sensors drift, loads shift, and faults arrive without warning. These tasks put the agent to work on physical systems over hours: fusing GNSS, IMU and wheel data on a microcontroller through tunnel outages, racing unseen circuits from screenshots alone, and carrying CAD assemblies between file formats without losing a part.
| # | Model | Pass@1 | Avg tokens | Avg steps |
|---|---|---|---|---|
| 1 | Claude Opus 5.5 max | 99.4M | 262 | |
| 2 | Claude Sonnet 5.5 max | 114.2M | 272 | |
| 3 | GPT-6 Astra xhigh | 24.8M | 125 | |
| 4 | GPT-6.1 Sol max | 17.5M | 100 | |
| 5 | GPT-5.6 Sol max | 70.4M | 330 | |
| 6 | Gemini 3.8 Flash high | 157.9M | 486 | |
| 7 | Qwen3.8 Max xhigh | 107.2M | 330 | |
| 8 | GLM 5.3 Flash max | 181.7M | 707 | |
| 9 | Muse Spark 1.3 max | 189.1M | 831 | |
| 10 | Kimi K3 max | 77.7M | 373 | |
| 11 | Grok 4.7 xhigh | 213.5M | 716 |
Results by task
| Success % | Task | Astra | Sol 6.1 | Sonnet 5.5 | Opus 5.5 | Qwen3.8 | Sol 5.6 | Kimi K3 | Grok 4.7 | Muse 1.3 | GLM 5.3 | Gemini 3.8 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 34% | Inertial Navigation on MCU | 3/4 | 4/4 | 4/4 | 4/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 |
| 25% | CAD Assembly Exchange | 2/4 | 2/4 | 4/4 | 3/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 |
| 7% | Vision-Only Racing Driver | 2/4 | 1/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 |
| 2% | ns-3 6LoWPAN Neighbor Discovery | 0/4 | 0/4 | 0/4 | 1/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 |
Example task · 12 hours · C
Task overview. Cars lose GNSS in tunnels and city canyons, and the telematics chip still has to know where it is. The task is to write the navigation engine in C for a 64 MHz Cortex-M4F. It fuses consumer GNSS, a MEMS IMU with a loosely known mount and wheel ticks with a stale scale factor, using float32 math and no heap. It has to hold position and heading through GNSS outages and multipath on real road drives, within per-call deadlines and tight CPU, stack, RAM and flash budgets.
How it's graded. The source is built with fixed flags, linked into the firmware harness and run on an emulated Cortex-M4F with a cost meter, on 39 held-out real drives. It passes only if every drive stays within every budget, at least 36 of the 39 drives are within the accuracy tolerance for each metric, and no more than one metric exceeds twice its tolerance.
- Toolchain
- arm-none-eabi-gcc 12.2 (newlib), QEMU 10.1 (mps2-an386) with metering plugin, Python 3.12 with NumPy
- Compute
- 4 CPUs · 8 GB RAM
- Time limit
- 12 hours
Other example tasks in Engineering & Robotics
- CAD Assembly Exchange
- CadQuery is a Python library for building CAD models in code. Rebuild its assembly import and export so nested instances, placements, names and colors survive an exact round trip through STEP and its native formats. The task is complete when assemblies round-trip intact through OpenCascade's document model on the hidden cases.
- Vision-Only Racing Driver
- A racing game exposes only screenshots. Write a driving controller that races unseen 3D circuits from pixels alone. The task is complete when every one of 24 races on freshly generated tracks finishes no slower than an oracle driver that sees the full game state.
- ns-3 6LoWPAN Neighbor Discovery
- ns-3 is the standard open-source network simulator. Implement IPv6 neighbour discovery for 6LoWPAN (the protocol that puts IPv6 on tiny wireless sensors) in ns-3, including address registration, duplicate detection and the compressed wire formats. The task is complete when simulated sensor networks discover neighbours correctly on the hidden test topologies.