Domains

Software Engineering

The hard part of a real feature is rarely the core idea. It's everything the feature touches. Coroutines in the Haxe compiler have to survive every loop, try/catch, and closure. Mergeable containers in Loro have to converge on every replica, whatever order the edits arrive in. Wide-column entities in RocksDB have to come back intact after flush, compaction, and garbage collection. Every task starts from a real upstream commit, and hidden tests probe exactly those corners.

Software Engineering results

Software Engineering results Pass@1 results
#	Model	Pass@1	Avg tokens	Avg steps
1	GPT-6 Astra xhigh	60.42%	36.1M	136
2	Claude Opus 5.5 max	50%	91.8M	253
3	GPT-6.1 Sol max	50%	29.3M	129
4	Claude Sonnet 5.5 max	50%	147.3M	352
5	Qwen3.8 Max xhigh	12.5%	91.8M	328
6	Grok 4.7 xhigh	6.25%	62.2M	252
7	Kimi K3 max	4.17%	93.1M	411
8	GPT-5.6 Sol max	2.08%	201.7M	622
9	GLM 5.3 Flash max	2.08%	244.1M	941
10	Muse Spark 1.3 max	2.08%	132.4M	524
11	Gemini 3.8 Flash high	0%	153.0M	487

How it's scored

Results by task

Task results by model. Use arrow keys to inspect cells.
Success %	Task	Astra	Sol 6.1	Sonnet 5.5	Opus 5.5	Qwen3.8	Sol 5.6	Kimi K3	Grok 4.7	Muse 1.3	GLM 5.3	Gemini 3.8
43%	pixi Rich Platform Specifiers	4/4	1/4	3/4	4/4	4/4	1/4	1/4	0/4	1/4	0/4	0/4
34%	RocksDB Wide-Column Blobs	4/4	4/4	4/4	0/4	0/4	0/4	0/4	2/4	0/4	1/4	0/4
34%	systemd OCI Image Pull	3/4	4/4	4/4	4/4	0/4	0/4	0/4	0/4	0/4	0/4	0/4
25%	MiniZinc Named Arguments	4/4	2/4	4/4	1/4	0/4	0/4	0/4	0/4	0/4	0/4	0/4
23%	Turso MVCC Sequences	2/4	3/4	3/4	2/4	0/4	0/4	0/4	0/4	0/4	0/4	0/4
20%	Loro Mergeable Containers	4/4	1/4	1/4	3/4	0/4	0/4	0/4	0/4	0/4	0/4	0/4
18%	Ash Temporal Resources	1/4	1/4	3/4	2/4	0/4	0/4	1/4	0/4	0/4	0/4	0/4
18%	Logseq DB Sync over Cloudflare	1/4	3/4	0/4	2/4	1/4	0/4	0/4	1/4	0/4	0/4	0/4
16%	OCaml Polymorphic Parameters	2/4	2/4	0/4	3/4	0/4	0/4	0/4	0/4	0/4	0/4	0/4
11%	Coroutines in Haxe	2/4	1/4	0/4	2/4	0/4	0/4	0/4	0/4	0/4	0/4	0/4
9%	egglog Container Proofs	0/4	0/4	2/4	1/4	1/4	0/4	0/4	0/4	0/4	0/4	0/4
9%	mypy Closed TypedDicts	2/4	2/4	0/4	0/4	0/4	0/4	0/4	0/4	0/4	0/4	0/4

Example task · 4 hours · OCaml

Coroutines in Haxe

The compiler rewrites each coroutine into a numbered state machine.

Source (what the programmer writes)

@:coroutine
function sum(a, b, c): Int {
  var x = double(a);
  var y = double(b);
  return x + y + c;
}

Compiled (what the compiler produces)

// state = 0, saved: a, b, c, x, y
case 0:             // entry
  state = 1;
  return double(a); //  pending
case 1:             // resumed
  x = result;
  state = 2;
  return double(b); //  pending
case 2:             // resumed
  y = result;
  return x + y + c; //  done

sum(1, 2, 3) state machine with 3 states

Task overview. Haxe is an open-source language whose compiler is written in OCaml, which turns a program into many targets (C++, Python, etc). This task asks the agent to ship coroutines to the compiler, starting from the upstream commit. Each language construct has to survive a pause: loops, break and continue, switch, try/catch, closures and evaluation order.

How it's graded. The compiler is rebuilt from the agent's commit in a separate environment and tested on both the eval interpreter and JavaScript: 15 of the project's existing test suites with no new failures, every test in hxcoro's own suite, and 177 hidden generated programs in six families (sequential flow, value positions, exceptions, captured variables, concurrent tasks and failures), compared on printed output. A run passes only if every suite passes and each family reaches 90%.

Toolchain: OCaml 5.3.0, opam, dune, Node.js 18, neko 2.3 (haxelib)
Compute: 4 CPUs · 8 GB RAM
Time limit: 4 hours

Other example tasks in Software Engineering

MiniZinc Named Arguments: MiniZinc is a constraint-modelling language used in operations research and scheduling. Add named arguments and default values to its function and predicate calls. The task is complete when the compiler parses, type-checks and evaluates named and default arguments correctly on the hidden test suite.
Loro Mergeable Containers: Loro is a CRDT library for building local-first collaborative apps. Implement mergeable child containers so that when two users concurrently create a named child inside the same parent, the results converge to a single shared container with correct persistence. The task is complete when concurrent creation, nested updates and reload all pass the hidden convergence tests.
OCaml Polymorphic Parameters: OCaml is a functional language with a strict static type system. Add polymorphic function parameters to its compiler, so a function can take an argument it uses at two different types. The task is complete when hidden programs type-check or fail exactly as the language rules require, and every program accepted today keeps its type and meaning.