Agent-12

The problem

Matt needed to pick a local model to run as a coding agent. Trivia scores don't tell you whether a model will create the file, fix the bug, rename the function and delete the right backup (README.md). Agent-12 runs local models through 20 agent tasks (12 easy, 8 hard) in throwaway sandboxes and grades each one by checking the files the model left on disk.

Constraints

• One machine: an Apple M5 with 128 GB (site/meta.json). Models run through MLX; DeepSeek V4 Flash runs through ds4.c over HTTP.
• Determinism: temperature 0 and fixed token, step and shell time caps per suite (METHODOLOGY.md).
• Judges execute model-written code, so the Python version matters (envcheck.py, MIN_PYTHON = (3, 10)).

What he built

• runner.py creates a fresh sandbox per task, optionally pins a shipped test file by SHA-256 so the model can't edit the test to pass, runs the adapter, and judges the filesystem. Results go to results/*.json with a harness version.
• adapters/ keeps the harness separate from the model. anvil.py runs his own lean agent engine; command.py wraps any agent CLI; anvil_dflash.py swaps in DFlash speculative decoding from the third party mlx-dspark package with everything else held fixed ("One variable moves versus the plain anvil adapter: the drafter").
• validate_judges.py is the gate. Every judge must pass a known-good solution, fail a known-bad one, and pass the good solution again with a UTF-8 byte order mark added. In the file's words: "a broken judge is worse than no judge: it silently converts model skill into noise."

The central finding is about the harness, not the model. METHODOLOGY.md records Qwen3.6-35B scoring 12/12 in the roughly 550-token engine while spending an hour in a thinking spiral inside a 7,800-token, 66-tool harness and producing no files.

Upstream credit: the models (Qwen, Gemma, DeepSeek, Claude Sonnet 5 as a cloud reference), MLX and mlx-lm, mlx-dspark and the Inco AI DFlash 2 drafter, ds4.c.

Hard bugs

• BOM fails correct answers (9b8144e). Reported by an outside contributor. A byte order mark survives .strip() and breaks json.loads, so correct answers written by some editors failed on 7 tasks. answer_text() now reads with utf-8-sig, and the gate re-runs every good solution with a BOM to catch the class of bug. The commit reports 22/22 green with the fix.
• Python 3.9 scoring zero (e74a98b). Contributed by @galashko. Correct solutions using list[str] | None raised TypeError on macOS's system Python. The fix added an interpreter floor, records the interpreter in each result, and bumped the harness to 1.1.0.
• DFlash cache crash and swap (777e73e). mlx-dspark's server prefix cache crashed with a negative dimension error, so he wrote an in-process adapter. An uncapped MLX buffer cache then grew past 100 GB; it is now capped at 2 GB, and the contaminated runs are kept in results/archive/ instead of deleted.

Results (from the repo)

Every row in the README table matches the per-task sums in results/*.json:

• Qwen3.6-35B-A3B 8-bit: easy 12/12 in 64.2 s, hard 8/8 in 125.2 s
• Qwen3-Coder-30B-A3B 8-bit: 12/12 in 42.6 s, 7/8 in 391.5 s
• Gemma 4 31B 4-bit: 11/12 in 92.1 s, 8/8 in 347.9 s
• Qwen3.8-27B 8-bit with DFlash 2: 12/12 in 122.0 s, 7/8 in 401.4 s
• Claude Sonnet 5 through the same engine: 12/12 in 122.0 s, 8/8 in 131.0 s

The Qwen3.6 easy row uses the faster of two 12/12 runs; the other took 553.2 s.

What it shows

He treats evaluation as its own engineering problem: test the judges before trusting them, change one variable at a time, keep failed runs, and report the harness as part of the result. That is the habit a prototyping team needs when deciding which model to build on.