I Took Down Six of My Own Benchmark Videos. Then I Built the Local Agent Leaderboard.

🕵️I caught my own judge lying

My benchmarks are filesystem-judged: the model does real work in a sandbox — create the file, fix the failing test, rename a function across files — and a script checks the artifacts afterward. The model’s opinion of its own work is never consulted. I thought that made the scores bulletproof.

Then I built a validation gate: before any judge is allowed to score a model, it has to pass a known-good reference solution and fail a plausible known-bad one. Twenty tasks, forty checks. Nineteen judges passed. One didn’t.

The hard-suite cache task (build an LRU cache with per-item TTL) had a probe that asserted an eviction order its own spec forbids. I wrote a spec-perfect solution by hand and the judge failed it. Which means both models in my published video had been scored down by a bug in my test. The video said 6/8 and 6/8. The truth, under the repaired judge: 7/8 for Qwen3-Coder and a perfect 8/8 for Qwen3.6. My broken test made both models look worse — that’s the lucky direction of this failure, and it still wasn’t good enough to leave up.

🏟️So I built the league

Agent-12 is a public leaderboard for the question r/LocalLLaMA asks every single week: which model can actually be my agent on my own hardware? Not trivia scores. Twenty sandboxed agent tasks — 12 everyday chores, 8 hard reasoning problems — judged only by what lands on disk. Temperature 0, fixed step caps, fresh sandbox and fresh conversation per task, hash-pinned test files so a model that “fixes” a failing test by editing the test scores zero.

And the part I now consider non-negotiable: every judge is validated against known-good and known-bad reference solutions before it scores anything, and every model runs the full gauntlet three to four times. A lucky run counts for nothing. The repo — tasks, judges, runner, validation gate, methodology — is open, and any agent CLI can plug into the same tasks through a generic adapter.

🥊Day one: five local fighters, one cloud yardstick

All local rows ran on one Apple M5 with 128 GB unified memory, through my lean ~550-token terminal engine.

The repeat runs earn their keep. Qwen3-Coder failing the same task three sweeps in a row is a reproducible fingerprint you’d never see in a single-run benchmark. Gemma’s misses moving around between runs is a stability finding in its own right.

😮The row that surprised me

I ran Claude — the frontier cloud model — through the same engine, same tasks, same judges, as a clearly-labeled reference point. It went a perfect 20/20. So did my local champion. And on wall-clock, the local model won: 64s vs 122s on the easy suite, 125s vs 131s on the hard suite (cloud times include network round-trips — which is the honest end-to-end experience of using it).

That’s not a claim, it’s a row on a public board with the runner that produced it.

⚙️The harness is half the story

One more result, because it’s the reason this board exists. The same Qwen3.6 that goes 20/20 in my lean terminal engine spent an hour in a 32,000-token thinking spiral inside a heavyweight agent harness (7,800-token system prompt, 66 tools) and produced zero files. Same weights. Same machine. The harness was the only variable.

The ecosystem is currently running the other direction — plugging local models into big cloud-grade harnesses and blaming the model when it chokes. There’s fresh academic backing for harness-model matching mattering this much (AgentMeter, arXiv 2606.21140) — but it was measured on cloud CLIs. Nobody was measuring it on hardware you can buy. Now somebody is.

🔮What happens next

Every new model that drops gets a row and a fight video. The board lives at nicedreamzapp.github.io/agent12, the code at github.com/nicedreamzapp/agent12. If you think your favorite model belongs on it, tell me — it’ll get the same twenty tasks, the same validated judges, and the same number of chances to prove it wasn’t luck.