- 🏆 81.7% pass@1 on the full HumanEval — 134 of 164 problems — run by a local model on this MacBook with the Wi-Fi off.
- ⏱️ 14 minutes wall-clock. Single sample per problem, temperature zero, no retries.
- 📊 Companion number: MBPP (sanitized) 83.3% pass@1 on a 168-problem sample.
- 💸 Total monthly cost after the model download: $0 — launchers are open source so you can reproduce the whole run.
That’s not a slide deck. It’s three of the actual HumanEval problems being solved by the local model on this MacBook — prompt on the left, generated code streaming on the right, real test going green at the end.
Final score: 81.7% pass@1. One hundred and thirty-four out of one hundred and sixty-four. Fourteen minutes wall-clock. Single sample per problem, temperature zero, no retries.
🎯Why anyone should care about this number
The Qwen team didn’t publish HumanEval scores for any Qwen3-Coder variant. They consider the benchmark saturated — and for cloud-served frontier models, fair, it is. They went straight to agentic benchmarks (SWE-bench Verified, BFCL, Aider-Polyglot) and ran them on the flagship 480B sibling.
For the 30B variant — the one that actually fits on a laptop — there were no published HumanEval/MBPP numbers. Until this run.
🔬The methodology, in one screen
| Setting | Value |
|---|---|
| Benchmark | HumanEval — 164 Python tasks (full) |
| Metric | pass@1 (first attempt only) |
| Temperature | 0.0 — deterministic |
| Sampling | single sample per problem, no best-of-N |
| Execution | Python subprocess, 10s timeout |
| Hardware | M5 Max MacBook Pro · 128 GB unified memory |
| Model | Qwen3-Coder-30B-A3B-Instruct-MLX-8bit |
| Network | Wi-Fi OFF the entire run |
| Wall clock | 14 minutes |
🧩Three real problems from the run
Picked from the actual results file — these are the problems that play in the video, with the canonical solutions verified to pass:
- HumanEval/13 · greatest_common_divisor — model output: while b: a, b = b, a % b; return a · 1.76s · PASS
- HumanEval/2 · truncate_number — model output: return number % 1.0 · 1.83s · PASS
- HumanEval/14 · all_prefixes — model output: a list comprehension over range(len(string)) · 1.49s · PASS
And 161 more like them. With 30 failures along the way — that’s the honest part. The full results file is in the GitHub repo if you want to see exactly which ones failed.
📊The companion number — MBPP
I also ran MBPP (sanitized): 83.3% pass@1 on a 168-problem sample. Pass rate stable since n=120; full 427-run was impractical because a few outlier tasks induce very long model responses (10+ minutes each — likely repetitive generation patterns). At n=168 the 95% confidence interval is ±5%, so the number is solid.
🏋️For context — the 480B flagship’s agentic numbers
| Agentic Benchmark | Qwen3-Coder 480B | Claude Sonnet 4 | GPT-4.1 |
|---|---|---|---|
| SWE-bench Verified (500-turn) | 69.6 | 70.4 | — |
| Terminal-Bench | 37.5 | 35.5 | 25.3 |
| BFCL-v3 | 68.7 | 73.3 | 62.9 |
| Aider-Polyglot | 61.8 | 56.4 | 52.4 |
Source: Qwen team’s official blog. The 30B running on this MacBook is a smaller sibling — trades absolute peak agentic ceiling for fitting in 30 GB and running 24/7 on local hardware.
🔁Reproduce it yourself
- Open-source launchers (clone, run): github.com/nicedreamzapp/claude-code-local
- HumanEval dataset: github.com/openai/human-eval
- Hardware: any M-series MacBook with ≥32 GB RAM (128 GB Max preferred for the full 8-bit weights)
- Total monthly cost: $0 after the model download
For law firms, medical practices, and accountants running this on their own hardware — AirGap is what the full private build looks like: the data never leaves your machines, and the whole stack is open source.
— matt
Whose work this stands on
The benchmark is mine. Everything being benchmarked is not.
- MLX and mlx-lm — Apple’s ml-explore team, who made Apple Silicon a serious place to run models.
- Qwen — Alibaba’s Qwen team.
- HumanEval — OpenAI, for the benchmark itself.
- Claude — Anthropic, the cloud baseline everything here is measured against.
If your work is listed here and you would like the wording changed, write to me and I will fix it.