- ⥠My local coding agent got noticeably faster overnight â so we benchmarked it instead of guessing.
- đ Gemma’s sliding-window RotatingKVCache silently kills prompt-cache reuse once your transcript outgrows 1024 tokens â every turn re-prefills everything.
- đ§ Swapping every layer to a plain KVCache cut time-to-first-token from ~7 s to 0.36 s, with byte-identical outputs.
- đ§ A lean 550-token prompt beat the giant harness: 12 for 12 on a machine-checked eval.
This morning my local coding agent started answering noticeably faster and I could not explain why. I do not like speedups I cannot explain, so we benchmarked it instead of guessing. What came out of that is the biggest single improvement my local setup has ever gotten, and a bug that is probably sitting in your setup too if you run Gemma-family models on Apple Silicon.
đ§°The setup
I maintain claude-code-local, a repo for running coding agents against local models on a Mac. No cloud, no API key. The original approach pointed Claude Code at a local MLX server through a proxy. It works, and it is still in the repo. But Claude Code was designed for cloud models: its system prompt is tens of thousands of tokens, and parts of it change every turn. A local model pays for that twice, once prefilling a huge prompt, and again because a prompt whose head keeps changing defeats KV cache reuse completely.
So we built the obvious alternative. A small native engine, about 900 lines of Python on mlx-lm. Fixed 550-token system prompt, the same tools (bash, read, write, edit, glob, grep), and a KV cache that gets trimmed to the shared prefix each turn so only the new tokens are ever prefilled.
đThe bug
Benchmarking surfaced something I did not expect. Short conversations were fast, exactly as designed, about a third of a second to the first token. But past a certain conversation length, every turn suddenly cost six and a half to seven seconds, as if the cache did not exist. It was not gradual. It was a cliff.
The cliff turned out to be Gemma’s sliding window attention. Gemma-family models give five out of every six layers a RotatingKVCache capped at the window size, 1024 tokens on Gemma 4. The moment your transcript outgrows the window, those rotating caches report themselves as untrimmable, and mlx-lm’s prompt cache reuse silently dies. Every turn re-prefills the entire transcript. The longer your session, the worse it gets, which means the failure lands exactly where caching matters most. There is no error, no warning, nothing. It just gets slow.
If you are building a Gemma-based agent on mlx-lm, check for this. You probably have it right now.
đ§The fix
Give every layer a plain KVCache instead. That sounds like it should change the model’s output, but it does not: the sliding window attention mask is what enforces the window. The cache type only decides what gets stored. We verified this the honest way, with greedy decoding, same conversation, stock caches versus plain caches, and the outputs were byte-identical on every turn.
The numbers, on a Gemma 4 31B (4-bit) with a 4,500-token conversation:
| time to first token | |
|---|---|
| stock rotating cache | 6.5 to 7.2 s |
| plain KV cache | 0.36 s |
That is roughly 20 times less waiting per turn, on the same model and the same MacBook. The trade is that KV memory now grows with the transcript instead of capping at the window, so the engine shows a live context meter, and an environment variable restores stock behavior if you would rather have the memory ceiling.
đ§ But did removing the big harness make it dumber?
Fair question. Claude Code’s giant prompt exists for a reason, just not for a 31B model. We built a 12-task eval: create and run scripts, fix a failing test, rename across files, escape-heavy file content, precise edits, CSV work. All machine-checked by actually running the results, at temperature 0.
Qwen 3 Coder went 12 for 12 with the bare 550-token prompt. Gemma went 11 for 12, and its one failure was instructive: asked to write a file full of quotes and backslashes, it piped the content through shell echo, and sh’s echo silently collapsed the backslashes. A five-line prompt rule, never write file contents through the shell, always use the write and edit tools, took it to 12 for 12 with no regressions.
So no. For models this size, less harness turned out to be more capability, as long as the few rules you do include are aimed at failures you actually observed.
đŚWhere it all lives
Everything shipped today in the repo: the engine, the fix, the benchmark script so you can reproduce the numbers on your own machine, and the write-up. Existing Claude Code launchers are untouched, this is a second path and not a replacement. Credit where it is due, the prompt cache trim fix contributed in PR #46 is what made the deeper rotating cache problem visible at all.
Local AI on a Mac keeps surprising me. The models were already good.