Free Shipping on all orders · Priority Mail Shipping with fee of $8.00
Software / Case studies / claude-code-local
Case study 3 of 9

claude-code-local

A stdlib Python server that runs open-weights models through MLX and answers Anthropic's Claude Code CLI in its own API format, so Claude Code runs on a Mac with no cloud model. Fixes to KV cache trimming, prefill memory (34.2 GB to 20.9 GB peak) and sliding-window cache reuse (0.36 s to first token) are measured in the repo.

github.com/nicedreamzapp/claude-code-local, Python, 154 commits on main, 2026-03-25 to 2026-10-10

GitHub repoSoftware pageDocs site

The problem

Anthropic's Claude Code CLI is built to talk to Anthropic's cloud API. Matt wanted it to drive an open-weights model on a Mac with no API key and no network traffic. Two things stood in the way. Local models emit tool calls in many incompatible formats, often malformed, and Claude Code expects clean Anthropic tool_use blocks. Claude Code also sends a very large system prompt every turn (the commit message on 705755d puts it at about 24K tokens), so on local hardware prefill time and KV cache reuse decide whether the tool is usable at all.

Constraints

  • Apple Silicon only, through MLX and mlx-lm. setup.sh picks the model by unified memory, from Gemma 4 E4B under 16 GB to Qwen 3.8 27B 8-bit at 96 GB and above.
  • Memory headroom on 32 GB machines. A comment at proxy/server.py:57-78 explains the prefill chunk size choice in terms of the OOM killer.
  • Offline by default; the launchers turn off Claude Code's nonessential traffic.
  • No web framework: stdlib http.server only.

What he built

proxy/server.py (about 1,900 lines) speaks the Anthropic Messages API directly, including SSE streaming, tool_use, and /v1/messages/count_tokens, and runs the model in process through mlx-lm. The decision to serve Anthropic's format natively instead of sitting behind an OpenAI-style translation proxy is documented in docs/FULL-GUIDE.md.

  • parse_tool_calls() (server.py:469) accepts the formats models actually produce: Gemma 4's native <|tool_call> syntax, <tool_call> JSON, Qwen's <tools>, fenced JSON, bare JSON, and a recover_garbled_tool_json() path for XML-in-JSON hybrids. The commit message on 236bf72 gives the reasoning: retrying "can't help because it re-asks for tags the model has already declined to use."
  • generate_response() (server.py:1130) diffs the new token sequence against the cached prefix, trims the KV cache, and only prefills the delta.
  • Code mode (optimize_for_code(), server.py:1099) replaces Claude Code's system prompt with a compact one and strips tool descriptions to names and parameter types.
  • agent/agent.py ("Native Engine") keeps a fixed system prompt of about 550 tokens so the KV cache survives across turns.

Upstream credit: Claude Code is Anthropic's; MLX and mlx-lm are Apple's; the models are from Google, Alibaba Qwen, NousResearch and others. The DFlash 2 numbers below run through the third party mlx-dspark server, not this one. Several fixes came from outside contributors and are credited in the repo.

Hard bugs

  • Stale KV cache (705755d). An earlier change trimmed the cache using KVCache.step, which is a fixed 256-token allocation increment, not the token count. For any shared prefix over 256 tokens the trim amount went negative and never ran, so every session after the first appended onto old KV state and degraded to empty output. The fix uses .offset and falls back to a full prefill for caches that can't be trimmed.
  • Prefill memory (8099bb0). mlx-lm runs the LM head on every prefill chunk. With an 8192-token chunk that allocates a large logits tensor that is thrown away. Dropping the chunk to 512 cut peak memory on a 21.4K-token prompt from 34.2 GB to 20.9 GB (table at server.py:69-73).
  • Sliding window cache (3d5f749). Gemma's RotatingKVCache becomes untrimmable past its 1024-token window, so prefix reuse silently stopped. The Native Engine swaps in a plain KVCache; the commit reports byte-identical greedy output.

Results (from the repo)

  • Turn start in the Native Engine: 6.5 to 7.2 s down to 0.36 s time to first token at 4.5K context (docs/FULL-GUIDE.md).
  • Gemma 4 collapsed tool-call opener: 7/14 to 14/14 per run (236bf72).
  • Peak memory 34.2 GB to 20.9 GB (8099bb0).
  • On the Agent-12 benchmark, Qwen 3.8 27B 8-bit with DFlash 2: easy 12/12 in 122.0 s, hard 7/8 in 401.4 s (docs/BENCHMARKS.md). Single runs on an M5 Max with 128 GB.
  • 3,361 stars on 2026-10-15 (GitHub API). His profile README refreshes the count from the API every four hours.
  • Tests: six script files, the largest with 20 parser cases. No CI.

What it shows

He works where the model meets the runtime: KV cache semantics, prefill memory, and the gap between what a model should emit and what it does emit. Fixes are measured before and after.