Free Shipping on all orders · Priority Mail Shipping with fee of $8.00
Software / Case studies / NarrateClaude
Case study 5 of 9

NarrateClaude

A voice loop for Claude Code on macOS using Apple's speech recognizer, with stability-based endpointing and watchdogs for the recognizer's specific failure modes. A local Llama 3.2 3B on MLX decides whether each sentence was meant for the assistant.

github.com/nicedreamzapp/NarrateClaude, Swift, bash, Python, MIT, 18 commits, 2026-04-11 to 2026-10-09

GitHub repoSoftware pageDocs site

The problem

Talk to Claude Code out loud at a desk with open speakers, and hear it answer, with speech recognition on the Mac. That sounds like wiring a recognizer to a terminal. In practice the hard parts are deciding when a sentence is finished in a noisy room, keeping Apple's speech service alive for hours, ignoring speech that wasn't meant for the assistant, and typing into the right window without stealing focus.

Constraints

  • macOS on Apple Silicon, Apple's SFSpeechRecognizer (on-device recognition when the recognizer supports it), and Terminal.app through AppleScript.
  • Open speakers, no headphones. The project log states this as a hard rule (BARGE_IN_ATTEMPTS.md).
  • Apple's speech daemon is unreliable over long sessions: synchronous XPC hangs, silent stalls, slow degradation.
  • Fully local only if Claude Code points at a local model, which the README says plainly; the sibling repo claude-code-local provides that.

What he built

  • dictation/src/Listen.swift: a continuous listener with all state on one serial queue. Endpointing is based on transcript stability, not audio level: when the partial transcript stops changing for 2.5 s, the request ends. The header comment gives the reason, that it holds up against background noise where volume-based silence detection doesn't (fans and HVAC, per the README). startWatchdog() has three exits: a buffer backlog over 200 (catching the XPC hang), a "silent wedge" where audio is flowing but no recognition events arrive for 20 s, and a preventive recycle every 300 s.
  • dictation/bin/dispatch: a bash supervisor running listen | filter.py | relevance.py | inject in a respawn loop with backoff.
  • dictation/filter/relevance.py: a "was that for me?" filter. A mute command, a few tight regexes, then an LLM judge, Llama-3.2-3B-Instruct 4-bit served locally by mlx_lm, with six lines of rolling context and the assistant's last spoken reply. It fails open, and each special case has a dated comment about a real line it wrongly dropped.
  • dictation/bin/inject: types into one bound Terminal window id with AppleScript do script, which writes to the tty without the clipboard or focus. It drops noise markers and very short fragments.
  • dictation/bin/speak-on-stop: a Claude Code Stop hook that reads the transcript, strips code, URLs and markdown, and hands the text to a user-supplied speak command.

Upstream credit: Apple Speech and AVFoundation, Anthropic's Claude Code, MLX and mlx-lm, Meta's Llama 3.2, Resemblyzer for the voiceprint filter, and third party TTS models for the voice.

Hard bugs

  • Typing into a closed window (f6c448b). The watchdog sent SIGTERM to the supervisor, but bash defers signals while blocked on a pipeline read, so the listener kept transcribing into a window that no longer existed. Shutdown now goes through a flag file the loop checks.
  • Silent recognizer stalls (8cdb598). Apple's speech daemon could stay alive with audio flowing but stop emitting events, which slipped past the backlog check until the old 600 s recycle. The fix compares the time of the last recognition event with the time of the last non-silent audio and restarts after 20 s; the recycle dropped to 300 s.
  • Rosetta crash loop (2008e0e). A bash running under Rosetta spawned x86_64 Python, the numpy wheels aborted, and the listener crash-looped. The filter is now launched with arch -arm64.

Results (from the repo)

  • The v1 code in the repo has no tests and no measured benchmarks; the numbers in it are configuration (2.5 s stability window, 20 s wedge detector, 300 s recycle).
  • BARGE_IN_ATTEMPTS.md logs a separate v2 full-duplex echo cancellation effort; that code is not in the repo.
  • The same log records what failed, including a personal voice activity model that was disabled because it misfired.

What it shows

He builds around unreliable system components instead of hoping they behave: stability-based endpointing, watchdogs keyed to the actual failure signatures, and a small local model used as a filter with fail-open rules. He also keeps an honest log of approaches that failed.