Song Forge

The problem

Turn a one-line description such as "sunset reggae with steel pan, 78 bpm" into a finished song with lyrics, music and vocals, using models running on a Mac rather than a cloud API. It started as a personal tool and became the render backend for a consumer app listed on the App Store and Google Play (README.md). The native iOS client is not in this repo.

Constraints

• An Apple Silicon Mac as the render host. Music generation runs through ACE-Step (MPS and MLX), stem separation through demucs on MPS, lyric timing through mlx_whisper.
• GPU memory contention. Two voice conversion model loads at once on MPS hang (forge_server.py:1074-1077).
• Long jobs: 90 s timeout for lyrics, 600 s for voice conversion and stem separation.
• No web framework. forge_server.py is a stdlib ThreadingHTTPServer; its docstring says "Deliberately small. Everything else lives in ACE-Step."
• Android release rules. Google Play's R8 requirement forced code shrinking on, which can break JavaScript bridges at runtime.

What he built

ios/forge_server.py (about 2,050 lines) orchestrates several upstream models:

• Lyrics. _llm_lyrics calls a local Gemma model through LM Studio with an anti-cliché system prompt and a persistent banned-phrase list. The list is injected into the prompt and also enforced afterward: any output that still contains a banned phrase is rejected and falls back to a template.
• Generation. _seed_prompt treats the user's style text as authoritative and only adds defaults when a hint is missing (comments cite user feedback by date). Jobs are queued on ACE-Step's REST API and polled by a _worker() thread.
• Voice swap. _run_swap_impl separates vocals with demucs, converts them with seed-vc while streaming progress lines to the UI, and remixes with ffmpeg loudness normalization. A global lock serializes swaps.
• Karaoke timing. _align_lyrics uses Whisper word timestamps only for timing, never for display, because it mishears sung lyrics. Each written line is matched monotonically against a window of one to three segments with a recall-weighted fuzzy score, since "symmetric Jaccard underrates real matches." Unsung lines are skipped and word times interpolated.
• Serving. _file_ranged implements HTTP Range so audio seeking works; health checks are cached so the status endpoint never blocks.

Upstream credit: ACE-Step 1.5, seed-vc, demucs, OpenAI Whisper through mlx_whisper, Google's Gemma through LM Studio, MLX, ffmpeg, Three.js.

Hard bugs

• Hung voice swaps (62b153f). Two parallel seed-vc loads on MPS hung until the 10 minute timeout. Fixed with a global swap lock and a "queued" status.
• Seeking restarted audio (3dab85b). The audio route ignored Range, so every seek replayed the file. Added 206 responses, suffix ranges, 416 on bad ranges, and broken-pipe tolerant chunked writes.
• Deleted songs coming back (c4b7513). A job deleted mid-render was saved again by the worker and restored on restart; the worker now skips deleted jobs. The same commit made remote deletes require two consecutive misses so a transient manifest rewrite can't wipe songs, and added a heartbeat so the progress bar didn't freeze for 30 to 60 s during model load.
• Runtime breakage from R8 (a400649). Shrinking renames @JavascriptInterface methods only called from JavaScript; keep rules were added before it shipped.

Results (from the repo)

• Commit 29f3843 records the Android app live on Google Play (approved 2026-08-04), and 070fdec records the iOS app page verified on the US, UK and CA App Store storefronts.
• No tests and no checked-in benchmarks.

What it shows

He can chain several heavy upstream models into one dependable pipeline on a single machine, with the unglamorous parts done right: serialization where the GPU can't share, guards on model output, timing alignment that tolerates model mistakes, and enough release engineering to get through two app stores.