Free Shipping on all orders · Priority Mail Shipping with fee of $8.00
Software / Case studies / RealTime AI Cam
Case study 1 of 9

RealTime AI Cam

A free iPhone and Android camera app for blind and low-vision users that detects 601 object classes with YOLOv8 on Core ML and describes scenes with a fine-tuned Qwen3.5-0.8B through MLX Swift, all on the phone. The work is in the parts that make it hold up: thermal tiers, a Neural Engine workaround, jetsam fixes, stride-correct tensor decoding, and an OCR cross-check on the model's dollar amounts.

github.com/nicedreamzapp/RealTimeAICam, Swift (iOS) and Kotlin (Android), AGPL-3.0, 147 commits, 2025-07-26 to 2026-10-14

GitHub repoSoftware page

The problem

A camera app for blind and low-vision people that says out loud what is in front of the phone: objects, printed text, mail and bills, and a short description of a scene. Everything has to run on the phone. No account, no server, nothing uploaded.

Constraints

  • On-device only, iPhone and Android, including budget Android phones.
  • Memory. A comment at ios/CameraPreview.swift:375 notes the camera daemon holding about 1.5 GB. On 6 GB iPhones, the vision language model plus the camera got the app killed by jetsam.
  • Heat and battery. ThermalManager in ios/CameraViewModel.swift reacts to ProcessInfo.thermalState, and frame rate and input size come from a device tier (10, 15 or 20 fps; 352, 512 or 640 px input).
  • A Neural Engine bug on iOS 17.3.x. YOLOv8Processor.init uses .cpuAndNeuralEngine but falls back to .cpuAndGPU on that release (ios/YOLOv8Processor.swift:104-112).
  • Accessibility first: VoiceOver and TalkBack handoff, and numbers formatted for speech.

What he built

The models are upstream; the pipeline around them is his.

  • Detection. Ultralytics YOLOv8n trained on Open Images V7 (601 classes), converted to Core ML on iOS and TFLite on Android. YOLOv8Processor.swift does its own decode with a dynamic threshold, context boosts, NMS and conflict resolution. MetalImageResizer.swift letterboxes to 640x640 on Metal with a reused pixel buffer pool. ObjectTracker.swift smooths boxes across frames.
  • "What's this?" narrator. A LoRA fine-tune of Qwen3.5-0.8B, 4-bit, running through MLX Swift (MLXVLM) in OnDeviceVisionNarrator.swift, with a frozen prompt and the comment "Do not edit without retuning." On Android the same model runs through llama.cpp over a JNI bridge (cpp/narrator_jni.cpp) that probes each GPU and leaves a crash marker so the next launch falls back to CPU.
  • Guards around a small model. FrameQualityGate.swift scores a 256 px grayscale thumbnail (Laplacian variance, brightness, clipping) before inference, designed so it "can only ever skip a scan, never block one by mistake." MoneyCrossCheck.swift compares dollar amounts from the model against Apple Vision OCR and hedges when they disagree, because "A small vision model can read '$128.40' as '$126.40'."
  • Help Me Aim. Uses prompt-free YOLOE (4,585 classes) to talk a user into framing a shot. The steering logic in HelpMeAim.swift is pure box arithmetic so it can be unit tested without a camera.
  • Model delivery. VisionModelStore.swift verifies size and sha256, then swaps the model in atomically.

Upstream credit: Ultralytics YOLOv8 and YOLOE, Qwen3.5-0.8B, MLX Swift and mlx-swift-lm, llama.cpp, Apple Core ML, Vision, AVFoundation and ARKit, Google TFLite and ML Kit.

Hard bugs

  • Frames skipped forever (521e1ff). predictWithThermalLimits gated on a counter that only advanced inside predict(). After one skipped frame, every later frame was skipped.
  • Jetsam during inference (54e90d3, d40ea46). The camera session now pauses while the model runs, and the MLX Metal cache is capped at 20 MB.
  • Upstream crash (e432ad8). mlx-swift-lm kept M-RoPE position ids between calls, crashing the first request. Fixed by moving to mlx-swift-lm 3.31.4.
  • YOLOE boxes 2 to 4x too tall (0355486). The Core ML output's strides were not a plain 4x8400 layout, so y, w and h came from a neighboring anchor. Reads now go through the real strides and match a Mac reference.

Results (from the repo)

  • Android narrator on a Helio P35: 314 s down to 87 s after showing the model a 512 px copy of the photo (commit 36c7510; android/NARRATOR.md).
  • YOLOv8 on an MT6765: about 1630 ms CPU fp16, 870 ms GPU fp16, 690 ms GPU fp32 (YoloDetector.kt:24-26).
  • Tests: 148 Swift Testing @Test cases, 36 XCUITests, 100 Android @Test. No CI config.
  • Shipped on the App Store and Google Play.
  • The narrator that ships is Qwen3.5-0.8B: the bundled model in the iOS 1.3.0 build 30 archive has a 1024-wide, 24-layer text config and a model card naming Qwen/Qwen3.5-0.8B as base, and the Android 1.17 GGUF names Qwen3.5 0.8B.

What it shows

He takes models from research checkpoints to shipped phone apps on two platforms, and handles the parts that decide whether that works: Neural Engine quirks, memory limits, thermal tiers, tensor layouts, and guarding a small model's mistakes with cheaper classical checks.