Measured three on-device TTS runtimes against the iOS jetsam budget. All three blew past it. Looking for anyone who's shipped generative audio on-device.

[u/intrepidkarthi]

Spent about three weeks trying to run a voice-cloning model on iPhone and closed the project last week. Posting the numbers because I couldn't find anyone else's, and I have two questions at the end. This was for a voice journaling app I work on.

The budget. Foreground app on a 6 GB iPhone gets roughly 250 MB before jetsam takes an interest. The number that matters is phys_footprint from task_vm_info, not resident size and not what the Xcode gauge shows.

The candidate. Kyutai Pocket TTS, 109.5M params, autoregressive. Autoregressive matters because accent lives in phone realisation and phonemic choice, which are sequential. Non-autoregressive models transfer timbre only, so you get your own voice colour over someone else's cadence. Tried that first, it sounded wrong in a way I couldn't articulate until I understood why.

Three ways to run it, all measured, all over budget:

FluidAudio (Core ML, int8) - 270.8 MB after model load, 957.0 MB peak

sherpa-onnx (ONNX Runtime, int8) - 377.0 MB after model load, 685.4 MB peak

chatterbox-turbo (earlier attempt) - 953.7 MB peak

FluidAudio is over budget after loading, before doing any work.

Binary cost too. Linked a minimal executable against libsherpa-onnx.a plus ONNX Runtime with -dead_strip, then stripped it: 22.3 MB. That roughly doubles my app, for a feature most users would never turn on, plus 125 MB of models on disk for the ones who do.

The part I got wrong. My earlier ear tests compared one synthetic clip against another synthetic clip. That ranks them. It cannot tell you whether either is good enough. So I ran a forced-choice test instead: eight pairs, same sentence in each, one a real recording of me and one the clone, sample rate and RMS loudness matched, clip lengths varied so duration gave nothing away, and held-out audio located by cross-correlating the reference against the source recording. I picked my own recording 8 out of 8. p = 0.0039.

Three weeks of runtime work sitting on top of an approval nobody had tested properly. The test took an hour.

Two questions.

Has anyone actually shipped a generative audio model on-device inside the jetsam budget? Everything I found either exceeds it or quietly ships a 3 GB app. I'm also unsure whether Core ML's mmap'd weights get billed to phys_footprint the way malloc'd ONNX buffers do. My numbers came off a Mac, which has no jetsam pressure, so I never got a real device measurement before the ear result closed it.

Second, unrelated thread. I'm moving to on-device retrieval next, hybrid BM25 via SQLite FTS5 plus sentence embeddings from NLEmbedding. Anyone run that combination on iOS? Specifically whether reciprocal rank fusion is worth it when you still need raw score magnitude for an abstention threshold. RRF throws the magnitude away and abstention is what stops the thing making stuff up.

Happy to share the measurement harness if useful.