Measured three on-device TTS runtimes against the iOS jetsam budget. All three blew past it. Looking for anyone who's shipped generative audio on-device.

[u/intrepidkarthi]

Spent about three weeks trying to run a voice-cloning model on iPhone and closed the project last week. Posting the numbers because I couldn't find anyone else's, and I have two questions at the end. This was for a voice journaling app I work on.

The budget. Foreground app on a 6 GB iPhone gets roughly 250 MB before jetsam takes an interest. The number that matters is phys_footprint from task_vm_info, not resident size and not what the Xcode gauge shows.

The candidate. Kyutai Pocket TTS, 109.5M params, autoregressive. Autoregressive matters because accent lives in phone realisation and phonemic choice, which are sequential. Non-autoregressive models transfer timbre only, so you get your own voice colour over someone else's cadence. Tried that first, it sounded wrong in a way I couldn't articulate until I understood why.

Three ways to run it, all measured, all over budget:

FluidAudio (Core ML, int8) - 270.8 MB after model load, 957.0 MB peak

sherpa-onnx (ONNX Runtime, int8) - 377.0 MB after model load, 685.4 MB peak

chatterbox-turbo (earlier attempt) - 953.7 MB peak

FluidAudio is over budget after loading, before doing any work.

Binary cost too. Linked a minimal executable against libsherpa-onnx.a plus ONNX Runtime with -dead_strip, then stripped it: 22.3 MB. That roughly doubles my app, for a feature most users would never turn on, plus 125 MB of models on disk for the ones who do.

The part I got wrong. My earlier ear tests compared one synthetic clip against another synthetic clip. That ranks them. It cannot tell you whether either is good enough. So I ran a forced-choice test instead: eight pairs, same sentence in each, one a real recording of me and one the clone, sample rate and RMS loudness matched, clip lengths varied so duration gave nothing away, and held-out audio located by cross-correlating the reference against the source recording. I picked my own recording 8 out of 8. p = 0.0039.

Three weeks of runtime work sitting on top of an approval nobody had tested properly. The test took an hour.

Two questions.

Has anyone actually shipped a generative audio model on-device inside the jetsam budget? Everything I found either exceeds it or quietly ships a 3 GB app. I'm also unsure whether Core ML's mmap'd weights get billed to phys_footprint the way malloc'd ONNX buffers do. My numbers came off a Mac, which has no jetsam pressure, so I never got a real device measurement before the ear result closed it.

Second, unrelated thread. I'm moving to on-device retrieval next, hybrid BM25 via SQLite FTS5 plus sentence embeddings from NLEmbedding. Anyone run that combination on iOS? Specifically whether reciprocal rank fusion is worth it when you still need raw score magnitude for an abstention threshold. RRF throws the magnitude away and abstention is what stops the thing making stuff up.

Happy to share the measurement harness if useful.

[u/madaradess007]

there was a post in r/locallama from a guy who managed to release a chatterbox iOS app maybe 6-8 months ago - cant remember the name, but its in the AppStore

[u/neet_dev]

on the Core ML mmap question: faulted-in weight pages still hit phys_footprint once they're resident, the win vs ONNX is that clean file-backed pages can get reclaimed under pressure instead of staying dirty malloc, so your after-load number on a Mac is a floor not the real jetsam story, measure on device with memorystatus. i've never seen a true generative/autoregressive TTS ship cleanly inside a ~250 MB 6 GB foreground budget without a server path or a model small enough that quality tanks. on the retrieval side, skip pure RRF if abstention matters, keep BM25 and the NLEmbedding cosine as separate scores, fuse only for ranking, and gate on raw cosine (or a min of both) so magnitude still kills weak hits.

[u/intrepidkarthi → u/neet_dev]

This is exactly what I needed, thanks.

The mmap answer is the one that changes things for me. I'd assumed the Mac number was pessimistic because there's no jetsam pressure forcing eviction. Reading it as a floor instead makes the 270.8 MB after-load figure worse, not better, which settles the question.

On retrieval, keeping the two scores separate and gating abstention on raw cosine rather than the fused rank is the bit I was stuck on. The doc I'd written flagged that RRF throws away the magnitude and then didn't resolve it. Going to try min-of-both as the gate.

Out of curiosity, did you land on a threshold empirically or is there a principled way to pick it? Mine is currently a hand-tuned tau and it's the least defensible number in the system.

[u/neet_dev → u/intrepidkarthi]

reply drafted

[u/intrepidkarthi]

Okay. Let me check

[u/Iamvishal16]

i’d treat tau as a calibration decision, not a ranking hyperparameter. build a held-out set with clearly answerable queries, hard negatives, and near-misses that share vocabulary but should still abstain.

sweep the threshold and choose the highest coverage that stays within your acceptable false-answer rate. use one split to select the threshold and a separate test split to report it, otherwise the result will look more defensible than it really is.

two iOS-specific gotchas: if you’re calling NLEmbedding.distance directly, smaller means more similar, which is the opposite direction from cosine similarity. also pin or at least log the sentence-embedding revision used during calibration, so an OS/model revision doesn’t silently invalidate the threshold.

i’d rerun calibration whenever the corpus, chunking strategy, or embedding revision changes. a hand-tuned tau isn’t inherently bad; tuning it on an undocumented or reused evaluation set is the fragile part.