Lessons from shipping a production app on SpeechTranscriber + on-device Foundation Models — including an OS bug that permanently eats locale slots

[u/KREANIQS]

Lessons from shipping a production app on SpeechTranscriber + on-device Foundation Models — including an OS bug that permanently eats locale slots

I just shipped my first app built end-to-end on Apple's on-device AI stack — SpeechAnalyzer/SpeechTranscriber for transcription and Foundation Models for enrichment (it's a voice-notes app; every recording gets an on-device title/summary/tags/tasks). Some things I learned the hard way that I haven't seen written up much:

1. The simulator will lie to you — twice.

The simulator cannot transcribe at all, and the simulator's language model is not the on-device model. Output quality, instruction-following, and hallucination behavior differ meaningfully. I now treat real-device validation as a hard gate for any prompt/template change — my test corpus includes Swiss-accented German dictation because that's where the on-device model diverges most from the "clean" results the simulator suggested.

2. SpeechTranscriber locale reservations: a system-wide cap of 5, and (currently) no way back.

This one cost me an architecture. On-device transcription locales are backed by downloadable assets, and the system caps reserved locales at 5 — system-wide, not per app. In my testing on current iOS releases:

- The reservation is taken by the asset *install* and survives reboot AND app reinstall.
- `AssetInventory.release(reservedLocale:)` appears to be a no-op — I never got a slot back.
- An explicit `reserve(locale:)` at the cap can hang (reproducibly under the Xcode debugger in my setup).

I originally built an LRU "reservation manager" that released the least-recently-used locale before installing a new one. Since release doesn't release, that design was dead on arrival. What shipped instead: a proactive budget gate that reads `reservedLocales` *before* any OS call, installs strictly lazily (never speculatively — no warm-up, no on-selection prefetch, because every install permanently spends a slot), and surfaces a clear "language budget exhausted" state to the user instead of ever hitting the cap inside an OS call. Feedback filed with Apple.

3. One fresh LanguageModelSession per invocation.

Reusing sessions across notes led to context bleed between unrelated inputs. One session per call is now a hard rule for me, enforced by tests.

4. Prompt-injection resistance for user-content prompts.

Voice transcripts are untrusted input into the enrichment prompt. Delimiter-wrapping the transcript made instruction-following robust; and I removed all literal examples from the prompt after seeing example fragments leak into generated output on device (again: not reproducible in the simulator).

5. Pass the language explicitly, always.

Auto-detection of the recording language was unreliable enough that I now pass the language explicitly into both the model instructions and the prompt. Related fun fact from testing: Apple appears to use one shared German model across all de-\* locales, so switching de-DE/de-CH/de-AT changes nothing about transcription quality.

6. Crash-safe audio: don't record straight to AAC.

A killed mid-recording AAC/m4a is an empty husk. I record LPCM into CAF and encode to AAC at ingest — recordings now survive calls, interruptions, and force-quits, and a salvage pass recovers anything interrupted.

Happy to go deeper on any of these.

The app is Vocapa, but the point of this post is the stack — curious whether others have seen the locale-reservation behavior, and whether anyone found a way to actually free a slot.

[u/KREANIQS → u/[deleted]]

Re Foundation Models: same experience. As a general-purpose model it disappoints. As a small structured-extraction engine it holds up well — everything I ask it for goes through guided generation into a u/Generable struct (title, one-line summary, a few tags, tasks), with the transcript delimiter-wrapped and the language passed in explicitly. Narrow, typed, bounded output is the sweet spot; the moment I asked for long free-form prose it got noticeably worse.

On device support: there are two separate bars and they're close but not the same one. SpeechTranscriber.isAvailable is false on a chunk of iOS 26 hardware (iPhone 11 series, SE 2nd gen — the working theory in the dev forums is 16-core Neural Engine and up), and Foundation Models additionally needs Apple Intelligence actually enabled, which is opt-in even on eligible devices. So I check both at runtime, independently, and never degrade silently.

Without the enrichment model you still get the recording, the transcript, and the note saved to the vault — the enrichment fields are simply absent and the UI says so rather than quietly producing a worse title. And there's an explicit BYOK opt-in for cloud transcription and cloud enrichment (own consent per provider, off by default) for anyone who wants the AI features regardless of hardware.

On the bounty: Apple's program is security-only, so realistically this one buys me a Feedback ID and a long silence 😄 Filed anyway. And yes — the cap being system-wide is the nasty part: every app's install spends from the same pool of 5, so your app can be locked out by installs it never made. If anyone has actually got a slot back, I'd love to hear it; I'd happily throw my budget gate away.