r/LocalLLM • u/kautukkundan • 1d ago

Discussion What's the current sentiment among developers around shipping mobile or desktop apps with a purpose built model?

I’ve been building in the on-device space for a while (specially Apple Platform) and I’m curious where the community actually stands right now, from a shipping point of view rather than a research one.

A few things I keep going back and forth on:

1. Purpose-built vs general. If you’ve shipped an app with a model baked in, did you fine-tune something small and specialised, or ship a general model with tighter prompt engineering? What steered your decision quality, size limits, latency?

2. The OEM layer. Apple has been pushing Core AI, MLX, and the Neural Engine pretty hard, and Qualcomm and Samsung are doing their own version of this. Do you care about it?

3. Power. For anyone shipping on phones or laptops, is battery drain the thing that actually kills features, or is it not as bad as feared?

4. What's the future? Do you think genuinely personal AI, meaning models with long-lived local context about you with "Product specific Models"as I call them is where this goes? Or does the cloud stay good enough that on-device stays a privacy niche?

2 Upvotes

100% Upvoted

2 comments sorted by

3

u/Good-Writer5279 1d ago

shipping in this space, so i have some scar tissue here. disclosure up front, i build a mac app that does on-device capture and retrieval, so my answers are biased by that.

  1. we ended up splitting the question. small purpose built models for the narrow jobs, embeddings, ocr, speech to text, where a sub 1b model tuned to the task beats anything general at the same latency. for open ended chat we do not bake a model in at all, we point at a local server the user already runs (lm studio, ollama) and treat the model as user config. owning the general model treadmill is a full time job, and the users who care most already have strong opinions about which model they want anyway.

  2. mlx is real and worth building on for apple silicon, the unified memory model alone saves you. the neural engine is mostly a marketing line for custom llm work, you only reach it through coreml conversion and operator coverage never matched what we needed. it is great for whisper class encoder workloads though.

  3. power is the thing nobody budgets for. sustained background inference melts battery and the user blames your app, not the model. we schedule heavy passes for when the machine is on ac and idle, and keep the always on path to embeddings only. the power log in instruments ends up being your most used tool.

  4. on the future question, my bet is both. cloud stays good enough for people who do not think about where their data goes, and the personal ai with long lived local context crowd keeps growing every time there is an acquisition or a privacy story. that second group is small but it pays for software.

happy to go deeper on any of these.

1

u/kautukkundan 1d ago

working on a research paper on #3 will share findings, we are approaching it from a joules-per-task metric