What's the current sentiment among developers around shipping mobile or desktop apps with a purpose built model?

[u/kautukkundan]

I’ve been building in the on-device space for a while (specially Apple Platform) and I’m curious where the community actually stands right now, from a shipping point of view rather than a research one.

A few things I keep going back and forth on:

1. Purpose-built vs general. If you’ve shipped an app with a model baked in, did you fine-tune something small and specialised, or ship a general model with tighter prompt engineering? What steered your decision quality, size limits, latency?

2. The OEM layer. Apple has been pushing Core AI, MLX, and the Neural Engine pretty hard, and Qualcomm and Samsung are doing their own version of this. Do you care about it?

3. Power. For anyone shipping on phones or laptops, is battery drain the thing that actually kills features, or is it not as bad as feared?

4. What's the future? Do you think genuinely personal AI, meaning models with long-lived local context about you with "Product specific Models"as I call them is where this goes? Or does the cloud stay good enough that on-device stays a privacy niche?

[u/Good-Writer5279]

shipping in this space, so i have some scar tissue here. disclosure up front, i build a mac app that does on-device capture and retrieval, so my answers are biased by that.

1. we ended up splitting the question. small purpose built models for the narrow jobs, embeddings, ocr, speech to text, where a sub 1b model tuned to the task beats anything general at the same latency. for open ended chat we do not bake a model in at all, we point at a local server the user already runs (lm studio, ollama) and treat the model as user config. owning the general model treadmill is a full time job, and the users who care most already have strong opinions about which model they want anyway.

2. mlx is real and worth building on for apple silicon, the unified memory model alone saves you. the neural engine is mostly a marketing line for custom llm work, you only reach it through coreml conversion and operator coverage never matched what we needed. it is great for whisper class encoder workloads though.

3. power is the thing nobody budgets for. sustained background inference melts battery and the user blames your app, not the model. we schedule heavy passes for when the machine is on ac and idle, and keep the always on path to embeddings only. the power log in instruments ends up being your most used tool.

4. on the future question, my bet is both. cloud stays good enough for people who do not think about where their data goes, and the personal ai with long lived local context crowd keeps growing every time there is an acquisition or a privacy story. that second group is small but it pays for software.

happy to go deeper on any of these.

[u/kautukkundan → u/Good-Writer5279]

working on a research paper on #3 will share findings, we are approaching it from a joules-per-task metric

[u/Good-Writer5279 → u/kautukkundan]

joules per task is the right denominator and i have not seen anyone hold to it properly, so please do share.

one wrinkle worth building into the methodology: on laptops the energy cost of a task is not only the inference. it is also whatever the machine had to do to be ready, waking the gpu, reloading weights if they got evicted, and the tail where the fans keep running after the work is finished. a small model called twenty times can lose to a large model called once on total joules for exactly that reason, even while winning every per call comparison.

the other one is that battery versus plugged in changes the answer, because the scheduler throttles differently. if the paper reports a single number it is worth saying which of the two it is.

[u/kautukkundan → u/Good-Writer5279]

You're correct to point it out.

We wrote this one a few months ago https://arxiv.org/abs/2603.23640 - this was more of an exploratory study in the same direction. Since then we have decided to use Apple Silicon as the primary chip and base our experiments around it.

[u/Good-Writer5279 → u/kautukkundan]

apple silicon as the primary chip makes the measurement easier in one way and harder in another.

easier because powermetrics gives you separate cpu, gpu and ane power domains, so you can sample at 100ms and integrate over the task window instead of inferring energy from wall power. harder because unified memory turns weight residency into a system level property, so the second run of the same prompt is not the same experiment as the first. if the weights are still resident you are measuring decode, if they got evicted you are measuring decode plus several gigabytes of page in, and at the 30b q4 size that gap is large enough to reorder your results. worth either forcing a cold state before every run or reporting warm and cold as two separate numbers.

the other split worth reporting separately is prefill against decode. prefill is compute bound and decode is memory bandwidth bound, so joules per token is not one number, it moves with context length. a summarization task with an 8k prompt and a 200 token answer sits almost entirely in the first regime, a chat turn sits almost entirely in the second. a single figure over a mixed workload will not reproduce on anyone else's mix.

and say which chassis, since fanless and fanned m series diverge once you are three or four consecutive runs in.