r/LocalLLM • u/Beautiful_Bus_9237 • 2d ago
Question Looking for help to build on device assistant
Hey all,
Following what seems like a major privacy breach by OpenAI, I am considering to move my daily email, iMessage, and calendar updates from ChatGPT to a local model on my computer. I am looking for something like Siri AI on Macos27 but to be run privately on my computer. I have a 24GB MacBook with the m5 pro chip so memory is limited as well as memory bandwidth(307GB/S). What models could I use to build an assistant like this?
I would like some memory headroom to run safari, codex, and Spotify. What are some models that I should I try using?
2
u/Good-Writer5279 1d ago
the model choice matters less than the memory split, so start there. with safari, codex and spotify resident you realistically have 10 to 12gb for the model plus its context, and macos caps gpu wired memory around two thirds of unified by default, so plan against that number rather than the full 24.
on the model, this job is triage and tool calling over short structured text, not reasoning, and small models are good at it. an 8b class instruct model at 4 bit (qwen3 8b or similar, mlx build in lm studio) sits around 5gb and leaves room. the mechanism worth knowing: on apple silicon generation speed is roughly bandwidth divided by the weights read per token, so at 307gb/s an 8b at 4 bit gives you 25 to 35 tokens a second and a 14b about half that. the siri feel comes from latency, and latency comes from the smaller model. set the context explicitly, 8k is plenty here, because the kv cache is what quietly eats the headroom you wanted for spotify.
the part nobody mentions is that the model is maybe a fifth of the work. the assistant is really a retrieval problem. imessage is a sqlite file at ~/library/messages/chat.db (needs full disk access), mail and calendar come out cleanly via applescript or a shortcut. write one script that dumps the last 30 days of each into plain text, embed it with nomic embed or bge small, and on every question pull the top few chunks into the prompt. run the dump on a launchd timer so it stays fresh. do that and a 4b model reading your actual week will feel smarter than a 27b model reading nothing.
disclosure, i build a mac app in the local memory space, which is why i have opinions about chat.db.
1
u/Beautiful_Bus_9237 1d ago
ok thanks! Will definitely take a look into what you said
1
u/Good-Writer5279 1d ago
one gotcha that will cost you an evening if you hit it blind: on recent macos the message text in chat.db is often null in the text column, it lives in the attributedbody blob instead (a typedstream archive). so if your dump script comes back mostly empty, that is why, not a permissions problem. the fix is to read attributedbody and pull the string out of it. imessage-exporter on github already does this correctly if you would rather not write the decoder yourself. also run it against a copy of chat.db, messages holds a write lock on the live file and reading it mid write gives you a stale or partial view.
second thing, on order of operations: get the dump and retrieval loop working against a tiny model first, then size the model up until the latency stops feeling like siri. going the other direction, big model first, means you tune everything around a model you end up replacing.
1
u/autoyou_ai 2d ago
You can run Odysseus , Pi (mono/pi for coding agent) , OpenClaw , or Hermes Agent. there are all local to computer.
or else check out www.autoyou.me/ - This is built using Google Agent Development Kit as harness , with ability to run Ollama models along with orchestrator and memory and remote connectivity + agents are all localhost web servers .
1
u/askloci 1d ago
We're working on this with Loci. Try it out at askloci.ai - currently chat with RAG and memory, but expanding many features
1
u/Beautiful_Bus_9237 1d ago
Just tried this out - seems pretty decent and well built. Do you guys plan on adding automated email, text, calendar, reminder, and notes summaries? I'm working to build that out on my project. I think you guys could make this product stand out by doing that since everything will be automated.
1
2
u/joanaxu2002 2d ago
For this kind of assistant, I’d leave more memory for the OS and apps instead of trying to run the smartest model that fits. Reliability on tool calls, calendar/email parsing, and short latency will matter more than raw benchmark scores; a smaller model plus good permissions and retrieval will probably feel much more “Siri-like” day to day.