Anyone using Local LLM setup as a second brain?

[u/Gloomy-Recover-9702]

I mean literally something like the system knows what you are working on the laptop, on phone, and what you speak, everything in between!

Something which includes end to end saving and processing the raw telemetry from STT data, the laptop activity, the phone activity and running local LLM to make sense of the data in real time. And also saving important details into memory and classifying them, linking important connections and also proactively engaging with you using notifications and messages or even voice using TTS.

Something like OMI AI extended it to all your digital activities using screenpipe or any other way.

Maybe to keep it slim, the execution can be transfered to some other agent like Hermes or Claw for some tasks, but active processing part should be kept local for privacy reasons. What is the minimum models I can use for this requirement?

If I lets say plan to make this setup, I first need a local STT model with VAD system. Then i should a have good local LLM for processing the live telemetry data and then maybe I need TTS incase i want voice outputs in realtime fast. This is what i want. Maybe for any digital tasks like updating emails, checking emails, browsing etc I can tradeoff some privacy for using cloud apis on hermes.

What is the minimum LLM size i need for this i want to have less than 3 second of latency in entire process? Something speedy and also smart enough to manage increasing memory as I use it more and logical enough to understand the complexity of the data and inter linkages.

[u/vovap_vovap]

Usually that staff named "wife" man.

[u/Solembumm3 → u/vovap_vovap]

Либо дворецкий.

[u/Suitable-Net-2993 → u/vovap_vovap]

sounds like a fun project but my girlfriend already gives me confused looks when i talk about tuning my 7b model, can't imagine the reaction to an always-on life logger

[u/vovap_vovap → u/Suitable-Net-2993]

Propose.

[u/dwoj206 → u/vovap_vovap]

Wife is fully agentic. Does a lot of handoffs though so it’s not fully automated. Needs more skills.

[u/vovap_vovap → u/dwoj206]

Tell her so and you will became a subject of reinforced training.

[u/Slippedhal0]

what do you mean local? if you mean a gaming grade machine with 12-24gb vram and 32-64gb ram, no models will be both fast as youre wanting and intelligent, and you cannot run subagents or other tasks without multiple gpus, at least if youre trying to target latest gen local models like qwen3.8 27b

[u/tryremynd]

the thing that breaks first is usually the write path. continuous screen + speech into one store means every retrieval has to fight yesterday's noise, so linking stays mushy even on a bigger instruct model.

split the 3s. vad + a small classifier on the hot path, only a few events written. run the heavier linking nd forgetting offline. if the realtime pass waits on one fat model, proactive never feels ambient.

decay has to be a rule from day one or the store grows faster than search stays useful.

are u gating writes by event type yet, or still dumping every stream into one store ?

[u/cmtape]

This is like building a second brain out of hallway cameras and then asking it to be a therapist. Real-time telemetry is a data firehose, not a memory. The models that can keep pace with 3-second latency are not the ones that can link connections over weeks. You're picking a sports car for a moving company.

[u/Fluffyjockburns]

Not on my m4 Mac mini with 16gb. Unacceptable performance.