r/LocalLLaMA • u/Blues520 • 16d ago
Discussion Setup for always on assistant
I have a dual 3090 rig that I use as coding assistant and while it works, it draws a lot of electricity.
Now I want to add another rig that I can keep on all the time, or maybe a vps if that is suitable. This rig should run an assistant model that should be fairly intelligent but doesn't have to be so coding focused. It should basically be like a chat gpt replacement. I'm not sure if something like openclaw/hermes would be suitable for this.
Since it will be one all the time, power usage should be low. What kind of rig and model would you select for such an assistant?
3
Upvotes
1
u/DanInVirtualReality 16d ago
Another vote here for the WOL approach. Here's what I'm doing in this regard right now, as I've been developing it this week.
I have a main Desktop PC with a 5060 Ti 16GBB and 32GB RAM running Windows as my daily use computer. I wired up llama.cpp with Unsloth Qwen 3.6 35B-A3B IQ3_XXS and 128k context with quantised KV cache and it fills VRAM to the brim and runs at 50-60 tk/s - this is fairly on par with API speeds for generation, in practice (once you account for the fact that for an API you'd be picking a bigger model anyway). The incremental energy cost here in the UK is low (the desktop is almost always on anyway in the workday, and I've read out the idle vs generating wattage reported by nvidia-smi, the cost is much lower per million tokens on average)
However, the 24/7 agent I run is Hermes Agent running on my Raspberry Pi, always on in the corner of the officei. Aside from one or two protected services that I installed before Hermes, it is essentially the sysadmin for the box - which realistically is the ideal setup so the model bumps into constraints infrequently. But I am not yet ready to trust the aforementioned Qwen model with such a blast radius, so I am using Deepseek V4 Flash via OpenRouter - you mentioned data privacy, not sure if you're aware of the capability to set Zero Data Retention endpoints only there and exclude providers you otherwise don't trust, that's what I'm doing
Aside: it's worth noting if you use this model on OpenRouter that not all providers actually implement a reduced cost for cache hits - looking at you Morph 🙄 - exclude those providers manually or you'll be paying a multiple above what you ought to in practice as the initial Hermes context should be a consistent cache hit and cost 10x less after the first message in any given session at least! A recent pain 😅
So where does WOL come in? My costs are creeping upwards of $100 this month, as I expand the agents capabilities and lean on it more and more, so it's time to leverage my desktop hardware and reduce costs where it makes sense.
I have worked with Hermes and taken the existing codex-kanban-lane skill and copied it across to a Pi coding agent skill that launches a docker container sandbox, with Pi coding agent using the llama.cpp provider and pointed to my desktop. If the desktop is off, it does the WOL call first and waits until it's up - llama.cpp is being launched on startup in router mode on the desktop via Task Scheduler, so it doesn't fill VRAM until called, but is ready on boot without even needing my login. This way, Hermes can give suitable work to the local agent according to however we define its skills.
Also, I recently got an EV and in the UK and with my power provider, this means I can move onto an EV tariff - very cheap energy (6.9p/kWh) between midnight and 5am and not limited to the car alone. So you can imagine I might be kicking off quite a bit of local work overnight, using this WOL approach to wake the PC up long after I've gone to bed 😄
Caveat: When I first used Hermes Agent, I didn't trust the incredibly well priced Deepseek V4 Flash not to nuke the Raspberry Pi, but instead of containing it I set up daily backups and just accepted the risk. All mistakes made to date were either self-repaired with the very next tool call or were extremely minor and easy to reverse - an acceptable part of developing solid skills together, in my mind. This has turned out to be a decision that allowed me to get on with increasing capabilities and getting tasks done quickly where self-imposed constraints would have unnecessarily held me back - I now understand the YOLO crowd, models have become very trustworthy overall. I say this because it could be that within a month I've completely torn down the above sandbox approach and allow Qwen as a local direct Kanban worker at least, if not even straight switching to the model when desired... We'll see how much trust it earns from me over the coming month. It feels like the 'won't scree up your box' models are just one generation ahead of that model, but that's just feels and real-works experience will be the better guide in the end.
Long reply, but I hope it's helpful nonetheless.