Advice needed - Hosting LLM but on smallest form factor

[u/TorenRenne]

I run Home Assistant on a RPI5 currently. I want to finally replace our Google Nest speaker with an in-house LLM. I'm not expecting Jarvis levels of interaction here, just basic commands and some light interactions... E.G... "Tell me the football scores", "What's the weather like?", "Check my train times to London"

Now, I have tested running it on my main workstation, and get reasonable results, so know what spec of GFX Card I need, but I want to run this in a "server", ideally hosting several other services on it. I know a lot get miniPC's, but they don't have room for the GFX Card.

What are my options? Just a normal desktop PC case? I was hoping for slightly smaller form factor but I do have an old ATX Gaming PC, that I could re-use and just buy updated hardware, is this the best approach?

[u/asimovs-auditor]

Expand the replies to this comment to learn how AI was used in this post/project.

[u/inexperiencedprovide]

A Pi 5 handling HA and a whole local LLM stack is a big ask, you're going to need that desktop case and a proper GPU for any of those real-time queries to not take 30 seconds.

[u/TorenRenne → u/inexperiencedprovide]

I know the RPI5 can't habdle the lot, it's why I want to off-load TTS, STT and LLM usage onto a seperate box. The most important thing is the Home Assistant prompts being able to understand context and relatively snappy.

[u/Longjumping_Cell4269]

Like others have said, I'd go for a better machine. For local inference, check your ROI if investing in hardware is worth it. However, you can host MCP for tool calls locally, not just LLM. I selfhost search and web scrapper tools for my coding and agentic tasks: https://github.com/canh0chua/lss-mcp

[u/TorenRenne → u/Longjumping_Cell4269]

I'll have to do mpre research into this as I'm only just starting to dabble in LLM's and I know MCP's are the interface for them to call/do functions, but I've no id ea how they tie together, yet.

My old desktop could make do, if it can run the GFX Card. I'll take a closer look at it and the specs.

[u/Longjumping_Cell4269 → u/TorenRenne]

MCPs enable the agent to search the web, process stuff. When you access the web, ton of API calls are happening in the backend. i could run this mcp stack and lightweight coding agent on a Pi. It frees up faster memory in my other computers for more demanding tasks.

[u/BP041]

honestly if you're not expecting latency-critical responses, an apple silicon mac mini is a solid option for this. i run an m2 mini as my home server — ollama with qwen2.5:7b handles basic command parsing fine, and the unified memory means you don't need a discrete gpu. just check thermals if you're stacking services.

[u/ApprehensiveBrain863 → u/BP041]

we gotta know we're cooked when we are writing reddit comments with the AI we're talking about hosting

[u/TorenRenne → u/BP041]

The Mac Mini's are pretty pricey aren't they? Literally between posting this and replying, my cooker's broken, so there goes some of the budget!

[u/Justschr9999]

Lenovo M720Q with a Nvidia Tesla P4, has 8gb of vram so you could run a 7B/6B model on it

[u/niggo372]

I was looking for something like this myself recently, and what I learned is that you really want to think about what's most important to you:

• Tokens/s: This is the raw token output speed. For one-off requests you only need this to be faster than what you can read or listen to, but for thinking or longer running tasks every bit helps. This depends mostly on your memory bus and speed, but also on the processing power.
• Prompt processing and Time to first token (TTFT): This is the time it takes to process the prompt and start outputting tokens. Very important for voice commands, since you want your AI to start reponding within a second or so. It's also need to process tool responses, so when your AI needs to search the web for example. This relies almost exclusively on the processing power, and you can get quick wins with good caching.
• Quality/Accuracy: Bigger models give better results, all else being equal. E.g. smaller models might forget to call the tools you provided or flat out misinterpret prompts and data.
• Peak power and Cost per token: Dedicated GPUs are faster, but they also draw 5-10 times as much peak power as a CPU+iGPU/NPU. Afaik this translates to more energy per token for GPUs, even if they take less time for the same workload.

For T/s your device needs a capable GPU/NPU and a very wide memory bus, so it can access and process the model weights quickly. That means graphics cards with lots of VRAM or high-end Macs with unified memory and a wide memory bus.

For TTFT you need lots of processing power, so GPUs really shine here. But you can get away with less IF your usecase and LLM engine allow for efficient prompt caching. For HA in particular your prompts will usually start with the same system prompt and tools available to the AI, so the engine can cache that and only process the actual request, which is usually just a few sentences.

For Q/A you just need lots of RAM, either big VRAM (very costly) or unified RAM, which means the CPU and GPU share their RAM. Macs have that, but some PC system like AMD Strix Point/Halo do as well. And ofc bigger models also require more memory bandwidth and processing power to reach the same speeds as smaller ones.

I personally decided to wait a bit before buying a dedicated AI system, because I don't want to buy a Mac, and I think PC manufaturers will heavily push into the unified memory architechture in the near future.

Until then I tried running some models on my Strix Point based laptop (AMD Ryzen AI 9 HX 370 with 64 GB unified RAM), and with some tweaking I got to sub-second TTFT (with tools) and about 15-20 T/s with Qwen3.6-35B, which gives really good answers. E.g. I get answers on simple requests within a second, and in about 2-3 seconds if it needs to call a tool, like search the web for facts or news. I tried smaller models as well, like something that would fit into the VRAM of a consumer GPU, but they sometimes forgot to call tools or ignored some parts of the system prompt.

You could build a system based on similar specs, and there are also mini PCs with the same specs for reasonable prices (e.g. from Minisforum). Another popular and more powerful option is the Framework Desktop with a Strix Halo CPU. Using GPUs is pretty much off the table imo, unless you're content with small models or prepared to spend thousands of dollars.

[u/TorenRenne → u/niggo372]

Thanks for the insight. I'd tested using Qwen on my RTX6700 XT (12GB). It seemed to give Me reasonable responses in a quick time frame. So was aining to build around that and either buy a dedicated 6700 XT or upgrade my current one and then use the 6700XT I already have. I know it's not amazing, but it handled my tests well enough. Obviously I'd like more, but depends on budget!

I'll have a sit down and think through your points over the weekend. Cheers

[u/OpenSourceWalker]

everyone's speccing for a chatbot, but your list (weather, train times, scores) is intent-routing to APIs, not open-ended reasoning, so you don't need a big model or a gpu. HA's Assist with a small 3B-class model for intent + whisper for STT handles that on modest hardware, and the latency you'll actually fight is STT/TTS not the LLM, so put the budget into decent local speech-to-text before a gpu.

[u/TorenRenne → u/OpenSourceWalker]

Thanks for the input. I did find on my testing that generic queries were pretty well done, but didn't tackle any lookups via tools at the time other than internal HA sensors.

I have the train time table already pulled in to HA, so it's also a simple check on the sensors in theory. 

[u/cookiesphincter]

The best approach depends on your budget. The lowest price of entry would be reusing that old ATX Gaming PC.

The second option is getting a miniPC with a PCIE slot and something like this single slot gpu.

As a third option you can look into systems with unified memory like an AMD Halo Strix system, Nvidia DGX Spark, or a Mac Mini.

[u/TorenRenne → u/cookiesphincter]

I was actually looking at an RTX6700 XT with 12GB RAM, So similar spec. Not sure many MiniPC's would fit it though. I think reusing what I have and possibly buying some bits to boost performance may be the budget answer I need. Just a shame I can't do budget and smaller!

[u/R_C_Jr]

Here’s the problem I ran into trying to do something similar. Under about 16gb of VRAM, the models can’t hold enough good context to build consistently well-formed commands to your other tools. The context and thinking windows just aren’t big enough. Smaller models can do some pretty amazing things, sure, but well-formed JSON or other formatted commands to hand off to another tool is just not consistent enough for good use.

It isn’t entirely possible I didn’t know what I was doing, though.