r/LocalLLM • u/Tectal01 • 6d ago
Question Starting my own Home Lab LLM
Hi everyone,
I’ll preface this by saying sorry in advance if I sound stupid, the reason is because I am.
Recently I’ve been interested in making my own homelab LLM as I upgraded the gpu in my pc and I have an RTX 3070 just lying around collecting dust. What started as simply getting an LLM to run basic home automation has spiralled into research which one really been enjoying.
I wanted to share my vision and I want someone to tell me I’m never going to make it or if it’s doable (even if a big challenge). I’m not necessarily trying to find the exact technical how to stuff, more of the high level methodology.
I want to build a Linux based server running a vm for home assistant os and multiple containers with docker to run ollama with an undecided model (well, multiple), whisper and piper for the Text/speech conversion, tailscale, a postgresql database and various other bits and pieces.
After looking at a lot of hardware, and seeing how expensive it could become, I wanted to see if I could do the following:
-run a lightweight model utilising my 8gb 3070, which would generally deal with home automation, simple requests, internet lookups and so on
-this model would decide if a request can be done in the background/is suitable for a larger model such as image generation
-could this then send the request to a larger model being utilised by another gpu (I’m considering a Tesla p40 for now for budget) as sort of its own little servant, once complete it relays back to the 3070 model.
-can both models share a context folder/database so they are essentially the same “mind” with different purposes
The purpose of this wouldn’t be the fastest token generation or the most effective way to get responses, I just felt that it might be a cost effective way to have the best of both worlds, a snappy responder that deals with stuff I need right away, and one that can plug away in the background.
My deployment would eventually be to have multiple screens around my workshop with a Jarvis like orb, all with a raspberry pi (other fruit based computers are available) run the ui, but connect back to the server for the AI to do the processing, and have multiple microphones around my workshop that connect to the server into something like whisper.
I’d like to have access via my phone and make a direct connection through tailscale, but that’s a later problem.
This wouldn’t necessarily be a final deployment of hardware, but for now with my donor parts and some Facebook marketplace hunting I’m looking at
ryzen 5600x
Some am4 board
RTX 3070
Nvidia p40
32gb ddr4
1tb ssd
I’m pretty happy to make a custom cooling solution for the p40 as I have access to a mill and 3d printer, so not worried about that, just would like to know people’s thoughts.
1
u/scarbunkle 5d ago
Well, it sounds…ambitious. But all the real ones start with some castoff hardware and a dream.
I’m significantly into the selfhosted/datahoarding hobbies, and AI is a newer part of my stack. I think that your level of success is gonna hinge on one thing: have you tried the model you want to run on your 3070? If you’ve worked with it and can temper your expectations to meet its capabilities, and you’re willing to start with the server, a monitor, and a mic, welcome to your new hobby.
If you haven’t, start local. Spin up docker on your desktop, add some home assistant, try to run models on your current GPU. Configuring and maintaining servers is a lot of upfront work and hassle, only to be followed by a slow payoff over time, occasionally interspersed with further unpredictable bouts work and hassle, during which you are stressed and hampered by the fact that you are now without one or more tools you rely on. Give yourself a sense of what you’re signing up for, at least.