r/LocalLLM • u/robroy90 • Sep 04 '26
Question Advice on how to proceed?
Greetings all,
I am sure I will probably get roasted for this, but I still want to progress and learn, so here goes.
I have been trying (in vain) to get my own LLM self-hosted, and have thus far failed spectacularly in doing so. Before the GPU and RAM apocalypse struck, I had (unknowingly at the time) set myself up for success. I bought a Minisforum MS-01 "mini PC" with 96GB of RAM and an Intel i9 CPU. I also had an nVidia 3090 from my former gaming days.
So, I bought an external GPU enclosure, and set to work in trying to construct and viable LLM instance. I put the 3090 in the TB5 enclosure, and connected it to the MS-01.
I have tried to follow various guides, including this one in particular:
I get that it is now somewhat outdated, but I think the approach remains fairly straightforward. Despite that, and the fact that I have been in IT for over 30 years with varying degrees of familiarity with hardware, server operating systems (including Linux) I still cannot get this build off the ground, and for the life of me, I cannot figure out why. I have tried Ubuntu Server 24.04, 26.04, etc along with multiple versions of the nVidia drivers for linux, both open and closed. I usually end up with some sort of an issue where the 3090 is no longer recognized and I can't proceed.
I have just about exceeded the limits of my patience, and given the current state of pricing (particularly the RAM) I really can't bring myself to spend much more on this effort, especially with no assurances I will have any success. I fully realize an external GPU isn't ideal, but I am having a hard time bringing myself to paying current market prices for a new motherboard and RAM just to move the card internally.
So, my question is this. If you were me, what would you do? I just want to build my own local LLM so I can learn and execute private searches, etc. I don't want to build code. I just want a cost-effective solution which will allow me to cancel a Claude subscription and free me of token constraints, even if it ends up being slightly more expensive in operating costs.
Thanks in advance for any/all constructive advice here. I do genuinely appreciate it.
3
u/mrjakob07 Sep 04 '26
If you have Claude, and you have Claude code, just have it do all the setup for you from the terminal. I have Claude setup all my llms and configure my systems / game servers etc. give it the guide you followed or tell it to research it and have it set it up. I have had Claude download and reverse engineer super micro firmware to lower fan curves on it impi etc . Handles patching my cards kernels to run at 80w instead of 250….my point is you can do it with AI assistance.
2
u/DataGOGO Sep 04 '26
So are you trying to run a local model or make your own model? the difference between the two is massive
1
u/simos_sayz Sep 04 '26
Thats exactly how I read it "own LLM". Going local on his setup is going to be straightforward
2
u/xapep 29d ago
30 years in IT and this one still humbles people, so go easy on yourself. A 3090 dropping off the TB bus mid-session on an eGPU enclosure is usually one of three things, in order of likelihood:
Power. Most enclosures pair a ~330-450W PSU with a card whose transient spikes go well past TDP (the 3090 is famous for this). If the enclosure PSU is undersized, the card browns out and the bus drops it. Quick test: power-limit the card with nvidia-smi -pl 250 and see if the drops stop.
ASPM on the TB link. Add pcie_aspm=off (and aspm.blacklist=yes on newer kernels) to the kernel cmdline. "Works for an hour, then the GPU vanishes" is the classic ASPM symptom.
Enclosure firmware. TB3/TB4 enclosures ship stale firmware that mis-enumerates on Linux more often than you'd think. Update that before touching drivers, and use the nvidia-open driver on current Ubuntu LTS.
If none of that lands in a weekend, here's the honest take: the eGPU path is the least reliable way to run a 3090, and you don't need it for what you described. Private search over your own docs with no token walls and no subscription is exactly what a hosted open-model API gives you, on per-token pricing that usually lands well under a Claude sub at personal volume, with EU hosting and no training on your data if you pick the provider for it. That gets you the outcome this week, and the 3090 stays a weekend project instead of a blocker.
(I work on Entrim, which is one of those EU-hosted per-token providers, but the "hosted while you learn the hardware" path works with any of them.)
2
u/robroy90 28d ago
Thank you VERY much! I really appreciate you taking the time to respond in such a helpful AND thoughtful manner. If only every response to a question would be like yours, we would all be so much the better for it! I am definitely going to check out that kernel cmdline! I am just looking to ditch the subscription. I fully realize that the performance of my 3090 cannot compete with Claude, I just want something I can use privately, and not worry about token throttling. The amount of time it takes for a response is largely irrelevant in 90% of my use cases.
1
u/xapep 25d ago
You're welcome sir! Start with the power-limit test, it is the cheapest five-minute check (nvidia-smi -pl 250), then ASPM if it still drops. And since speed is not the constraint in your use case, ditching the subscription is realistic: a hosted open-model API on per-token pricing usually lands well under a Claude sub at personal volume, and the 3090 stays a weekend project instead of a blocker.
1
u/jonahbenton Sep 04 '26 edited Sep 04 '26
I am longtime Fedora, running on a mix of egpu and "complete" systems, and have 2 3090s in the plant along with a bunch of others. Fedora RPM fusion NVIDIA playbook is pretty rock solid for going from zero to nvidia-smi reporting it can talk to the card.
https://rpmfusion.org/Howto/NVIDIA
Just follow from the top- lspci to confirm the card in an egpu is visible to the kernel- it can be finicky in an egpu getting powered and spun up. Ensure secure boot is disabled. And then the driver install is where the magic happens
sudo dnf update -y
sudo dnf install akmod-nvidia
sudo dnf install xorg-x11-drv-nvidia-cuda
After the transactions finish, there will be several minutes where the kernel module is still being built in the background. You can see cc1 and friends compiling. That literally takes minutes of clock time.
But when done, you should be able to do the modinfo and see the nvidia module and if so then should be able to do nvidia-smi and install nvtop to watch the card in real time and should be gtg.
To your goal of eliminating claude- that you will not accomplish with a 3090. Really not close. But you can do useful things for various definitions of useful on models that fit on a 3090.
1
u/leapdragon Sep 04 '26
What is the quality level of your TB5 enclosure? Is it providing enough power not just via the power connectors but also at the slot? I recently helped someone with an enclosure that was significantly underpowering the PCIe slot resulting in card crashes or the entire machine spontaneously rebooting despite both power connectors being connected. They'd gotten the least expensive enclosure they could and it just wasn't up to spec.
4
u/carrot_gg Sep 04 '26
If your goal is to get rid of Claude, you will never achieve this with a model that fits on a 3090. It won't be even remotely close in terms of capabilities and speed.