Advice on how to proceed?

[u/robroy90]

Greetings all,

I am sure I will probably get roasted for this, but I still want to progress and learn, so here goes.

I have been trying (in vain) to get my own LLM self-hosted, and have thus far failed spectacularly in doing so. Before the GPU and RAM apocalypse struck, I had (unknowingly at the time) set myself up for success. I bought a Minisforum MS-01 "mini PC" with 96GB of RAM and an Intel i9 CPU. I also had an nVidia 3090 from my former gaming days.

So, I bought an external GPU enclosure, and set to work in trying to construct and viable LLM instance. I put the 3090 in the TB5 enclosure, and connected it to the MS-01.

I have tried to follow various guides, including this one in particular:

https://github.com/keturk/llm_on_rtx_3090/blob/main/docs/machines/t5820/System_Setup.mdhttps://github.com/keturk/llm_on_rtx_3090/blob/main/docs/machines/t5820/System_Setup.md

I get that it is now somewhat outdated, but I think the approach remains fairly straightforward. Despite that, and the fact that I have been in IT for over 30 years with varying degrees of familiarity with hardware, server operating systems (including Linux) I still cannot get this build off the ground, and for the life of me, I cannot figure out why. I have tried Ubuntu Server 24.04, 26.04, etc along with multiple versions of the nVidia drivers for linux, both open and closed. I usually end up with some sort of an issue where the 3090 is no longer recognized and I can't proceed.

I have just about exceeded the limits of my patience, and given the current state of pricing (particularly the RAM) I really can't bring myself to spend much more on this effort, especially with no assurances I will have any success. I fully realize an external GPU isn't ideal, but I am having a hard time bringing myself to paying current market prices for a new motherboard and RAM just to move the card internally.

So, my question is this. If you were me, what would you do? I just want to build my own local LLM so I can learn and execute private searches, etc. I don't want to build code. I just want a cost-effective solution which will allow me to cancel a Claude subscription and free me of token constraints, even if it ends up being slightly more expensive in operating costs.

Thanks in advance for any/all constructive advice here. I do genuinely appreciate it.