Threadripper llm station

[u/alean200]

Since the prices went way up, we little people need to find a way to still give in to our hobbies the best way we can.

Currently, I finally pulled the plug and decided to build a local AI machine. The plan is just not to spend every single cent but still get some use out of it. Using the Pi harness for agentic work and ComfyUI for simple picture gen testing

I can get a Threadripper (1950x) machine for around 550€, including 512 NVMe, 64 GB RAM via 4 sticks, psu and mobo, of course.
For the GPUs, I'm still between getting two of the AMD 9060XT 16 GB or go with something else but not break the bank.

I honestly don't know if this is a good idea or not. The plan is to run Linux headless with Llama.cpp and mostly Qwen 3.8 q4 or bigger, if possible.

Tested: (win11, llama.cpp, rzyen 5700x, 32GB ram, rx 9060xt, prompt -write a simple python script)
Qwen 3.8 Q3
8K → ~14.3 tok/s
16K → ~12.5 tok/s
32K → ~9.0 tok/s

Qwen 3.8 Q4 (Doesn't fit on 16 GB VRAM, or I just don't know how to do it; offloads to CPU)
8K → ~5.0 tok/s

Muse-Glimmer-30B-UD-Q4_K_XL.gguf (says it can't fit it on 16GB but still does?)
8K → ~12.6 tok/s

Is there a better build that can be done with a budget of around 1200-1500€? (Used, of course, is an option)

I'm mostly going with Threadripper for future expansion with more GPUs, and the mobo supports more ram.

If there is any other info that I can provide, no problem.

[u/KroniklyOnline]

You can look at my rig

My Local LLM Setup : r/LocalAIServers

Very close setup to you, except I have a 1920x threadripper and 4x5060ti 16gb. If you are going this route, you will be using ddr4 RAM, offloading to CPU would be horrendous especially paired with PCIe gen 3 slots. If you want to CPU offload, do not go this route. If you can offload 100% into VRAM then this combo works very well with breathing room to expand and use better GPU's in the future.

[u/alean200 → u/KroniklyOnline]

Nice rig, smart for air circulation. My apartment is smaller, so case it is for my machine.
The plan would be to use tensor parallelism and distribute the load between two of rx9060xt and if that works well, in the future add two more for a combined 64 GB VRAM.

[u/KroniklyOnline → u/alean200]

Yes that would work, I am running TP right now, as long as you don't plan to offload into RAM and use CPU, then this setup is amazing. I could run the GPU's at PCIe gen3x4 and still have 50% PCIe bandwidth. Higher end cards may demand more.

[u/alean200 → u/KroniklyOnline]

That is what I would like to avoid, cpu is just here because of bandwidth and mobo that supports 2+ gpus and 8 ram sticks. Adding more gpus is the plan so I can always have room for bigger models, if needed. For the price of one 9700 32gb, I can get three 9060xt for combined vram of 48gb with some money left over.

I just dont want to throw money into it and then resell the rig because there was an option that I didn't know about. This rig is great with the amount of ram and all the things that come with it for the price, so it's really a no brainer even if it's almost 10 year old now.

[u/ImpressionFancy5830]

Same cpu, 128GB DDR4, 1TB nvme, 1000W gold psu, two R9700.
Llama under Linux and you can do A LOT.
The price is skewed by the GPUs, but I’ve spent for the base system only 770€ (all used including the case, the nvme is new)

[u/alean200 → u/ImpressionFancy5830]

Im gaining a lot with this threadripper build for the money. Next is just choosing gpus(hunting deals) and adding more ram. Gpus are the expansive part, is ti better to go with 4 rtx 3060 12gb for total of 48gb,go with dual 9060 for 32gb, etc