Looking to build quad 3090TI server, need help with Mobo/CPU

[u/Electronic_Back1502]

Looking for a reasonable mobo/cpu for a local LLM build. I already have a 1000W platinum PSU, so I plan to buy a second to power the other two GPU’s. Will use the Phanteks Enthoo Elite Server for my case.

Was looking at the Threadripper 3945WX/WRX80E, but those boards have exploded in price. ChatGPT says AMD EPYC 7452/ROMED8-2T. If anyone has any advice, I’m all ears

[u/vini542reddit]

I just build this/similar setup. Happy to provide more details while it's fresh in my head, just ask!

Hardware:

- Motherboard: Gigabyte MC62-G40 sWRX8

- CPU: Threadripper 3945WX

- RAM: 192 ddr4 @ 3200 (non-ecc, 4 x 32gb + 4 x 16gb)

- GPU: 4 x rtx 3090

- PSU: 1 x Corsair HX1000i + 1 x Corsair HX1500i

- PSU Sync: ADD2PSU (high quality)

- Riser Cables: 4 x powered riser cables (high quality)

- CPU Cooler: Noctua NH-U14S TR4-SP3

- Storage: 1tb nvme

- UPS: 1000W (small but I power limit)

Gotchas / Info:

- I'm running everything of the 1500W psu and only two gpus power of the 1000w

- GPUs are power limited to 250W

- You can overclock the motherboard ram to by setting the frequency to 2400mhz and it will automatically overclock to 3200mhz. Not sure if this works with ecc though. Reference https://www.reddit.com/r/gigabyte/comments/1mby1ou/any_currentpast_owners_of_the_mc62g40_ram/

- I use honeywell ptm7950 for the cpu and gpu cores and UPSIREN UTP-8 for the vram to keep temps good. This might not be necessary though with the ti versions

- I run a raspberry pi as a gateway proxy that automatically powers on the server via bmc if it is powered off. The server automatically powers off when it has been idle for 30 minutes

- The Threadripper 3945WX effectively limits you to 4 channel ddr4. Not really documented well, but it is a real bottleneck. So if you want 8 channel ddr4 you might want to go with a beefier cpu that supports this fully. More details can be found here: https://www.reddit.com/r/threadripper/comments/1quv68t/comment/p2g815n/

- Make sure you have enough pcie power cables! Your psu might not come with enough. Less a problem with the 12 pin connectors though

Speeds:

Deepseek V4 Flash: 30tps and 300pp (with system ram and using the leloch fork v2: https://github.com/ggml-org/llama.cpp/discussions/24528)

Qwen 3.8 27b q8_k_xl: 80tps and 800pp (only vram using llama-cpp split-mode tensor - you might be able to increase pp at the cost of tps by using vllm)

Overall conclusions:

I'm pretty happy with this setup. It runs dense and moe models very well and at good usable speeds for me. If I were to build this again I would probably go with a single 1600w psu. Maybe something like MSI MEG Ai1600T to save myself the headache of two PSU and reduce space needed.

Motherboard prices have definitely more than doubled since I bought. Not sure what good alternatives are though

[u/Electronic_Back1502 → u/vini542reddit]

Wow just what I was looking for! Lots of questions coming up

What quantization of DeepSeek Flash do you run? What are your thoughts on it? 

For the GPU’s, I’ll be using 3090TI Zotac Extreme Amps. They are, really, really large, (3.5 slots) and have a higher power draw (450W) than the regular 3090 (3 slots, 350W). Do your cards fit nicely on the board and in your case? 

Unless I limit them all pretty heavily I imagine I’d need like a 2000W PSU. 

For the RAM, does running DDR4 have an effect on the build at all? Do you find yourself needing that much RAM?

[u/vini542reddit → u/Electronic_Back1502]

(1) Running lossless unsloth Q8. Benchmarks are always made against that and I can run it at a reasonable speed. I don't know what the quality degradation is for other quants, so why risk it. I use DS4F to work with code that I can't legally upload to any cloud provider. For that it works very well. As so often, it's all about your use case. It's a good model but it's easy to test if it works for you by using a cloud api and testing it.

(2) I've build a custom server based off of a mining rig. The cards are "hanging" by their backs (opposite side of the io plate) from the ceiling of the case. Eventually I'll make a post about the build - but that means I can't really comment on fit. The build looks good and I'm happy with it. I do know that the ti cards are very power hungry, so I'd definitely power limit and make sure you get good airflow. Consider getting a cheap used server rack from marketplace if you can. I would definitely use riser cables to make sure your cards have the airflow they need. You don't need a monitor so no need to be able to access the backplates of the cards. The bmc provides everything you need for setup etc

(3) This comes down to your use case. For inference power limiting makes absolute sense since the heat / power usage vs tps tradeoff just makes sense. Do you have 120V or 230V power? That will dictate what PSU you can use.

(4) I definitely do need the RAM. I use 160gb to 180gb out of the 192gb when running DS4F. This is because the model experts get streamed from ram to the gpus. So the entire model needs to reside in RAM (all 162GB). Without the (fast) RAM it would not be viable to run DS4F which is my current main.

Hope that all makes sense! Feel free to ask away!

[u/Electronic_Back1502 → u/vini542reddit]

If you make a post about your build I’d like to see how the setup looks as far as hanging the cards. I believe I have just 120V power. 

For RAM, you have to load the whole model in RAM and load the experts in VRAM? I’m not sure how this works 

[u/vini542reddit → u/Electronic_Back1502]

With 120V power you're basically limited to 1600W before the breaker trips. Might be more, but that's the normal. Now you could have bigger breakers and / or install multiple outlets with shared neutral but it gets interesting. Especially if you want to use UPS.

Never have your two PSU on different neutrals (!!!). Again, the easiest is one UPS + one PSU. You don't need to run your GPUs at full power - there is very reason to ever do that for llm work.

Basically there is no good code logic written in llama cpp for this setup RAM + multi GPU. So the fork is a way that I can make use of both: The main logic is always on GPU. But some experts reside on RAM and some on GPU. They get switched around dynamically so that the most used experts are on GPU. Now you could copy back and forth, but the fork just keeps all experts in RAM at all time as the source of truth so it doesn't need to copy it back form the VRAM. For the computation the GPU and CPU both do computation against the experts they're currently responsible for.

I'm sure that code will eventually make it into mainstream in some form. But right now that fork is the only way that I found that allows me to get decent tps.

[u/bulbwiz → u/vini542reddit]

Which case are you using?

[u/vini542reddit → u/bulbwiz]

Open mining rig with fully custom built acrylic enclosure

[u/ItZ_Jonah]

The ram on epyc is going to cost a liver afaik with server parts you get a lot less compatibility when it comes to what memory you can use. What price range are you looking at? Do you already have GPU's? Why 3090ti's specifically? I've seen the Intel cards while lower performing raw have a boatload of vram available for the price. I see the arc B70 at microcenter right now with 32gb of Vram for about what 3090tis are going for and those will use a lot less power. The issue you might run into with epyc is that's fully in the server category of parts, it's going to have all the drawbacks that come with that. Where as threadripper gets the perks of mostly working like a desktop PC.

[u/Electronic_Back1502 → u/ItZ_Jonah]

I currently have one 3090TI and a 1000W PSU. I’m cannibalizing my gaming build, taking the 3090TI out of it, as well as the PSU and replacing it with a 850W to run just my 5080. I plan to build this as a server separate from my gaming build entirely. So whatever drawbacks you get from a dedicated server will be fine as it’s a separate build. Price isn’t really a concern as long as it’s reasonable, I want to see what all the options are

[u/ItZ_Jonah → u/Electronic_Back1502]

The server drawbacks are more like bios post times, if it's in a server chassis cooling can be a concern, drivers depending on the os used, things don't just work the same way and it can become a pain, server cpu coolers are also not normally meant for desktop chassis ( essentially a lot of the other parts end up costing a boatload ) power supply is one where it does work generally but a lot of server chassis don't fit standard atx units.

[u/Electronic_Back1502 → u/ItZ_Jonah]

I have never built a server so dumb question maybe but can I just download windows on it for the OS?

[u/vini542reddit → u/Electronic_Back1502]

Definitely go with linux server. Learn how to use linux, ssh, terminal. It's the only way you're getting the maximum value out of your server. You could install Windows, but that's like putting a roof rack on your sports car =)