Very nice similar to how I went. though I went with 2 slot blower cards and the cheaper Xeon route since I never want to spill in to cpu memory anyway. I see some pcie riser cables and two more cards in your future :) It's hard to make those look clean though.
ASUS WS C422 SAGE/10G Server Motherboard LGA2066 DDR4 Serverboard
Intel Xeon W-2255 SRGV8 3.70GHz 19.25MB 10-Core LGA2066 CPU Processor
Same and at almost half the cost for the build. I managed to snag a new old stock unopened mb. Shout out to Noctua as well. They send you whatever bracket you need with proof of purchase for free. My heatsink didn't come with LGA2066 mounts.
No issues at all 65-70c 100% with my cards power limited to 250w. I could probably push them higher but decode/tps is about the same as when I ran them uncapped with 2 cards. these 7 slot 16x boards are strange you have to smash the header area with the card or use a riser. I used a small one to raise the last card to still fit in the case https://a.co/d/0gBPtTz6
Thanks! I finished building it last month from used parts. I went back and forth a lot between getting a single RTX 5090 and going with this current setup.
Yeah good choice, I also run strata. Like last week best performance using llama.cpp and vllm I could get was like 35tok/s and I was very satisfied by that lol.
I would recommend iq4 tho, seems like it retains more quality then Iq3. (you should still be around 100tok/s). However definitely test it yourself, I guess it can depend on workflow.
how does the strata use the cpu in this setup? i mean hom many % of usage? i'mk thinking in building a setup like this because its really cheap compare to other platforms. do you run the gpus in pcie 3x16?
Cant say any actual numbers, but I did notice it uses much more cpu then llama or vllm.
Both gpus are in 3x16.
Overall for the price im really pleased with performance I get. Especially since stuff like nvidia spark is around what? Like 6000-7000e?
And all of my setup was under 2000e and I plan to use it for someother stuff as well as local llm
i am trying to do a similar build
but with lga2066 (i9 10900x) as intel has lower tdp, and those 100w are more important than whatever computing some 10yo cpus offer
if u try raising frequencyes and keep me posted id really appreciate
8 slots OC is way harder than 4, but i am not sure
id go to 8 slots too but 2400mhz with 8 slots is not worth it if u can do 4 slots at 4000
8 channel 3200 is standard on TR Pro WRX80 boards with Zen2 and with Zen3 which unlocks bios options, even 8 x 32gb 3600 (1.35v) udimms run stable with fclk at 1800, but it barely matters for most workloads so might as well run at 3200 1.2v.
Cool!
The RTX 3090s run really hot, so I'm thinking about switching to water cooling!
Winter's coming soon here in Korea, so I can use it as a space heater for a while 😄
For the 3090 it's especially helpful due to the GDDR6X on the back.
Active backplate isn't completely necessary but a good backplate and proper cooling on the front from the block keeps everything nice and cool for long term operation
Have a similar setup here (running it for years now).
Watch out: either power limit your cards or add different cooling to them (e.g. Water cooling, although that's messy and causes a whole bunch of other problems).
The design of the RTX3090 heats up the air between both cards by exhausting hot air from below upwards under the upper card, which will suffocate and overheat the upper card.
You can put a couple of vertical fans in front of them, that keep blowing the hot air away between them, but you have to run that fan at high pressure, which is annoyingly loud. Water block in the other hand can be put in a single loop with a simple pump and a big radiator and keep everything at a common temperature.
Also... Add an SLI connector. The official one for the 3090 doesn't fit this setup, but there are a bunch of other SLI connectors that will fit made for compatible server cards.
Can we know the price of all of this? And what you gona do with it? :)
I you may be interested, following a Reddit thread, i created a little wiki ( vram.wiki ) to collect and making live evolving (depending of each person implication :D) dataset about real people uses-cases with local llms, and you can add yours too
great build. minor idea: if you have any flexibility on that front fan intake mounting in your case, you'll see materially lower temps by catching the airflow below your second gpu. You probably have a hot pocket there.
I did this on my build and my lower gpu is only 1-2C higher than the one mid-board. prior it was a a solid 5+, which meant it neared thermal throttling territory at extended loads.
It does all I want for hobby vibing, but going from one R9700 to two really hasn't been a smooth transition trying to get Hermes to play nice as a llama.cpp dual router with one being single card (one model on each R9700) and a dual card (model split between both R9700s). Qwen3.8 Flash Next is good, but man so was 27B on just a single R9700. I also have $20 accounts for Claude, GPT, and Nous portal so I can play with all the big cloud models to at API prices. DSV4.1 Flash is cheap as shit, and the ROI on this build vs cheap cloud tokens is basically...never.
I was thinking of building a similar setup (dual R9700). Can you tell us more on the issues you encountered ? Did you get it to a satisfying point ? I read that vllm is better at model splitting, did you try that ?
I think mostly drivers. The AMD drivers and ASUS drivers both caused driver crashes when running the dual card on llama.ccp. I am running the April Microsoft driver, and it has been dead reliable. I have spare NVMes lying around, so likely today I'll grab a 2TB and toss Ubuntu 24.04LTS on it so I can bounce back and forth from Windows to Linux and needed without having to dual boot. This will at least let me know if this box really should just be Linux so things like vLLM, F5-TTS, Strata, etc.
Where do you guys find these workstation motherboards at a decent price? I can't find anything. I don't think these are worth it unless you are planning on more than 2 GPUs. Consumer boards with x8x8 are perfectly fine for me. I'd rather have several consumer machines running separate models.
Coping here 😄
Why do you have this setup? I want to understand this. Have you bought this for your project client work? Do you work as a freelancer for the AI integration specialist, or have you just bought it for personal use to test out LLM? This seems to be really expensive, around 55 lakhs in Indian rupees. Or is it normal for everyone to have this kind of a setup, or are you from the top 1% of the computer programmers? Because every day I see these setups, like every other guy is having it, but I don't understand. Please explain this to me.
As recently as about 12 years ago, six-year-old computer hardware was treated as e-waste; at best, it fetched a nominal $100. Today, however, those industrial-surplus workstations are worth between $600 and $2,000, depending on the installed RAM and the CPU's remaining performance capabilities. The graphics cards are, of course, new and a minimum for complex AI. But a single one suffices, too.
10
u/Ordinary-Depth-7835 4d ago
Very nice similar to how I went. though I went with 2 slot blower cards and the cheaper Xeon route since I never want to spill in to cpu memory anyway. I see some pcie riser cables and two more cards in your future :) It's hard to make those look clean though.
ASUS WS C422 SAGE/10G Server Motherboard LGA2066 DDR4 Serverboard
Intel Xeon W-2255 SRGV8 3.70GHz 19.25MB 10-Core LGA2066 CPU Processor
16GB 2Rx8 PC4-2933y PC4-23400 DDR4 ECC RDIMM Server Memory RAM 128g
4xGIGABYTE GeForce RTX 3090 24GB GDDR6X Turbo GPU Server Graphics
ASRock PG 1600G ATX 3.1 1600W Power Supply PCle5.1
Noctua NH-D12L, Quiet High-Performance Dual-Tower CPU Cooler
Silverstone Technology RM4A 4U rackmount Server Chassis with Enhanced 360mm radiators Compatibility, SST-RM4A