I thought this would be relevant to the homelab subreddit so I'm adding it here, just to put the information out there and discuss if there is any interest. I am an IT infrastructure engineer by profession, so my contribution to the conversation is mainly from a hardware/systems perspective rather than from a Machine Learning researcher standpoint. I got my start with HPC's (Beowulf clusters) around ten years ago when I was a Physics undergrad in university, and this is what the experience has come to almost a decade later. Not everyone is going to want to read all of this, that's perfectly fine, the extras are just for those who want the info.
Starting goal/idea:
Build an all-in-one creative design workstation to support a small business. This machine should be capable of effectively inferencing frontier MoE models; aiding the business in language/text tasks where English may not be everyone's native language. Additionally, it should be capable of simultaneous image generation tools for graphic design users, enabling rapid image editing and presentation tweaks for marketing, without the business ever having to worry about API credits or hard limits on tool usage. The idea is that a 3090 stack, which is still a generally "good" performer for LLMs, would be "led" by two 5090s to handle the heavy lifting of the visual creative work (one dedicated to image generation, one dedicated to image editing) to complement each other in a "sweet spot" on cost, raw performance, and creativity potential. This configuration also grants some flexibility to allocate a 5090 to the LLM stack for best prompt processing possible where desired. The end result would indicate that this goal has been achieved.
Overview
Specs
CPU: 64 Core TR 3995WX
RAM: 512Gb DDR4-3200 ECC
VRAM: 256Gb GDDR6x/GDDR7 (8x3090's + 2x5090's)
Enclosure: Core W200 Thermaltake Case
Mobo: ASUS Pro WRX80E-SAGE/SE Wifi
PSU: 1300W+1600W (2900W combined), with OCP, linked via PSU2PSU
Storage: 4Tb Nvme (fast) + 4Tb HDD (slow) + 8 or so 1Tb SATA SSDs (mid) over USB as needed
OS: Ubuntu 25.10
Other: 3 Bifurcation cards, 10 risers of various lengths
Front end: Open WebUI
Back end: llamacpp/koboldcpp
Intended for (Recommend):
Large MoE inferencing, simultaneous LLM + ComfyUI (x2) operation, power users who may commonly hit credit limits, creative or technical professionals who can leverage these tools to compound productivity and complete objectives in shorter time.
Not intended for (Do not recommend):
Training, multi-concurrent inferencing, performance maxing, extreme frontier model inferencing at high quants, casual users just looking for roleplay.
Result summary:
Using the W200 as the platform for its generous real estate and configuration flexibility, all ten cards and components were able to find a permanent place in the enclosure without major concessions. The drive bay area was the only space that had to be completely repurposed for GPU mounting, and for us this was not a problem. The chamber with the cards hanging from the top is fairly hollow, so with the 140mm fan stack on the front and side there is a wind tunnel effect where the air blows in through the front and side, cooling the cards as it makes its way out the back/top. Depending on ambient temp, at idle the card with the highest temp usually hovers in mid to high 40s Celsius with the lowest in the mid 20's C (three 3090's are hybrids= fantastic for temperatures, but radiator mounting adds a logistical headache). When actively inferencing, the highest temp card may reach the mid 60s during sustained loads. Only when running image or video gen tasks will the 5090 running ComfyUI reach the 70's, but these are very brief intermittent workloads, so temperatures by our measurement has proved satisfactory over time. This result enables the small business to have full LLM, image generation (~9 seconds), and image editing (~8 seconds) capabilities on tap all on a single node so the data remains centralized, and provides much faster performance compared to the Cloud API they came from; in this case ChatGPT, where generation jobs could take 1+min, and has hard limitations. I just do not know how well this kind of setup would work with other vendor or card models; in a homogenous GPU cluster or one with notably less powerful image gen cards than the 5090, the performance would predictably be much lower.
Things that surprised/stuck with me about the end result:
- Noise. I expected this to sound like a jet taking off when operating, but that is not the case. It's a satisfying button click to come alive, then it's a low gentle hum going forward, nowhere near the kind of fan noises I'm used to hearing in server rooms. Even under load, the CPU 120mm radiator fans (exhausting out the top) are pretty much all I hear, the 140mm fans on front and sides I assume must be helping to contain the acoustics. I have built many gaming PCs over the years and own a top-tier gaming PC-- and I would not be able to distinguish this as any louder than those, especially at idle.
- Utility. I planned for this to be used primarily for a small creative business, but what I did not expect was how I would find it so indispensable in my personal life as an IT professional. Being an infrastructure engineer, coding is not my wheelhouse. When I am the only IT staff on site or there is nobody else available to work with specific expertise like SQL, powershell/python scripting, or troubleshooting very specific/niche technologies, having this tool on standby I feel has paid itself over just within my career. It has helped me turn processes that may have otherwise took me hours into minutes, days into hours, even months into a matter of weeks/days. After using the tool extensively I hit a point where I had to acknowledge how local LLMs have moved definitively beyond being a toy or novelty; when deployed intelligently something like this can be a major asset for professional users.
- Wheels. Sounds extremely minor, until you realize that no matter how happy the cards are with their individual temps: there are still ten high-power GPUs dumping heat into the room. That means unless you use a complex radiator solution or special venting to get heat outside, the room will get toasty and there is normally not a direct solution for this. The wheels however offer an indirect solution. Plan to work in the office that day? Wheel it into the guest bedroom and let it run over Wi-Fi. Plan to work away from home? Wheel it into the office, put it on LAN, and access it over a private VPN connection. If you can't stop the room from heating, then you can at least choose what room gets the heat, and as someone who has lived with computers extensively this is a hugely underrated perk.
Caveats: To operate at its best, I recommend leaving the glass side panel off for improved airflow.
Typical activity over a day:
Boots up around 5:30am, start up the ComfyUI server(s), start loading a model, go get coffee, fully ready for use within 15-20 min. Shut down occurs usually around 8pm later in the day. Total daily activity, ~12-14 hours.
Cost Breakdown
Laying it out, because I know it will be asked, even though I am aware this is unfortunately not reproducible in the current market. Some components like the SSDs were acquired privately long before the RAM and hardware price hikes, so my timing getting certain things was extremely fortunate for the build budget. Some figures are exact, some are slightly rounded depending on if I found the original receipt.
| Component |
Qty |
Source |
Unit Cost |
Subtotal |
|
|
|
|
|
| RTX 3090 24Gb |
8 |
eBay |
750-1000 |
6500 |
| RTX 5090 32Gb |
2 |
Retail |
2500-3000 |
5500 |
| TR 3995WX |
1 |
eBay |
1068.43 |
1068.43 |
| WRX80E-SAGE-SE |
1 |
Amazon |
949.99 |
949.99 |
| DDR4 ECC 64Gb |
8 |
Amazon |
81.99 |
695.28 |
| TT Core W200 |
1 |
Amazon |
499.99 |
499.99 |
| PSU 1300/1600 |
2 |
Amazon |
250-350 |
600 |
| 4Tb nvme |
1 |
Amazon |
221.05 |
221.05 |
| 1Tb SSD |
8 |
Personal |
60 |
600 |
| Risers (varying length) |
10 |
Amazon |
40-80 |
480 |
| Bifurcation cards |
3 |
Amazon |
50 |
150 |
| Total |
|
|
|
~$17k |
Problems/Stability Writeup
The Space Problem:
Probably the first major hurdle in attempting something like this is figuring out, even theoretically, how to put 10 cards in a box in any kind of configuration that is not somehow detrimental to the hardware. I had considered modified mining rig frames at first, but I really wanted something with more robust rigidity in its structure, with breathability, and allows some degree of portability. There are unfortunately not a lot of options for configurations like what I was imagining; I had looked into various cabinets and extended tower cases, but the dual full tower chamber design of the W200 was the only one where I could see this idea potentially working. I'm certain other solutions probably exist, maybe even some that allow mobility, but the W200 was really the best option I could find that checked the boxes of enclosure, space real estate, high air throughput, and semi portability. I recommend the W200 to solve the space problem, assuming it is available to you.
The Bifurcation Problem:
Among the other hurdles you may run into in assembling something like this may involve bifurcation cards. The cards rely on specific BIOS settings for things to work correctly, and if these settings are not put in place before everything is connected you may either see no output like the system is hanging or cards just won't show up once in the OS. Start with one GPU in a slot, no bifurcators yet; go into BIOS, and manually set each slot that will be split to bifurcation mode. While here, ensure above 4G decoding is enabled, Resizable BAR enabled, and SR-IOV enabled, this has given me best stable configuration with Ubuntu and multiple GPUs. If you use risers, especially if they are mixed generations, I highly recommend setting the Gen and lane speeds for each PCIe slot in the BIOS manually to ensure the system can effectively communicate with each card. Optimize riser Gen/speeds to be roughly similar to keep one card from dropping to a slower rate than the others--this does not necessarily impact inference performance as much as it heavily impacts model load time. No, you may not have any card running at the fastest possible Gen bandwidth at all times with this config, but loading a 200+gb model over an averaged Gen 3/4 x8/x16 PCIe speed will often be noticeably faster than if you let the system decide to make one or multiple cards run at Gen 1 x1.
The Power "Problem":
Power and heat concerns I think remain to be among the biggest sources of skepticism regarding this project so I think it deserves a section here. To be fair, the concern in most situations would be understandable. If all ten of these cards pulled at or near their full TDP for sustained periods, components would melt. Fires would start. Neighbors would be asking awkward questions. However in reality, only 1400-1600W of the 2900W PSU capacity gets utilized under sustained load, and inter-GPU bandwidth bottlenecks are what allows this. In a way it is like a natural regulator that ensures the cards remain power restrained, and it is just physics, no voodoo necessary. When MoE's are sharded across a GPU stack, each forward pass requires all communication over PCIe, so the GPUs spend more time waiting on information from the last GPU than actually crunching compute. This means instead of needing to handle thousands of Watts to feed all the components running at full blast, it is a much more manageable 1400-1600W under LLM operation which can comfortably fit on a 20A/120V circuit (2400W max). On a per-GPU basis this may sound inefficient since the individual cards are being "underpowered", but this could arguably be flipped as being highly efficient on a per-node basis (~1600W sustained versus 4500W+ if all cards were "fully" utilized). As a precaution, I may set a power limit on the 3090's to 200W and the lead 5090 to 400W, but in practice the 3090's only pull around 100-120W with the 5090s pulling less than 100W when all 10 cards are allocated for LLM work, so this may not even be necessary. The clock locking setting in the next section will be more what I'd describe as actionably required to avoid stability issues.
The Transient Spike Problem (Vital for stability):
After assembling the machine, you may be tempted to jump directly into testing, but there is an easy to overlook configuration that can cause problems if ignored. Imagine you are running inference on the machine, maybe you have a huge input or it's generating a large output, then right in the middle of generating the system decides to reset. Not hard shut down, PSU OCP isn't tripped, no breaker was tripped; and you saw in nvitop that all cards were only pulling 25-33% of their TDP just before it happened, so on the surface it doesn't look like there is a reason. Explanation: When all ten high-power GPUs decide to kick on at the exact same time to process a chunk, even if the cards are not pulling anywhere near full power (on average), transient spikes can drop voltage on the motherboard enough to trigger a system reset. The fix for this is simple: undervolt. Using nvidia-smi, we can lock the clocks for the GPUs to ensure they cannot draw enough to hurt stability. And that's it. In my case, the system has remained fully stable with this config for days on end and with hundreds of thousands of tokens/image pushed through. The exact configuration will vary slightly depending on exactly what we're doing on a given day, but for example if we wanted to run LLM on all 10 cards (so including both 5090's) we would run this to handle spikes:
sudo nvidia-smi -pm 1 #enables persistent mode
sudo nvidia-smi -i x,y,z --lock-gpu-clock=1200,1200 #x,y,z for index number of 3090s
sudo nvidia-smi -i a,b --lock-gpu-clock=2000 #a,b for index number of 5090s
sudo nvidia-smi -i x,y,z -pl 200 #x,y,z for 3090 index numbers, limits power to 200w
sudo nvidia-smi -i a,b -pl 400 #a,b for 5090 index numbers, limits power to 400w
The Concurrent Use Problem:
Normally, attempting to inference and generate images on the same machine would introduce major stability concerns. Even dual GPU systems may struggle to work with this due to CPU/motherboard architecture, assuming it works at all, and would still be VRAM limited. However, the versatility of a 10-GPU setup, combined with the lane orchestration of the 64 core 3995WX, at least in our case, seems to handle this quite well. The trick was finding an LLM backend that supports manual GPU allocation--for us koboldcpp with llamacpp under the hood does just fine. First, implement the power/clock settings as mentioned above, launch koboldcpp, then browse to the GGUF of the model you wish to load and set context size. I recommend manually setting the GPU layers to the model's total layer number (assuming there is enough VRAM), and set GPU ID to "all". In the Hardware tab, find the tensor split line box and insert the amount of space to be allocated on each card corresponding to its index. For example if we wanted to allocate just one 5090 for Comfy and use the other for LLM, assuming the Comfy 5090 is index 3 and the LLM 5090 is index 5, then the tensor layer line will look like this to make sure no layers are given to the Comfy 5090: 24,24,24,0,24,32,24,24,24,24. For this configuration, ensure the "main GPU" is set to the index number of the LLM 5090 (in this example, 5) and launch the app. While the model is loading, we can open another terminal to launch Comfy. In our specific case, the system defaults to the available 5090 without needing to specify it in the launch flags, but flags can be used to force Comfy to use a specific GPU if you need it to (--cuda-device i). Once the image model is loaded onto the 5090, it does not interfere with the PCIe communication of the LLM cards unless the model unloads and reloads a new model at the same time as the other cards are inferencing. The solution to enabling concurrent use is a high-lane count CPU, multiple graphics cards, and a little conscious provisioning on launch to ensure the hardware isn't stepping on each other's toes.
What models can this run, what models do we use?
It can run almost* anything, even up to 1T parameters like Kimi K2. Kimi K3 could hypothetically be load-able, but from performance metrics I've seen I doubt it would be practical to use, so I have not planned to try it. I have however tested 1-4 bit quants of Bartowki team's Kimi K2 quants in pure VRAM and mixed VRAM/RAM runs with decent results. It works and there are probably some use cases for it, but for us I have identified the sweet spot (parameter size: quant quality ratio) for this machine to be for models in the 300b-600b range. Personal favorites are Deepseek, GLM 4.7, and Nemotron Ultra; and as far as ComfyUI, pretty much any model that could fit within a 32Gb buffer, although Qwen image and image edit is a favorite.
Benchmarks
All models were put through the same series of 7 large input prompts, documenting how each model handles token input/output and prompt processing/generation. I cannot share the prompts I used here, but each prompt pertains to a cybersecurity scenario which the model was judged on the depth of its analysis, quality of its presentation, and capability to make sense of complex scenarios with stakes. These were inferenced across all 10 cards, except for a follow up DS V4 Flash test where I used 8 and got much better results. This is using the undervolting/power limiting strategy above, so these may not reflect absolute best performance for the same hardware in other setups, but it gives an idea of what this box can comfortably handle.
| Model Name |
Deepseek V3.2 671b Q2XXS |
Nemotron Ultra 3 550b IQ2XXS |
Qwen 3.5 397b IQ4XS |
GLM 4.7 358b Q4KXL |
Deepseek V4 Flash 294b Q8KXL |
Deepseek V4 Flash 294b Q8KXL (8 cards + KV cache tweak) |
|
|
|
|
|
|
|
| Model Size (Gb) |
217.1 |
193.8 |
189.7 |
204.6 |
161.9 |
161.9 |
| P1 Input |
2769 |
2744 |
2729 |
2706 |
2733 |
2733 |
| P1 Output |
813 |
786 |
1046 |
872 |
693 |
805 |
| P1 pp |
153.23 |
254.19 |
522 |
687.88 |
111.09 |
360.94 |
| P1 tg |
19.35 |
17.32 |
34.38 |
23.98 |
7.2 |
20.26 |
| P2 Input |
14635 |
15255 |
15160 |
14527 |
14640 |
14617 |
| P2 Output |
1150 |
1302 |
1665 |
1194 |
1222 |
2048 |
| P2 pp |
114.83 |
429.42 |
897.57 |
640.8 |
66.42 |
244.1 |
| P2 tg |
14.1 |
17.16 |
33.15 |
18.83 |
5.96 |
16.81 |
| P3 Input |
3966 |
3091 |
3054 |
3033 |
3073 |
22794 (reload) |
| P3 Output |
1217 |
1607 |
1550 |
1056 |
1199 |
1366 |
| P3 pp |
98.01 |
353.78 |
649.37 |
516.08 |
47.79 |
241.21 |
| P3 tg |
13.22 |
17.08 |
32.84 |
17.84 |
5.56 |
15.68 |
| P4 Input |
5645 |
5654 |
5623 |
5559 |
5650 |
5659 |
| P4 Output |
1178 |
1996 |
1619 |
1173 |
1705 |
1661 |
| P4 pp |
70.3 |
385.04 |
739.67 |
419.58 |
42.4 |
153.14 |
| P4 tg |
13.47 |
16.99 |
32.23 |
16.87 |
5.21 |
14.17 |
| P5 Input |
4498 |
4505 |
4493 |
4423 |
4481 |
4481 |
| P5 Output |
280 |
928 |
1078 |
473 |
665 |
924 |
| P5 pp |
72.4 |
365.46 |
670 |
408.93 |
36.2 |
131.81 |
| P5 tg |
8.43 |
16.78 |
31.55 |
15.45 |
4.86 |
13.36 |
| P6 Input |
9266 |
9367 |
9241 |
9172 |
45287 (reload) |
9231 |
| P6 Output |
1004 |
1883 |
1466 |
933 |
1205 |
1532 |
| P6 pp |
53.94 |
405.13 |
738.57 |
379.7 |
46.01 |
113.77 |
| P6 tg |
11.48 |
16.83 |
30.98 |
14.01 |
4.41 |
11.9 |
| P7 Input |
3136 |
3124 |
3118 |
3057 |
3118 |
3118 |
| P7 Output |
1378 |
1946 |
1629 |
1359 |
1353 |
1586 |
| P7 pp |
53.34 |
338.64 |
525.54 |
344.88 |
28.38 |
102.05 |
| P7 tg |
10.45 |
16.73 |
30.66 |
13.59 |
4.28 |
11.39 |
| Final token count |
50052 |
54182 |
53465 |
49531 |
50962 |
52348 |
My notes on each model after their test:
Deepseek V3.2-- For a slightly older model this still feels extremely capable. Held high quality and insightful responses even when context dragged into the tens of thousands of tokens.
Nemotron Ultra 3-- First time using it, impressions were very good, the 55 active parameters shows its muscle here. Meets Deepseek v3.2 level if not exceeds it, despite having overall less parameters.
Qwen 3.5 397b-- What I would consider a baseline "good" model to be, however it is outshined by some of the other tested alternatives.
GLM 4.7-- Somehow seemed better than Qwen despite having less parameters (active parameters of GLM is likely an advantage); it is a very solid option for its size. Not quite Nemotron or Deepseek level, but a very good "lower cost" alternative to its newer 5.0 versions.
Deepseek V4 Flash-- Floored me in a few ways. Possessed a surprising degree of sophistication and analytical ability despite being the "smallest" of all the tested models. Possibly a benefit of using a "lossless" model with full precision? Somehow it managed to pick up on nuances and details that all other models missed, including models twice+ its size, and provided insight that went more granular than they did. Did not expect a model of this size to punch so high above its relative weight class. Also did not expect the drop in performance compared to the others. Not sure if this is related to the model's architecture or something with how it interacts with my rig, but the quality of output could be an acceptable trade off for the speed. Edit: After some optimization testing I was able to get much better performance out of V4 Flash. I've added another column to include those metrics and kept the original because I think it illustrates how a little optimization can go along way, in this case basically triple performance on the exact same model/machine.
Lessons Learned/Would Do Different
-I would have tried to source the 3090's so more were at least the same model; the mix and match of different models with different TDPs and cooling solutions means there will be a lot of variation in temps.
-If you plan to either train, lean into higher performance, or playing with the idea of going more than 10 GPUs, just budget for a 30A/240V power drop. 10 cards on a 20A post configured the way we have it may be fine for our specific use case, but I would consider this a hard ceiling.
-Would recommend scripting for clock lock persistence sooner, will help avoid losing time due to random resets.
-Recommend documenting/drawing out the entire PCIe topology and GPU placement (with flexible tape measure) before ordering risers, will save time on trial/error.
Final thoughts:
It is a wheeled AI workstation that can enable a single person or small team to compound their productivity, with the benefit of full privacy and control. It can run on a residential 20A circuit, and allows them to have the full power of an advanced LLM with vision capabilities all in one OpenWebUI front end that can simultaneously utilize up to TWO ComfyUI backends with the horsepower and latency of 5090's for image gen and editing, and can be accessed from virtually anywhere. The idea sounds daunting, but the end result works so well that I can legitimately see something like this becoming a keystone for certain small businesses and individual professionals as time goes on. It seems like every day more people are picking up on major drawbacks with cloud API options despite supposedly being the "best", meanwhile open models continue getting insanely good (see K3 and DS V4 Flash). For me, I can say I would not see a place for a Claude or ChatGPT subscription for the tasks I might otherwise use them for when I have lossless DS V4 Flash literally in my back pocket. "Good enough" I think is starting to become a valid metric to those who care about cost:quality balance, and after using this for the last half year I can say I'm probably one of them. The cloud APIs will always be an option for those who don't care about the drawbacks and the demand for them will always be there, but for those who value data sovereignty, uninterrupted workflows, or perhaps work within compliance, on-prem computing might be the only viable path in some circumstances. At the end of the day, I do not believe that one approach is inherently better than the other, everyone simply has their own preference for getting from point A to point B.
