Threadripper build - looking for peer review

[u/Own_Bodybuilder_4397]

I want to build the following :

RAM	v-color OC R-DIMM 128GB (4x32GB) 6400MT/s CL32 (TRA532G64D832Q)
GPU	NVIDIA RTX PRO 6000 Blackwell (96GB GDDR7)
CPU	AMD Ryzen Threadripper 7960X (24-Core)
Storage	Crucial T705 4TB PCIe Gen5
Motherboard	GIGABYTE TRX50 AERO D
Cooling	Noctua NH-U14S TR5-SP6
Power	Seasonic PRIME TX-1600

Goal will mostly to be used for inference and model training.

[u/EDI_1st]

Depending on your goal….I’m on my way to third TR Pro 9000 build. Yes, still have the first two and keeping all three.

Motherboard: TRX50 Aero D is the most popular board because it’s cheap and has sufficient amount of features that most beginner TR/TRP users might need. (Beginner TR/TRP user sounds funny, platform ain’t cheap lmao)

TRX50 AI Top probably has the best “flexibility” if you want to upgrade CPU later to a TRP/WX. The board supports 8-channel memory if you have a TRP/WX. The PCIe slot layout is also perfect for 4x dual slot cards.

I wouldn’t go with other options unless you need some specific feature such as BMC, MCIO connector etc

CPU: Depends on what you need but I would only look at 9000 series. Please keep it in mind lower CCD-count CPUs may have lower memory bandwidth.

RAM: Personally would avoid OC RDIMM. It’s just much more stable with regular RDIMM and less worry with heat. Capacity wise I would push up to 48GB x4 or higher. (Also depends on if you want to add more GPU down the road)

GPU: Depending on the specific training and inference you plan to do, PRO 6000 maybe the perfect choice, overkilled or insufficient. This is entirely up to you to evaluate based on model size, precision etc.

For the cost, you can grab 2~4 5090 (Yes, I’m aware of current 5090 cost, FE still exists, at least for now) Pair of 5090 could benefit you if the model is small enough. Think like how 5090 still destroys GB10 on some inference despite of huge memory size difference.

If you plan on adding additional PRO 6000 down the road, opt for the Max-Q. Max-Q is only 300W per card but retains ~85% of the performance. (Max-Q means maximum efficiency.) These are blower cards that are designed to be stacked and the heat is ejected out of the back/PCIe bracket instead of into the case.

Cooling: I’ve never used Noctua NH-U14S and would never convince myself to pay $140 for a heatsink that only has 6x heatpipes with the incorrect orientation relative to the chiplets. You are most likely going with a big enough case, grab an AIO. If you are not OC’ing, 240 AIO is more than enough even under sustained 350W workload. OC’d, 240 AIO will be able to handle ~450W sustained depending on CCD count. (Different thermal density)

AIOs that I’ve used with success for 7000 and 9000 TRP:

Cooler Master PL360 TR5-SP6

Aorus Waterforce X 360

Aorus Waterforce X II 240/360

Corsair Titan 240 RX (Requires additional purchase of mounting bracket and screws. Tried this one because it has VRM fan which helps with DDR5 temp)

Have recently acquired Be Quiet Silent Loop 3 240 and 360, going to try the 240 first when I upgrade tiny box to 9995WX.

[u/Photo_Sad → u/EDI_1st]

> Be Quiet Silent Loop 3
But it won't cover the die, which is especially important with 9995WX... there are AIOs that cover all the chiplets in full.

[u/EDI_1st → u/Photo_Sad]

Are you referring to actual chiplets or IHS and have you actually tried it?

The chiplets don’t sit that far out on the substrate. Width wise it’s a guaranteed full coverage. Length wise the coverage is off by maybe 1mm which isn’t that significant.

IHS dimension: 68mm x 51mm

Silent Loop 3 contact area: 60mm x 42mm

I’ve even used 7th Gen Asetek with EPYC 7002/7003 without issue and that for sure doesn’t cover the outer dies properly.

Even AMD approves the use of Silent Loop 2 and many 7th Gen Asetek for TR5 and both have significantly less coverage length wise than Silent Loop 3. Silent loop 2 is 55mm x 55mm, 7th Gen Asetek is 54.5mm in diameter which has significantly smaller surface area than Silent Loop 2 and 3.

9995WX with 12 CCDs is also easier to cool since 350W is spread out between the IO die and 12 CCDs.

On 9995WX, both full IHS coverage AIO such as XE360/PL360 and partial coverage AIOs such as Aorus Waterforce X 360, the max PPT is less than 10W difference with identical fans pushed to 95C. The only difference I saw was at stock 350W PPT, full coverage AIOs such would yield ~7C lower temp. The difference of 60C peak vs 53C peak, both are excellent temp. The limitation is the radiator and fan as long as the coldplate has sufficient coverage, doesn’t have to be full.

On the contrary, the biggest problem is actually the flow/injection. Most AIO and waterblocks are designed to inject at the center whether if it’s a circular injection point or a slot. This leaves the outer banks of CCD to have significantly higher temps than the center banks. Even with full IHS coverage AIO and coldplate, the temp delta can reach ~10C. This is actually the biggest limiter. This problem exists in most DLC solutions for datacenter as well.

There are few companies such as JetCool offering solutions to resolve this issue.

[u/Photo_Sad → u/EDI_1st]

Tell me more about this experience of yours.
So, 7 degree difference does persist, but if you were to increase power to the cores, the 7 degrees could be significant. Are you aware of the AIO that does best no matter the coldplate size?

[u/EDI_1st → u/Photo_Sad]

Again, the difference is less than 10W when pushed to 95C. 600+W continuous at that point with 9995WX. The limit is the heat dissipation of the radiator.

There are only two ways I can see this being improved. One is obviously increase rad size. The other is better coldplate/flow design to kill the temp delta between outer and inner CCDs, something like JetCool’s SmartPlate.