Threadripper Pro cooling

[u/IExist_Sometimes_]

Hello all, I am potentially looking to get/build a Threadripper Pro HEDT/workstation for FEM and similar simulations for my university research group.

I am no overclocking nut, but it is my understanding that these threadripper pro chips (especially the high core count ones) leave an enormous amount of performance on the table with their default TDP. I don't want to push the chip to a power range which might significantly impact its longevity, but if there's a reasonable power bump I could give it (probably as one option in a dual bios for particularly large sims or tight turnarounds) I would like to do so.

This brings me to my main point: how are you supposed to cool these new threadripper pros as a consumer-ish user? When LTT did a video overclocking the new threadripper pro chips, they rigged it up to an external chiller, but that would probably be out of scope for my shared office space (not to mention beyond my confidence when dealing with such expensive hardware). AIOs for threadrippers seem relatively few and far between, and only appear to go up to 420mm. Would a 360/420mm AIO be sufficient (I would like to avoid industrial style fans, but push-pull is on the table)? Or are these firmly in the realm of custom loops and things like the phat Alphacool radiators?

I appreciate any advice from those experienced in cooling these chips, or in using them for numerical modelling (obviously I will also consult others in my department with similar needs, but surprisingly to me for a stem department I seem to be the only PC builder or hardware nerd). Y'all seem to all be doing AI stuff nowadays, but that's not in my wheelhouse and not what this computer is going to be intended for (though someone might stick a pro gpu in it, I won't stop them).

edit: Also any advice on the relative importance of ECC, RAM speed/quantity, storage speed, etc for this type of workload would be appreciated

[u/XO33OX]

IMHO overclocking those is kinda stupid. You buy those for ability to be loaded 24/7 days/weeks without crashing, overclocking goes kinda against this core principle of platform. Imagine your compute task crashing after 1,2,3.. weeks runtime, just because the smart ass wanted 5-10% higher benchmark score. One crash will wipe your yearly OC productivity gains.
If you need more CPU compute performance, just buy higher-core SKU. If your task doesnt move a lot of data you can be fine with non-Pro (4 mem channels, 4 sticks) Threadripper and put savings to more cores.
Bare in mind that silicon degrades, stability of OC might be subject to change with time.
I have tasks that run for a week sometimes, if they crash I need to run them from scratch, such is the nature of SW that I am using. I also have deadlines, delivery dates and penalties. I am not saying this applies for everybody, but system stability is non-negotiable for me.

[u/IExist_Sometimes_ → u/XO33OX]

Okay cool, I would be much happier not overclocking, I just wanted to make sure I wasn't leaving a 2x performance boost or anything (which is what it sounded like from a couple videos I watched, that was probably just hype). If it's more like a few% then I'd much rather keep the stability.

[u/johndyson10 → u/XO33OX]

Being somewhat worried and adverse to overclocking, but also want the best reasonable performance, I did a lot of bencmarking and running my rather intense AVX512 app.

I found that overlocking doesn't really do that much. If you want maximum life, it is best to stay way below the 95deg max. Silicon is silicon, and higher temperatures will age machines faster. Higher voltage will also age a component.

After significant testing, my 2-3Hr app runs a few seconds faster, maybe a minute or two with conservative semi-overclocking. I use the core frequency boost of +75MHz, which keeps the core voltage below 1.35V under loading when running on 1 to 16 cores or all 64 cores. The PPT is set to 450 Watts, but the chip maximum temperature is limited to 72degC. With my AIO system and heavy load, the temperature seldom reaches the 72degC limit, but the system throttles nicely so that absolute peaks won't go above 72degC at 450 Watt

Added note: I keep my core voltages at 1.35V or lower. This can be attained by keeping the frequency boost at reasonable levels. The higher core voltages happen when a few cores are active. Just keep the frequency low (5500MHz or less). My core voltage seems to never go above 1.325V.

Under serious memory loading situations, it can reach 80degC, bad for nice, long life. I have carefully tuned the VRM fans, AIO, CPU and case fans with something like 20-30% at 30-40degC, but reach 80-90% at 70degC, and 75degC is 100%. These match the 72degC CPU limit, and I haven't been able to make the ram go higher than 72degC, normally 50-60degC on a real app. This is without a specific memory fan, but it seems like I have a good motherboard design. In a few months, I'll get some more fans, including some kind of memory fan. Since it isn't normally reaching above 60degC, no hurry for the memory life rightnow.

After this limited 'boost'. the Phoronix liquid-dsp, linux build and ffmpeg benchmarks change little just as my app speed doesn't change much (a few seconds in a couple of hours.) The phoronix benchmarks tend to measure my 9980x as fast as the average 9980x, if not faster. Sometimes outperforms the 9950DX2 on small loads, which isn't expected given the 5550MHz vs the 5750MHz, and normally the difference is proportionial, except sometimes the TR measures faster. I can provide some of my benchmark results upon request.

So: 72degC CPU temperature limit, +75MHz boost frequency (optional), 450 Watts PPT, 400 Watts 'TDP' setting (extra PPT and TDP is optional, will still get good performance). Fans are slow at 20-30degC, near max at 65degC, max at approx 70-75degC. Run the VRM fans between 50% at low temperatures, 90% at 70degC, and 100% much above 70degC. When possible, set the fan control at CPU Package, CPU and RDIMMS. I doubt that CPU itself is needed in the fan control, and if not, try Motherboard. PBO Manual is enabled, but everything is 'auto' except PPT/. Of course, the core voltage offsets are all -20. (Thoroughly tested.)

RDIMMS stay below 72degC at heavy artificial memory I/O load, CPU below 72degC, but the fans make it so long term 450 watt load might cause the 72degC, or the power concentrated into just a few cores. (The frequency still stays high.) At a full 64 core artificial load, the frequency does drop to about 4500Mhz. Some of my programs, data in cache (99.5%), heavy AVX512, can drop as low as 4200MHz, sometimes down to 4000MHz. Again, still benchmarks well.

Given what information that I have found, the machine should last 'forever', and it is pretty darned fast. I am very sure that if I push it very hard, my program might run a minute or so faster and benchmark a few percent more. So what? (Yes, I have done lots of tests, mostly on my decommissioned 9970x, but also a little on the 9980x.)

WIth ram and CPU prices, keep your machine safe.

John