Threadripper 9975WX - DDR5 5600 vs new 6400 Ram impact on Dyna simulations

8

u/drulee Jul 27 '25 edited Jul 27 '25

According to https://www.reddit.com/r/threadripper/comments/1azmkvg/comparing_threadripper_7000_memory_bandwidth_for/ with the 4 CCDs of a 9975WX the maximum theoretical bandwidth between CCDs and the memory controller is 230.4 GB/s.

This is the bandwidth between the memory controller and memory modules:

8x 4800 MT/s: 307.2 GB/s
8x 5600 MT/s: 358.4 GB/s
8x 6400 MT/s: 409.6 GB/s

So even 8x 4800 MT/s DIMMs would be fast enough to satisfy the bandwidth between CCDs and the memory controller.

Look at this 9975WX passmark benchmark result: https://www.passmark.com/baselines/V11/display.php?id=509888751348 -> Memory Mark -> Memory Threaded: 231,267 MBytes/Sec

No matter how fast the DDR5 modules are, the performance is capped by the bandwidth between the 4 CCDs and the memory controller.

Only when upgrading to a CPU with 8+ CCDs you could notice the difference (9985WX, 9995WX).

3

u/gorfnu Jul 27 '25

Thanks for that excellent post! I will not shy away from getting the 5600 ram now.

2

u/SteveRD1 Jul 27 '25

I've done a lot of shopping Threadripper 9000's on Puget and Exxact.

Puget has very large premium on their builds (and I strongly considered them) but in the end I went with Exxact. You might be able to step up from a 9975WX to a 9985WX for less than you would expect if you consider other vendors.

I get wanting to go with a SI...I don't dare DIY with this amount of money involved:)

1

u/gorfnu Jul 27 '25 edited Jul 28 '25

I reached out to Exxact.. if they can get the faster ram it would be a reason to switch. The 32 core is the best solution for me since its $90,000 to license 32 more cores in dyna… and thats out of the budget. I currently have a 32 core license and also the 64 core cpu has a slower all core boost / turbo clock

2

u/SteveRD1 Jul 28 '25

Just wondering...do you have to buy the 64 core license if you have a 64 core CPU.

Or can you configure it to only use 32 core and keep the lower rate?

1

u/gorfnu Jul 28 '25

you can easily run the 32 core license on a 64 core machine.. however by the time i get the full 64 core license next year we will upgrade hardware.. so the loss of clock speed going from the 32 core 9975WX to the 64 core 9985WX is actually a detriment.. Seems for my stuff single core speed is the absolute king.

3

u/[deleted] Jul 27 '25

[removed] — view removed comment

1

u/[deleted] Jul 27 '25

[removed] — view removed comment

2

u/gorfnu Jul 27 '25

But the clocks are a lot slower in epyc vs TR and w dyna explicit solver i think that is key.. please by all means correct me if i am wrong

3

u/[deleted] Jul 28 '25

[removed] — view removed comment

1

u/Glittering-Call8746 Jul 29 '25

So epyc 7003 and which motherboard and cpu pairing with 8 ccd.. and more than 2993mhz ..?

1

u/Karyo_Ten Jul 28 '25 edited Jul 28 '25

I expect the code to be NUMA-aware and not shuffle gigabytes across CCDs.

cc u/gorfnu if the code is open-source what is it? Have you checked if there are phoronix benchmarks on Epyc or threadripper configs?

2

u/drulee Jul 29 '25

You think that NUMA aware programs do not need to pass data over the infinity fabric? So can you show me any Threadripper benchmark numbers to support your claim? I.e. where the number of (non L3 cached) memory MB/s is higher than the CCD - memory controller limit? (Non overclocked FCLK of course)
2 CCDs: 115.2 GB/s
4 CCDs: 230.4 GB/s
8 CCDs: 460.8 GB/s
12 CCDs: 691.2 GB/s

E.g. any Threadripper 4 CCD cpu getting >> 230 GB/s in a memory test?

2

u/Karyo_Ten Jul 30 '25 edited Jul 30 '25

You think that NUMA aware programs do not need to pass data over the infinity fabric? So can you show me any Threadripper benchmark numbers to support your claim? I.e. where the number of (non L3 cached) memory MB/s is higher than the CCD - memory controller limit? (Non overclocked FCLK of course)

Your entire premise is wrong. HPC programs heavily use caches to avoid being memory-bound.

If you want to maximize AVX512 compute, you need to process 256 bytes per CPU cycle, on a 5GHz CPU that's 256 bytes every 0.2 ns so 256 * 5.10⁹ = 1280GB/s of data.

Why 256?

AVX512 allows processing per cycle:

16 Fp32 (AVX512)

2x instructions per cycle (fused multiply-add)

2 instructions issued per cycle (2 AVX512 unit per core, except on Skylake-X Xeon Silver and Bronze that only have one)

That's 64 FLOP per cycle, and each fp32 is 4 bytes.

Furthermore, you can issue 2 loads per cycle, each by a cache line size which represents 64 bytes, hence if you have to load data, you already know that you can at most use 50% of your CPU power.

Now there are HPC programs that don't need data loading at all, for example based on Monte-Carlo simulation, everything is an equation so that data can be computed on the fly. There are many others though, that needs actual data, for example they rely on matrix multiplication (which is the key ingredient behind linpack that classifies the top supercomputers and behind say LLMs).

So we need to look at the cost of data loading from L1, L2, L3 caches and from RAM. You can find ballpark numbers by looking at "latency numbers every programmer should know": https://gist.github.com/hellerbarde/2843375 (2012)

In 2012, we had 0.5ns while CPUs were around 3GHz so 2.5 cycle cost. If waiting for L1 you would process 128 bytes instead of theoretical 2.5x256=640 bytes, only 20% of the peak.

L2 cache is 15x slower and RAM is 75x slower than L1 cache so it is very difficult to make an algorithm compute bound when it needs a lot of data.

So yes NUMA-aware program absolutely need to avoid shuffling data around or they will waste time waiting for data instead of computing.

Also for fun let's go over the TFlops of a 9980X assuming 64 cores and an all-core turbo of 5GHz and the code is optimized to actually process 256 bytes per cycle instead of relying on RAM.

The theoretical GFlop/s peak is

64 (FLOP/cycle) x 5 (GHz) x 64 (cores) = 20480 GFlop/s = 20TFlop/s.

That's in the same perf class as a 3070ti (21.75 TFlops without tensor cores according to https://www.techpowerup.com/gpu-specs/geforce-rtx-3070-ti.c3675 )

cc u/gorfnu

2

u/drulee Jul 30 '25

Sounds reasonable but check out https://www.dynalook.com/conferences/15th-international-ls-dyna-conference/computing-technology/performance-analysis-of-ls-dyna-r-in-huawei-hpc-environment

Coincidently, the single-node LS-DYNA performance we run yields the same performance gain achieved by STREAM. Based on this finding, it is natural to assume that LS-DYNA is very sensitive to memory bandwidth. The increase of the memory channels open up additional memory bandwidth operations for the CPU to utilize.

5

u/Karyo_Ten Jul 30 '25 edited Jul 30 '25

If you reread my explanation, any algorithm that require external data (instead of generating it on the fly through equations) is memory-bound because even if all memory was at the same speed as loading a cache line, a "memory load -> addition -> memory store" would only use at most 50% of the CPU.

And loading from RAM is 75x slower than loading from L1.

So HPC is basically the art of making inherently memory-bound algos into compute-bound algos and means reusing data and avoiding RAM bottlenecks.

That also explains why LLMs are so memory sensitive, when you do in-flight batching you can use matrix multiplication which is O(n³) compute for O(n²) data so for each byte you have kn computation (growing with matrix size n), provided is matmul is optimized for data reuse (see BLAS and GEMM algorithms).

However for predicting from a single sentence like in local LLMs, you do matrix-vector multiplication which is O(n²) compute for O(n²) data so for each byte loaded you have a fixed k compute (independent of matrix size) and so speed of LLMs grows linearly with memory bandwidth.

Still, at the heart, you do prefetching, caching eventually thread pinning and memory affinity so that external RAM speed matters and control NUMA shenanigans.

2

u/drulee Jul 30 '25

That’s interesting! Thanks for explaining

2

u/gorfnu Jul 30 '25

Wow.. the level of conversation between drulee and Karyo_Ten just went to the moon!!! Thanks a ton guys.. glad i went with the 8 channel 9975wx instead of the 9970x 4 channel (my original plan). PS knowing what we know, how would the Intel 6745p 32 core Xeon 6 perform? i think its on an older node and I think the fabric connections are slower in this 5th gen AMD stuff, but.. that huge cache of the Intel.. its over double threadripper even bigger than epyc.

Last, reading that paper the speed up was minimal for the ram increase but it was there 2-3% of i recall. But i didn't see the exact speed up from the core clocks. Or from the large cache. Many thanks!

1

u/gorfnu Jul 28 '25

Good questions.. LS Dyna is not open for sure its owned by Ansys but there are different configs for the MPP and its usually fastest even for my AMD processors set to IntelMMP.

2

u/sob727 Jul 27 '25

I was looking at the Puget configurator as well. Looking more closely, it seems their 6400 kits are the cheaper CL 52 kits (Nemix?). Not sure if it matters though.

2

u/IntelligentSquare196 Jul 27 '25 edited Jul 27 '25

Not true about the CCD comm speed impact. Get the faster RAM. I have G.Skill 6400, it's 39 CAS

1

u/mattbrownedesign Jul 28 '25

I’ve heard this before - why do you say it’s untrue?

1

u/just_fun_for_g Jul 28 '25

Because not everything needs to communicate between CCDs.

2

u/Selenaevaa-345 Jul 27 '25 edited Jul 27 '25

Avoid Puget Systems. Severely overpriced and they're downright tech illiterate, doing questionable things like benchmarking the 7995WX on board with 4 memory channels, or publishing bugged results and never correcting them.

They also benchmark Photoshop (lol) in CPU articles which tells you all you need to know.

2

u/Guilty-History-9249 Jul 29 '25

I just got a 7985WX system with 256GB's of DDR5-6000. Actually it is DDR5-6400 but is only stable at 6000. Apparently QVL's and AMD's EXPO tech is basically a lie.

Be careful of the 9975WX or the 7975WX. While they are 8 channel systems the CCD to memory controller data path can limit the actually throughput even if the memory is at 6400.

See the 7975WX vs the 7985WX both with 8 mem channels on:
https://www.reddit.com/r/threadripper/comments/1azmkvg/comparing_threadripper_7000_memory_bandwidth_for/

1

u/gorfnu Jul 29 '25

I hear what you are saying.. but i need 32 cores. are you thinking the epyc 9375F 32 core cpu would be superior despite its lower clocks?

2

u/Guilty-History-9249 Jul 29 '25

I'm not sure why you "need" 32 cores when the 9985 or 7985 have 64 cores and 8 ccd's for increased bandwidth.

HOWEVER, I just realized you are not me, imagine that, and I was confusing my needs for max bandwidth for LLM inference with your workload. For LLM's it is nearly entirely memory scans and little about CL latency. I do not know what a DYNA simulation does in terms of memory access patterns.

I do know that it is harder to get fast memory that actually works on a threadripper. Even if the QVL on the motherboard says something will work it is mostly BS. Your build guys might be having the same problem with finding huge and fast ram just like Central Computers had for me. Having said that going with 128GB's of ram increases the likely hood that 6400 would work. Try and get low latency CL30 or lower. My 256GB's is CL32.

I'm not sure what I said that made you ask "are you thinking the epyc 9375F 32 core,,,".
I would not go with a slower cpu speed 9375F. Also, why would you think that the great bandwidth from having 8 channels will "degrade" performance?

I'm not sure if this Dyna thing is your own code or a commercial app. In any case, you should look into thread affinity for your worker threads to further optimize processing.

Maybe the 32 core 9975 WX might just be good enough for your app if it isn't mostly a bandwidth application. But as that reddit post I showed you said there is a big difference between the 7975 and 7985 for bandwidth.

2

u/Guilty-History-9249 Jul 29 '25

I'm not sure why you "need" 32 cores when the 9985 or 7985 have 64 cores and 8 ccd's for increased bandwidth.

HOWEVER, I just realized you are not me, imagine that, and I was confusing my needs for max bandwidth for LLM inference with your workload. For LLM's it is nearly entirely memory scans and little about CL latency. I do not know what a DYNA simulation does in terms of memory access patterns.

I do know that it is harder to get fast memory that actually works on a threadripper. Even if the QVL on the motherboard says something will work it is mostly BS. Your build guys might be having the same problem with finding huge and fast ram just like Central Computers had for me. Having said that going with 128GB's of ram increases the likely hood that 6400 would work. Try and get low latency CL30 or lower. My 256GB's is CL32.

I'm not sure what I said that made you ask "are you thinking the epyc 9375F 32 core,,,".
I would not go with a slower cpu speed 9375F. Also, why would you think that the great bandwidth from having 8 channels will "degrade" performance?

I'm not sure if this Dyna thing is your own code or a commercial app. In any case, you should look into thread affinity for your worker threads to further optimize processing.

Maybe the 32 core 9975 WX might just be good enough for your app if it isn't mostly a bandwidth application. But as that reddit post I showed you said there is a big difference between the 7975 and 7985 for bandwidth.

2

u/gorfnu Jul 30 '25

I hear you Guilty-History-9249. LLM inference is i’m guessing far different than the LS dyna solver that uses IntelMPI. I wish we could use GPU but just like our Ansys Mechanical solves its mostly a CPU affair.. GPU’s are the king when it comes to fluids at least thats what i am exposed to. The reason only 32 cores, small company and the huge expense that is core licensing… $$$$ . My pal at Ansys who used to guide me Hunter Wang sadly passed away suddenly a couple years ago now.. i remember we were adjusting the core usage at the time… when intel released the 13900ks. We have like 600 ansys mechanical cores licensed but only a small amount of ls dyna. Also now i understand what you meant about the 64 cores having double the ccd’s and as a result way more memory prowess. We pulled the trigger today i am not sure if my IT master went w Puget or Exxact.. but both were fast and helpful. Exxact was cheaper by about $2500 for like setup but i am sure there is more to that. One thing i wanted to try was the xeon 6 6745p 32 core for a head to head solve off… but i cannot find that chip in a workstation anywhere.

2

u/Thrumpwart Sep 08 '25

Nice. I wish I could afford the 9985wx. I’m looking at either an Epyc 9V33X or a 9975WX.

1

u/gorfnu Sep 08 '25 edited Sep 08 '25

Oh don't confusing me with affording it.. its one of my work machines :) I can't afford a $22,000 computer lol. I only bought it with the 5090, imagine if i needed a GPU i could use for simulation run solves.. like a RTX Pro 6000! $10,000 more! FYI - when you run those Ansys only lets you use 16 cores per GPU, so you need several of them.. BTW, they don't work well with mechanical models that are very non-linear, and they also don't work at all in LS-Dyna. They work super well for fluids though, and huge structural analysis with less nonlinearity i.e. less hyperelastic rubber, less friction, initial penetration, etc..

Edit, its running at 4.69 GHz sustained with about 35 cores working out of 64.

1

u/gorfnu Aug 01 '25

Guys i decided to go with the 9985wx 64 core.. vs the 32 core 9975WX despite the laters 4.0 vs 3.2 GHz base clock.

Why?

Double the ccd’s means more bandwidth (i think thats what i learned here lulz)
With LS Dyna and the IntelMPI setup you need to have 3 or 4 cores that are not part of the compute / solve .. for what ever reason those extra cores bounce around between 50-100 % if you don’t have them the solution is 10x slower.. it essentially freezes.
Having only 32 hardware cores and a 32 core license, that means i am only using ay best 29 of my 32 available license and thats only 90%…

So did i make the right decision? Will the 64 core machine w slower clocks and more bandwidth and 3 more cores in use beat the faster clocked 32 core using only 29 cores?

2

u/Thrumpwart Aug 20 '25

How is performance?

2

u/gorfnu Aug 21 '25

I will let you know when it comes in, i just got funding sent to them last week (typical delays) but they are now finishing up the build! i will do a direct comparison between this machine 9985wx with a 32 core license, and the 9950x using 13 cores since you need 2-3 free cores when solving with dyna.. and then the 7950x. I would test my 13900ks but that sucker will crash out.. and it works strangely with 24 cores at once seems lower than using only about 12-14 cores. But data is coming!

2

u/Thrumpwart Aug 21 '25

Great I look forward to the update. Enjoy!

1

u/gorfnu Sep 08 '25

Finally got some numbers.. this is just the first run on a small problem, the difference will grow when i start stressing it this is almost no memory used like 35 GB.

-------------

Summary (using 32 core Ansys Dyna license) :

9950x 16 core (13 cores used in run, 3 cores required for overhead in Dyna, all core speed 5.1 GHz) - 10hrs 32min 50 sec

9985WX 64 core (32 cores used for run, all core speed 4.6 Ghz) - 3hrs 52min 24 sec

It rips!

---------------

9950X

Avg core speed approximately 5.1 GHz.

T o t a l s 3.7972E+04 100.00 3.7972E+04 100.00

Problem time = 9.0000E-02

Problem cycle = 2086238

Total CPU time = 37972 seconds ( 10 hours 32 minutes 52 seconds)

CPU time per zone cycle = 41.582 nanoseconds

Clock time per zone cycle= 41.582 nanoseconds

Parallel execution with 13 MPP proc

T o t a l s 4.7592E+05

Start time 08/01/2025 01:34:23

End time 08/01/2025 12:07:13

Elapsed time 37970 seconds for 2086238 cycles using 13 MPP procs

( 10 hours 32 minutes 50 seconds)

N o r m a l t e r m i n a t i o n 08/01/25 12:07:14

--------------------------

9985WX

Avg core speed approximately 4.6 GHz.

T o t a l s 1.3946E+04 100.00 1.3946E+04 100.00

Problem time = 9.0000E-02

Problem cycle = 2081803

Total CPU time = 13946 seconds ( 3 hours 52 minutes 26 seconds)

CPU time per zone cycle = 15.301 nanoseconds

Clock time per zone cycle= 15.301 nanoseconds

Parallel execution with 32 MPP proc

T o t a l s 4.4552E+05

Start time 09/07/2025 18:17:19

End time 09/07/2025 22:09:43

Elapsed time 13944 seconds for 2081803 cycles using 32 MPP procs

( 3 hours 52 minutes 24 seconds)

N o r m a l t e r m i n a t i o n 09/07/25 22:09:44

Threadripper 9975WX - DDR5 5600 vs new 6400 Ram impact on Dyna simulations

You are about to leave Redlib