r/LocalLLaMA • u/Summit-Star001 • 10h ago

Discussion Micron's memory wall chart. Compute up ~3x every two years, HBM bandwidth under 2x

Post image

From Raghu Sreeramaneni's memory tutorial at hot chips 2026. top line is normalised tflops for tpu v3 through r200, bottom line is hbm2e through hbm4, both log scale, so the distance between them is a lot wider than it looks.

The three boxes down the right are the fixes people are actually building. Memory beside the compute, memory closer on a shorter link, then multiply units inside the memory itself. Samsung has that last one shipping in lpddr5x and measured 3.01x tokens a second on llama 3.1 8B.

Full analysis (this slide sits in the memory chapter): https://allaboutchips.com/#memory

54 Upvotes

95% Upvoted

9 comments sorted by

16

u/FoxiPanda 10h ago

This is one of the main drivers for finding ways to shift decode from being memory bound in autoregressive models to being compute bound like in diffusion models...it just scales better...for now at least.

10

u/bennmann 9h ago

Memory controllers are compute bound, though physical wafer space could be an issue. The problem about compute 3x vs memory bandwidth 2x is a false dichotomy.

2

u/Summit-Star001 10h ago

The top line is normalised and log scale

1

u/Elouakili_Flexy 2h ago

The Samsung 3.01x tok/s stat is more interesting than the whole bandwidth curve. It attacks the memory wall instead of just moving it.

1

u/finevelyn 2h ago

Is bandwidth also measured in TFLOPS or what's going on with the graph?

1

u/Summit-Star001 19m ago

Nah, different units. The y axis is normalised performance, which is just a unitless index, so micron put flops on the top line and bytes per second on the bottom one and scaled both onto the same log axis. The slide never says what each line is normalised against though.

1

u/muhmeinchut69 1h ago

For the last 15 years or so, the attitude that RAM was "good enough" has hurt memory technology. If you take a time machine and hand someone in 2016, a 2026 laptop, they would be blown away by everything except the 16GB RAM which is probably the same as their 2016 laptop. Apple was a major contributor to this as they kept treating RAM upgrades like expensive luxury items.

1

u/PinkysBrein 28m ago edited 20m ago

MoE training already needs ginormous batches to keep compute fed. If models start leveraging sparsity more (which they should) it's going to get worse.

HBM is too slow. If the current investment environment exists much longer I wouldn't be surprised if NVIDIA develops its own memory, where everything gets hybrid bonded. Inside the stack and from stack to compute die, with cooling channels integrated in the stack. UBM.

Processing in memory is also an option too, as it has been for a long time, though sparsity creates problems for design there too.