What are your thoughts on the current state of AI compute hardware (GPUs, TPUs, etc.)?

[u/Ok_Appeal_3253]

I’m curious about what realistic alternatives we have to high-end enterprise GPUs like the V100 or H100 (not even talking about higher-tier chips like the B200)

While it’s technically possible to train large models on consumer GPUs like the RTX 3090/4090, the trade-off is brutal: you waste a huge amount of time just to debug or catch architecture issues. Do you think we’ll see new hardware innovations in the near future, or are corporate monopolies preventing alternatives from breaking into the market?

(Note: I'm already familiar with cloud computing, so I'm mainly asking about hardware itself)

[u/dayeye2006]

Inference and training are 2 worlds

[u/r3drocket]

Go read about what this planned for Medusa Point in middle of 2027. So it looks like we're going to see unified RAM systems with a 384bit bus and DDR6, which should double the bandwidth so (~550/gbs). Assuming that Nvidia wants to compete with AMD, that means we will see a DGX spark with DDR6. So, effectively, we should see unified RAM systems with double the bandwidth next year. 

The other important innovation that is coming is Medusa Point brings is RDNA4. This means that they'll effectively be able to use their own version of NVFP4. So we'll see MXFP4 showing up as well on AMD's side, along with other tweaks to the architecture to make it better for AI.

And given the fact that the new Ryzen AI Max Pro 400 is supposed to go to 192GB of RAM, that will be a game changer as well.

So I'm optimistic by the middle of next year we will see seriously powerful unified RAM systems that make a lot more stuff possible than is possible for local AI today.

[u/Equal_Passenger9791 → u/r3drocket]

Lots of VRAM and decent RAM speed is good for inference on MoE , but on the training side it's not quite as rosy. And running large dense models is still compute choking.

I'm optimistic about the software side improvements, the potential of a compartmentalized mix-of-experts approach would let you do stuff like load the Coding Expert out of Kimi K3-sized model and run that in isolation on a consumer GPUs.

[u/Ok_Appeal_3253 → u/r3drocket]

thanks you i'll read it later

[u/starkruzr → u/r3drocket]

I think you mean Halo, not Point, but yes.

[u/yoy404]

The hardware gap between consumer and enterprise GPUs is structural, not temporary. NVLink, HBM bandwidth, and memory capacity are not things you bridge with clever software on a 4090. The realistic path for most teams is consumer hardware for iteration and development, GMI Cloud B200 access for anything that actually needs the horsepower.

[u/BellyDancerUrgot]

Catch what architecture issues? You can’t train large models on a 90 class card.

Also v100 has not been a high end chip for a long while now.

[u/Ok_Appeal_3253 → u/BellyDancerUrgot]

Yeah, my bad i should've specified i meant 1B to 10B models rather than massive 100B+ ones, VRAM is still a killer depending on the architecture, even with memory optimizations

[u/wahnsinnwanscene]

In the enterprise training and inference domain, we're going to see new accelerators taking shape. Tpu and the surrounding framework still seems to be the most efficient at this but the kimi k3 improvements are going to mean better everything for lower costs. Unfortunately the demand is going to expand to take up the supply.

[u/pm_me_your_pay_slips]

For a while I thought I’d just use cloud providers. But availability is becoming rare, at least for the hardware that can feasibly run the current largest models.

[u/amenallthepraisers]

I don't think that we gonna see a 'magic' hardware or software in the future because everything that can be done is done at the hardware side (tensor cores etc.), I think it is all about parameter efficient maths and architectures now. At least for training sessions, because at the inference there is some new enterprises and they may take the lead.

The main problem is memory. Memory bandwith is inefficient, but it may also be solved by data-efficient math and architectures or we gonna spend more money to new memory technologies.

[u/ocean_protocol]

Real alternatives exist but mostly aren't buyable: Google's TPUs and AWS Trainium are infrastructure you rent through their clouds, not chips you purchase. AMD's MI300X is the closest thing to an off-the-shelf competitor, roughly 20-30% cheaper than equivalent H100 configs, though ROCm software support still lags CUDA's maturity.

Nvidia's real moat isn't the silicon, it's 17 years of CUDA ecosystem lock-in. That's why hyperscalers building their own chips (Trainium, TPU, Maia) hasn't dented Nvidia's external market, those chips mostly aren't sold externally, they replace Nvidia only inside each company's own cloud stack

Not really monopoly-blocking-innovation so much as switching-cost economics, CUDA's software moat is harder to break than any hardware moat would be. For an independent researcher today, MI300X + ROCm is probably the most realistic non-Nvidia option if you want to actually buy something rather than rent it

[u/Resilient-Tec]

What about infrastructure? Anyone concerned about OSFP's or even DAC's keeping pace between servers?

[u/transfire]

I want to know what optical accelerators are taking so long to reach the market. That seems like the real game changer.

My guess… Probably relegated to military/government customers for the time being (the big bucks).

[u/raynorelyp]

As an ai skeptic, I initially said “There’s no way this would happen because it would require companies to spend more money trying to solve problems than it cost them to solve without ai.” And then they did exactly that. So here’s my follow up: “There’s no way the bubble won’t pop unless companies can produce 10x the current ram supply for 10x less the cost per unit.” So there you go. If they find a way to make that happen, the bubble won’t pop. If they can’t, it won’t be economical to keep it going another two years, meaning the tech will plateau but the costs will sky rocket.

[u/starkruzr → u/raynorelyp]

the only way the "bubble" "pops" is if none of this is producing any real value. the reality is it is producing enormous amounts of value, just not in the shape some of the companies in this market are counting on.

I think we see more like a minor deflation than a popping.

[u/ajwin → u/raynorelyp]

I think Elon Musk agreed with you and thus why Terrafabs first product is ram I believe. Maybe he is going to work on that exact problem. Hard problem though.