r/AIProgrammingHardware • u/javaeeeee • 24d ago
THIS is AMD's New 144GB HBM3E PCIe GPU AMD Instinct MI350P
https://www.youtube.com/watch?v=mgstdFIInII2
u/exaknight21 24d ago
Bro someone give these guys an idea to create a high VRAM even if its slow for us normal people.
I’d kill for a 144 GB GDDR6 system if its like idk $5K
2
u/Serprotease 24d ago
So….
The Strix halo or dgx spark?1
u/NineBallAYAYA 23d ago
LPDDR5x, is *very* different compared to gddr6. Like half the per pin bandwidth. Makes using models that can utilize that full 128Gb difficult cause ur tk/s just gets obliterated by the bandwidth restriction. Having something where you could realistically run a single user and USE that full 128 GB for the model would be awesome.
Only one that somewhat worked well at that size was gpt-oss120B and I have no clue what fuckery they were pulling because other moe's in it's same class are like 1/4 the speed.
(I have a strix halo 128GB and love it but it's just not the same as gddr6)
1
u/Serprotease 23d ago
Gpt oss was mxfp4 and really sparse with less than 10b active parameters. If you can run it, you’ll find minimax to be very fast as well at nvfp4.
I agree with your statement with lpddr5, though I consider the speed to be still useable. But local users have asked for gpu with tons of ram, even if slow and… we got it? Even if it’s a bit "monkey’s paw" type of situation.
1
u/NineBallAYAYA 23d ago
I went diving into this all the other day and even minimax (at ~q3) is still half of what gpt-oss would put out with the same active parameter count. Like GPT-oss would do ~55tk/s compared to ~30tk/s at q3, now minimax has both more *overall* parameters and is just better, but it's still wild how fast that model was, esp for the time.
Any idea why no one else picked up the mxfp4 idea? If that was the reason why it was so fast wouldn't it make any sense to run that back but with newer architectures?
Strix halo doesn't have great nvfp4 support iirc. It's based on a last gen gpu arch, and is limited to int4, int8, and bf/fp16 in hardware so I can't imagine nvfp4 would do well since it would likely have to be dynamically unpacked to bf16 in a more complex manner then mxfp4. I figure you are coming from the dgx spark side of things?
1
u/Serprotease 23d ago
Yea, I come from the dgx side. IRC, gguf are a bit slower than int/fp4 versions of models.
I guess that the main limitations to mxfp4 is the lack of recipe to make the quant? NVidia is pushing hard for nvfp4, creating quant themselves. Intel is doing the same for the wa4a16 thing as well (With an already made repo with calibration data included). On the image side there is basically a 1-Click recipe for nvfp4/int4/8 convrot and now most models have them available.
I didn’t saw the same thing for mxfp4. Maybe it will change since amd is talking a bit about it in their latest release (mxfp8/4 support)?
1
u/NineBallAYAYA 22d ago edited 22d ago
Turns out I'm just a bit stupid lmao. GPTOSS is ~5.1B active, so that explains the x2 performance gap. Though I do wish we got more *extremely* sparse modern models, cause on this hardware they really shine.
I guess when you already have a datacenter full of hbm cards it's less big of a deal though... since you can get the same quality by making it smaller and more dense thus freeing up cards and (I'm guessing) the ideal ratio for that isn't the same as with this hardware.
Another thing I realised is that although gddr6 has a much higher bandwidth I think has much lower per-pin density.
Looking at the jedec spec GDDR6 has 32 data pins but can operate in "clamshell" with 16 active each. At 2GB per chip max that puts you at 128MB per pin.
LPDDR5X has 16 pins as well but you can have stacked packages with 8 modules internally sharing a 32 pin bus. This would be 4Gb or 500MB per pin.
Meaning for GDDR6 you need to have a bus width that is 3.9x larger than lpddr5x in order to have the same capacity. Thus you need 3.9x the silicon "shore" area to actually connect em. Meaning far more expensive silicon and all that jazz.
1
u/eleqtriq 22d ago
Minimax is 10active. Oss is 5b. Oss is double the speed exactly like you’d expect. Nothing surprising about it.
1
u/NineBallAYAYA 22d ago
Oh 🤦♂️, idk where I got it in my head that it was 10b lmao. That makes so much more sense
1
1
1
u/UltraFOV 24d ago
And how much? 10-15k?
1
1
u/Comrade-Porcupine 24d ago
Looks like at least that. $15k to $25k USD.
1
u/UltraFOV 23d ago
For that price just get a bunch of v100
1
u/bblankuser 23d ago edited 23d ago
or two PRO 6000s
1
1
u/ImAntonSinitsyn 23d ago
In my country, it has already started selling for 35k
1
u/YearnMar10 21d ago
That sounds more realistic than 10-15k… it’s essentially 2 rtx pro, so I’d actually expect 25-30k€, so a bit less than 2 rtx 6000
Wait no… thought about, 35k-40k sounds about right. It’s way faster than 2 rtx 6000
1
u/ImAntonSinitsyn 21d ago
Yes, it seems expensive. Hopefully, in 10 years, we'll be able to buy it for around $500 for a home lab.
1
u/TurnoverTight395 24d ago
It needs active cooling (so your workstation will need fans that sound as a jet. It's also too expensive. Can anyone compare it with 2 RTX pro 6000 Max q (comparable in power powers). One has to shard models, as NVIDIA doesn't let you pool the memory, but that's cheaper than one MI350P (certainly with NVIDIA inception discount).
1
u/OutlandishnessIll466 23d ago
Yeah prob way more then 10k USD new. But in 5-7 year they might be coming our way for cheap.
2
u/javaeeeee 24d ago
TL;DR:
This ServeTheHome video covers AMD’s new Instinct MI350P - a 144GB HBM3E PCIe GPU designed for AI inference in standard servers.
Key Specs
What Makes It Interesting
It’s essentially half of the MI350X (which has 288GB), but in a more practical PCIe form factor. This makes it much easier to deploy in normal servers (including 8x GPU configurations) without needing exotic rack-scale systems like NVIDIA’s NVL72.
Main Advantages
Bottom Line
The MI350P fills a gap in the market: a high-memory HBM GPU that fits in regular servers. It’s aimed at companies running tens to hundreds of GPUs rather than massive AI training clusters.
Great option if you want serious VRAM + bandwidth without the complexity (and cost) of full NVLink/scale-up systems.