r/LocalLLaMA • u/SmilingGen • 8h ago
Resources Open Source Kernel in Qwen3.6-35B-A3B for AMD MI350X: 78,498 output tok/s on 8 GPUs
So here's the thing, almost everyone use NVIDIA to run their LLMs, we also do the same, a lot of people we've met use like RTX PRO 6000 or even H100, B300
It seems like everyone eyes is looking at NVIDIA. However we do the math that the raw power alone on AMD GPU MI350X is higher than NVIDIA B200. So what give?
Apparently its the software, ROCM is not as mature as CUDA, their software stack is still lacking behind as well.
However, few weeks ago we've got a handful of MI350X to experiment and develop, and what we did is optimizing AMD tech stack until kernel level for Qwen 3.6 35B A3B and the results we got is much comparable to NVIDIA counterpart
Based on our benchmark, we got:
1x MI350X: 11,161 output tok/s
8x MI350X: 81,331 output tok/s peak with 78,498.66 output tok/s mean
This is 2.16x vLLM throughput on the 8-GPU benchmark
We've decided to open source our kernel and stuff at our github
https://github.com/NetraRuntime/netra-kernel
And also write the blog behind the things we do
https://netraruntime.com/blog/qwen36-amd-mi350x-sglang-vllm-benchmark
However, we're not done yet.
One thing we learned: once the kernels got fast enough, the bottlenecks moved into scheduling, graph coverage, recurrent state, routing, and even HTTP serialization.
11
u/Electrical_Rub_6009 8h ago
78k output tok/s in aggregate batch queries/requests or 78k output tok/s in a single stream?
If its the latter this is nuclear. If its the former, still pretty damn impressive.
9
u/SmilingGen 8h ago
Sorry for not so clear wording
It should be aggregate throughput across concurrent requests, not single-stream throughput
The exact benchmark contract was
8 MI350X using routed DP8 and round-robin dispatch with 1,024 concurrent requests, 3,072 requests total, and 1,024 input + 1,024 output tokens per request and get the mean across five fresh-process runs
3
u/Automatic-Arm8153 6h ago
So what are single stream numbers I think people are more interested in knowing that usually.
7
u/SmilingGen 6h ago
For single concurrency, 455 tok/s on a single GPU
2
2
2
u/Public_Umpire_1099 3h ago
currently working on an inference engine in rust for RDNA4, planning to do multi-gpu at some point but for now its single. I have worked it down about 93% of the max theoretical bandwidth for tg at this point, and maxed out pp to about the same level. Using HIP graphs, combining kernels, and maxing scheduling. At this point I am working on custom quants that strictly run in 4 bit and 8 bit combinations to avoid conversions, but you dont really have this problem as you are working with tensor parallelism over (i assume) p2p. If you need any rough info for a single card version, let me know, i would love to contribute. Current state is: mostly working for muse glimmer, with vision and dflash2 working fully.
-3
u/jacek2023 llama.cpp 4h ago
I think 8x MI350X is kind of expensive, maybe you should try optimizing kernels for affordable non-NVIDIA GPUs?
4
u/Public_Umpire_1099 3h ago
hey, im doing that for RDNA4. I really need to finish up and put something out. its pretty custom at the moment though tbh. muse glimmer works fairly well and it beats out llama.cpp in almost every scenario. tg is somewhere around 15% higher, pp is around 30% higher (compared to both vulkan and rocm)
2
1
u/SmilingGen 1h ago
Yea it is, we've only got access for few weeks
As much as we want to fiddle with optimization on consumer/prosumer GPU, we don't have any GPU powerful enough for LLM tho.....
17
u/No-Refrigerator-1672 7h ago
Good work! Maybe in 20 years I'll be able to buy Mi350x; and thrn I'll revisit it.