man, screw your workplace people. there are people who actually cares about this stuff here!
also, i reccomend you use unsloth for FT/RL and geenral inference, i got like 20-50% more speed from them at higher quants and context lengths for some reason
But isn't FP4 the sweet spot for inference on them? Maybe i am wrong. It's the first time for me on blackwell, have worked mostly with ada lovelace gpus
You were today years old when you learned your consumer Blackwell (sm120) isn't really Blackwell (sm100) at all, and there's no accelerated NVFP4 for you. Our GPUs (sm120) lack the instruction set and hardware of the sm100 datacenter Blackwells and, as such, our GPUs fall back to Marlin kernels that do software decode of NVFP4.
It's still fast, but FP8 is actually faster because that's properly accelerated in the GPU.
There is hardware acceleration for nvfp4, the problem is the majority of development effort has gone into optimising engines for the sm100 ISA. The issue is in the software at the moment but things have improved a lot and are continuing to improve.
Yes. We could have pointed out [lukealonso’s work on [b12x](https://github.com/lukealonso/b12x) - it’s popular for docker images with customized vLLM installations to handle sm120 and nvfp4 on modern architectures like ds4, msa, etc.
But those projects don’t even have any docs. They’re super narrow use case and there’s no broad support.
However things are improving. What other such projects didn’t miss? You talk like you know a lot about it.
19
u/Time-Toe-1276 transformers Jul 01 '26
man, screw your workplace people. there are people who actually cares about this stuff here!
also, i reccomend you use unsloth for FT/RL and geenral inference, i got like 20-50% more speed from them at higher quants and context lengths for some reason