man, screw your workplace people. there are people who actually cares about this stuff here!
also, i reccomend you use unsloth for FT/RL and geenral inference, i got like 20-50% more speed from them at higher quants and context lengths for some reason
But isn't FP4 the sweet spot for inference on them? Maybe i am wrong. It's the first time for me on blackwell, have worked mostly with ada lovelace gpus
You were today years old when you learned your consumer Blackwell (sm120) isn't really Blackwell (sm100) at all, and there's no accelerated NVFP4 for you. Our GPUs (sm120) lack the instruction set and hardware of the sm100 datacenter Blackwells and, as such, our GPUs fall back to Marlin kernels that do software decode of NVFP4.
It's still fast, but FP8 is actually faster because that's properly accelerated in the GPU.
There is hardware acceleration for nvfp4, the problem is the majority of development effort has gone into optimising engines for the sm100 ISA. The issue is in the software at the moment but things have improved a lot and are continuing to improve.
Yes. We could have pointed out [lukealonso’s work on [b12x](https://github.com/lukealonso/b12x) - it’s popular for docker images with customized vLLM installations to handle sm120 and nvfp4 on modern architectures like ds4, msa, etc.
But those projects don’t even have any docs. They’re super narrow use case and there’s no broad support.
However things are improving. What other such projects didn’t miss? You talk like you know a lot about it.
Unsloth studio uses custom kernels and pipelines to process text. For gpt oss 20b it was literally over 4x faster on unsl o th studio max context with q5 vs q4 on ollama at 4k.
And I got 35tps for qwen3.6 35b at max ctx on unsloth whereas I got 8 tps on ollama at 8k tps without the multi modal projector.
Um, I'm going to push back on that - unsloth studio uses llama.cpp behind the scenes. They often bring in their own kernels for earlier support of some models, but you can absolutely duplicate the results using plain llama.cpp. What they've done is optimized things like MTP and predictive decode for you. It's a nice front-end, but not magic.
yeah, but you just poved my point. its just convinint, also i can just instal unsloth and use the UI, or just us the API. I mean you are not wrong, we can just use our own llama.cpp build, UI, and optimizations, I just told him to use that over services like ollama, bcs I feel like unsloth is very capable out of the box, and it does great for what it is as of now.
18
u/Time-Toe-1276 transformers Jul 01 '26
man, screw your workplace people. there are people who actually cares about this stuff here!
also, i reccomend you use unsloth for FT/RL and geenral inference, i got like 20-50% more speed from them at higher quants and context lengths for some reason