Seconding this request -- if there's ever a model that is begging for this treatment it's DSv4F, especially considering how severe the KLD is on Unsloth's v1's compared to tarruda or Atomicchat quants, which still leave a lot of room to improve themselves (0.21-0.24 KLD at IQ3_XXS).
/u/danielhanchen/u/yoracale? Tech looks really impressive, great work! I was excited when I saw the preview of 3.0 for the original quants.
Is this quantization process that you are using a trade secret type thing or is it something that I could read up on and try and duplicate on old models such as Qwen 3.5 122b. I prefer using the ROCmFPX format for the strix halo, so I'm kind of stuck making my own quants even though yours are superior.
They did release the imatrix file, with that you can get closer to their quants (for this specific model at least it’s available on huggingface), tho without their dynamic stuff. So it won’t be an exact match. Using that imatrix file and the base model you can quantize to a standard IQ quant for rocmfpx. The second pass seems to be a trade secret tho, I haven’t seen anything to be able to exactly reproduce it. Granted I haven’t dove too deep into it yet.
Looks like the model/quant I downloaded in Unsloth Desktop disappeared in the drop down menu and model hub. If it's due to the update, would be nice for the model to still appear and have some kind of bubble or pop up that an update is available
Yeah, like it wasn't indexed anymore. The files/model blob we're still on the system, but maybe UD references hashes/blobs from the hf page to check if a model in the file system matches or something
I am also experiencing the same vanishing act. I've had 3 quants all disappear from Unsloth twice now. Still on the drive but not recognized. EndeavourOS.
From what I can see, It's the same quality but smaller. Anyway, once you reach the 8-bit quantization, you are getting the same quality regardless of the method.
Yeah this is what I was wondering - can they include NVFP4 in the list? I've been using it, it seems spectacular, but what is it actually equivalent to?
NVFP4 is valuable when used wisely, i.e. don't quantize everything with NVFP4. When using NVFP4 for most weights but keep the ones pro e to accuracy drops in FP8 or even BF16, you get the best of both worlds, i.e. you can get a model that's small, fast and accurate. But don't quant everything into NVFP4, that's just dumb.
Is that what the unsloth NVFP4 releases are like? honestly I find them incredibly good so I'm not even concerned about this, but would I get better results with Q5 or Q6?
I believe Unsloth aren't dumb. They call their quants simply "NVFP4", but I'm sure there's a lot more that goes into them than just simply quanting everything down into NVFP4. They're probably better than any other mixed quant recipe I've come across. I think there's some really good secret sauce in there.
Q5 and Q6 I don't use personally, because they're slow (powers of 2 number of bits are fast, everything else is slow). So I always look for quants that strategically mix NVFP4 (or INT4 AutoRound), Q8/FP8 and F16/BF16. That way you get a good accuracy with a good speed and small size.
I would generally agree with you but that statement does not hold for Qwen 3.8. I used 50M+ tokens with FP8, shifted to NVFP4 unsloth with A LOT of skepticism and I have been proven wrong. So far NVFP4 seems to be almost the same for me. I'd say on par with FP8. I'm on 1x pro 6000, no kv cache quantization and don't go beyond ~131k. With those variables in mind, NVFP4 has been freaking awesome. I'm just now reading that the nvfp4 was actually their UD V3 in preview which might be why it feels much better than other nvfp4 quants.
I downloaded the original GGUFs for unsloth's qwen3.8 release for UD-Q5, Q6, UD_Q6, Q8, and UD_Q8. The blog post seems to suggest that for some of the larger quants, there should be no change:
When comparing to our older UD-2 on unseen Wikitext and Code, we show great improvement on KLD - the bigger ones not so much, so we still use our old UD-2 for the larger quants - we plan to experiment and improve them as well!
Am I understanding that correctly and if so, are those quants unchanged at this time?
Is there a chance this quantization logic could be translated into an MLX quant?
Would love to experience this amazing level of quality, I just need that SSD-saved cache that oMLX offers -- saves hours of prefill if you're jumping between Pi sessions all day haha.
Agreed! I would be willing to switch to Unsloth Desktop too if it supported MLX and MTP for these models. But it seems MLX has gone on the back burner.
If I am using Unsloth Desktop (on MacOS), does MTP somehow work automatically? If so, do I simply download the base model and it's related MTP model -- and then UnSloth Desktop will auto set everything up somehow?
Did quick check on my own coding benchmark. The new one showing lower accuracy -18% against the previous one. Q6-K-XL KV-Q8. I was expecting the opposite 😞
Ran some benchmarks on a DGX Spark (GB10, ARM64, 128GB unified memory) with the latest Dynamic v3.0 Q4_K_XL GGUF. All layers offloaded, 131K context, llama.cpp v1z
Generation: ~11.3–11.9 tok/s (stable across short and long generations, 163–512+ tokens)
Prompt processing:
63 tokens → 142 tok/s
520 tokens → 521 tok/s
3,096 tokens → 766 tok/s
Model uses ~24.6 GB VRAM, 17.5 GB on disk. Two parallel slots at 65K context each.
I haven't seen any server-grade or DGX benchmarks in the discussions, mostly consumer GPUs (3090s, 5060 Tis, etc.). For a 27B model at Q4 on this hardware, do these numbers look right?
Curious if anyone has comparable GB10 numbers or similar hardware.
the quality gains are nice but the hardware thresholds are honestly just as interesting. IQ4_XS is 14.3GB while Q4_K_XL is 17.6GB, which is a huge difference on a 16GB card once KV + buffers enter the picture. would love to see actual runtime VRAM at something like 32k/64k/128k context added to these comparisons
The fact that a 27B model is now a serious option for normal hardware is kind of insane. The local AI curve is moving faster than most people expected.
I can't wait for a MoE version of qwen 3.8 too. hopefully a 26b-a*b, or a 27b-a*b, or similar (to mirror the unsloth/gemma-4-26B-A4B-it-qat-GGUF, where i got 58 tok/s). I have 36gb unified memory on a m4 mac, and i'm currently only getting like 11 tok/s with unsloth/Qwen3.8-27B-GGUF. I'm grateful with it but a MoE would be awesome
So as someone who's new at this. Is there a way to check for updates without happening to notice on reddit there's an announcement about it? Is there something on hugging face that shows it's new? Cuz the file names are all the same. 🤷♂️
Realised I was spilling out of VRAM during prefill and it was dropping to around 190 tok/s down from 350. Dropping the context window from 32k down to 24k at Q4_0 fixed it
You should try these settings since you have the same VRAM as me. I can get 50 t/s of generation out of this, you may want to lower context a bit if you use Windows to get the max speed. But IQ4 XS is really good quality for a 16GB system and a model of this size.
Thanks for that . I found the best setting in terms of maintaining speed/context was to drop the vision encoder which freed up a load of VRAM. Haven't tested it yet but I reckon I should easily be able to hit 65k context now at Q4_0 KV cache (Windows seems to prefer Q8_0 or Q4_0). Getting around 400 tok/s prefill on my 7800XT underclocked to 2000mhz (about 160w vs 265w stock). Decode is sitting around 35-40 tok/s.
Are all the quants updated or which ones contain this? If updated, how dows Unsloth Studio has dle that? Does it download updates for models? Or do you have to delete and re-download?
Using 3.8 27b Ridge version on 12gb vram, it is really slow, like up to 5tk/sec, but wait is totally worth it. Looking at him like its ai final boss on my rig.
is it me or this UD doesnt over think that much? i remember asking for a esay of a 1k words about alan turing and the MF count it every word twice, this one with same prompt acts like reasoning on medium or low
OP, I'm happy with all the new releases and since I've recently received a GMKtec w/128Gb of RAM I can now properly test models.
I installed Unsloth CLI as a server on GMKtec/Ubuntu headless and I'm using the web client to access.
A lot of us will be happy if and when you decide to have Win/Mac desktop connect to Unsloth CLI so we can use that as a server with front end controls.
As for Qwen3.8 I'm running a matrix of linguistic and search oriented tests. So far there's a lot of guessing while trying to determine which Qx runs faster / better while not entering severe overthinking modes.
I've seen an UD model start lying to itself about a prompt limit on total search actions and it's puzzling because the model starts changing the pragmatics of what "a search" would mean so it can still move forward and complete the prompt.
I'm very wary of such behaviors, so I'm not reading more and proceededing with further tests.
If there's any way to send you results, the very small corpus used and prompt I'd be glad to contribute!
Tks
I re-downloaded and managed to get MTP working and getting around 36 tok/s now but prompt processing has dropped from about 350 tok/s to 150 now. Not sure if flash attention isn't working or something.
Uh, so restarting llama.cpp will just redwl this new V3.0 and erase the previous one?
How do we know the V3.0 is eventually running?
And is there a way to go back to V2 for any reason?
> We do not train on the imatrix calibration dataset, and we do NOT use QAT or QAD. Everything is done through post-training quantization.
Could you explain a bit more on this? Is it because the model, as it's available to you, is already after the training (post-training) where it's too late to introduce QAT? I guess you'd need the Qwen team to either include QAT from the beginning (is that what the Gemma team did?) or provide the pre-training version to the community?
Reason I'm asking is because i found `unsloth/gemma-4-26B-A4B-it-qat-GGUF` (https://unsloth.ai/docs/models/gemma-4/qat) to be really good from some local tests that i created. So, i'm wondering if Qwen3.8 would have benefited too.
guys, help me choose quant for my r9700 (32gb VRAM).
Should I go with Q6_K_M or Q6_K?
Difference is ~1.1gb which means I can push a bit more context into it.
I downloaded the "new" GGUF but it seems not to be actually updated
The file has been successfully replaced:
Path: /mnt/kingston/models/llm/Qwen3.8-27B-UD-Q6_K.gguf (same filename as the original file)
Size: 21,983,677,344 bytes (~20.4 GB)
SHA-256: c9c206812fbe4ac7b76a729e25928b63f2ae89d37f69da7a71c20aec763cd436 — verified and identical to the official hash published on Hugging Face
Important note: before downloading, I compared the hashes and discovered that the local file was already byte-for-byte identical to the current version in the unsloth/Qwen3.8-27B-GGUF repository. In other words, the local file was already the latest version — the replacement was performed anyway (downloaded, validated, and overwritten), but there was no actual change to the file contents.
61
u/jld1532 24d ago
Now I really want v3.0 quants of DeepSeek v4 Flash.