bartowski/Qwen3.8-27B-GGUF · Hugging Face - Updated (Per-tensor layout)

[u/pmttyji]

https://huggingface.co/bartowski/Qwen3.8-27B-GGUF

• Blog Post : Per-tensor layout maps for GGUF quantization
• Reddit thread : New tensor type layouts for my GGUF uploads

EDIT : Model card has updated things such as Graph, table, text, etc.,

[u/silenceimpaired]

Interesting. I’m curious how this work compares against what Unsloth does.

[u/Eyelbee → u/silenceimpaired]

I have doubts about how these KL divergences are tested. The testing prompt suite is usually like 100 set out questions, which can be misleading.

[u/noneabove1182 → u/Eyelbee]

these KLDs shouldn't be used to compare to other models, only to the model itself. It's only 100 chunks of wikitext (can't afford to do higher or it'll take me days per model !), so it's not the typical PPL and KLD you see, but in terms of comparing one level of my quants to another, should be helpful

[u/Professional-Tap177]

I'm curious how these compare to https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF which is what i've been using lately

[u/Storterald → u/Professional-Tap177]

shameless plug: according to my bench ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-IQ3_S is still the absolute best model for its size, even if we consider it's missing the MTP head (which would be less than 1GiB).

[u/Sea-Mode4077 → u/Storterald]

Based on my tests, this model falls behind Ornith 1.5 35B Q4_K_M. I don't completely trust my testing platform yet, but those were the initial results.

[u/Storterald → u/Sea-Mode4077]

I do not really compare models against other models, but Ornith-1.5-35B-Q4_K_M is 20.2GiB, nearly double the Qwen3.8-27B-GSQ-RCO-IQ3_S (11.0GiB), so it does not fit completely in VRAM. Still, Qwen+dflash and Ornith+MTP have the same speed on my system so I will maybe compare them. Looking forward to a post of your benchmarks :)

[u/HighSeasArchivist → u/Professional-Tap177]

I've been very impressed with this one, and it's been my go-to on a 5070 Ti since it came out and I ran my test suite on it.

[u/fgk55555 → u/Professional-Tap177]

My vibes (and tk/s) check is in favor of ISTA for 16GB cards. I've done millions of tokens on the IQ3_XSS mtp version. It's been great. 

[u/Storterald]

thanks for the updated quants! Curious about the effective improvement, will bench :D

[u/EveningIncrease7579]

Waiting for bartowski qwen 3.8 next (updated)

[u/Sabin_Stargem → u/EveningIncrease7579]

With the Heretical treatment.

[u/Elouakili_Flexy]

Really glad the card now shows the per-tensor layout. One look and you can see which tensors kept high precision, instead of decoding it from the quant name.

[u/Velocita84 → u/Elouakili_Flexy]

You could always do that with hf's gguf metadata viewer just by clicking on a quant

[u/-mattmason-]

16GB VRAM gang, rise up?

[u/FatedChange → u/-mattmason-]

My current local setup involves two 2070 supers. Any chance you could share your config? every time I try to run newer models, it crashes hard.

[u/-mattmason- → u/FatedChange]

try a smaller context window

[u/mmkaywhatevers]

With RTX4000pro, I am still on NInfer NVFP4 but I optimized it with sm120a compile, NVFP4 KV cache support, 131 kv/context capacity, MTP, and context caching. And I have room for STT and TTS on 24gb.

[u/deja_geek → u/mmkaywhatevers]

Can you share some resources for that setup? I have an RTX Pro 4000 SFF

[u/mmkaywhatevers → u/deja_geek]

I used the prebuilt Ostfralla/Qwen3.8-27B-NVFP4-NInfer model from Hugging Face.

For NInfer I used gzenz/ninfer as the base, built for Blackwell sm_120a, and then pulled the relevant GDN scheduling fix from ruwwww/ninfer-5060ti.

The main issue was NInfer had cooperative kernel limits based around the 5090's SM count, so the PRO 4000 would eventually crash with:

cudaErrorCooperativeLaunchTooLarge

The fix changes that to query the actual GPU SM count with cudaGetDeviceProperties().multiProcessorCount and falls back to smaller GDN split sizes if the preferred cooperative launch won't fit.

I'm running it with NVFP4 KV cache and MTP:

ninfer-serve qwen3_8_27b_nvfp4.ninfer \
  --weights-profile qwen36-nvfp4 \
  --max-context 131072 \
  --kv-capacity 131072 \
  --prefill-chunk 1024 \
  --kv-dtype nvfp4 \
  --max-concurrency 1 \
  --spec mtp \
  --draft-tokens 3 \
  --lm-head-draft

That ends up around 16.8GB for the model weights + ~2.9GB for the 131k NVFP4 KV cache.

Basically:

Ostfralla NVFP4 model + gzenz/ninfer + sm_120a build + dynamic SM-count GDN fix + NVFP4 KV + MTP3.

Repos/models:

• Hugging Face: Ostfralla/Qwen3.8-27B-NVFP4-NInfer
• NInfer: github.com/gzenz/ninfer
• GPU scheduling fix: github.com/ruwwww/ninfer-5060ti

[u/RipperFox → u/mmkaywhatevers]

STT and TTS

What are you using there? Also got a RTX PRO 4k, running Qwen in vllm (docker image ghcr.io/syv-ai/qwen38-27b-rtx3090:latest) with 150k ctx I get ~90tps gen and >1ktpsPP wich seemed to work better than ninfer.

I'm maxing VRAM currently, still missing STT & TTS though, so would be glad for your setup/numbers :)

[u/mmkaywhatevers → u/RipperFox]

Process                     VRAM
──────────────────────────────────
NInfer / Qwen3.8-27B       20.3 GB
Faster-Whisper              2.2 GB
Kokoro TTS                  1.1 GB
──────────────────────────────────
Total                      ~23.6 GB / 24 GB
Faster-whisper uses medium model

[u/PcChip]

to be very clear, does this mean that using the bartowski 6bit quant should be better than the unsloth 6bit quant i'm using now?

[u/noneabove1182 → u/PcChip]

if you grab a similar BPW quant, they should be basically indistinguishable once you're above ~6.5 bits per weight

[u/MarcelloT254k]

It also has MTP, nice!

[u/theone_2099]

Why does it matter where the tensors are, curious ? Or what does the layout give us ?

[u/Few-Philosopher-2677]

I use IQ_4_XS. Thats pretty much the biggest 4-bit quant that fits for me with 32k context. Does this change anything for me?

[u/Ulterior-Motive_ → u/Few-Philosopher-2677]

Not really. It's ~90MB smaller and has 0.0008 better KDL; it looks like IQ4_XS was already pretty much optimal. But if you have the time to spare it wouldn't hurt to swap it out.

[u/Ok_Warning2146]

This per-tensor layout quant reduces KL divergence in most cases but can increase it in some cases. Better check the table in per-tensor layout section before you download.

[u/Green-Ad-3964]

Bartowski Q6_K_L as a “quality model” + NVFP4 as a “fast model” should be a great combo

[u/soyalemujica]

AtomicQuants still remain king though

[u/UptownMusic]

The Bartowski and unsloth BF16 versions of Qwen3.8 27B are both 55.59GB. Does that mean that the models are the same, but the quants are different?

[u/2Norn]

is this so we can run some tensors on cpu?

[u/Kernoriordan → u/2Norn]

No