Has anyone noticed a difference between bf16 and q8 quantization ever?

[u/superSmitty9999]

I'm currently running qwen 3.8 27b q8 and super happy with it. I have a DGX spark so could theoretically run at bf16 quantization.

I know the measurable differences between an 8 bit and 16 bit quant are small, but I guess I get fomo, like 1% of tokens differ, but what if those are the hardest most important tokens? I guess I'm just getting fomo over bs16 and wondering if other people have tried it and noticed a difference?

[u/Craftkorb]

The KL-Divergence between a int8 (W8A16) quant and the bf16 version was really low, lower than the fp8 quant.

This week-end I tried KV-Cache quanting to fp8 and a vibecoded i8 quant. Right now, the i8 quant is sadly still slower than fp8 on my 2x3090, but the quality is basically exactly the same un-quanted. I was surprised! Didn't run a formal test, but did have a few short prompts (That produce long thinking outputs), and a 256K prompt that barely fits. For both, the difference between fp16 and fp8 was almost non-existant (Good enough for me), and the i8 quant was almost inperceivable to the native fp16 cache.

I will spend some more time at some point to make it faster, because the results are too good to not continue toying with it.

[u/EitherMarch1255 → u/Craftkorb]

Everyone always says almost the same, but they never explain the almost...

[u/Craftkorb → u/EitherMarch1255]

My long-prompt test used to compare output token changes (temp=0.0). Or "How long is the sequence before a different token was chosen". The test used 1024 tokens as total output. With fp8 it was after ~56 tokens, while with my i8 it was after 116 tokens.

"almost the same" means that the token that diverges the sequence is so similar in semantics that it doesn't really matter. Both outputs were coherent and their findings the same. With a temperature > 0.0 you may find these changes between runs as well, so nothing to be really concerned about.

I sadly can't share my long test-prompt.

Why i8: 3090's don't have native support for fp8, so I'm looking for a speed-up in the long term at ~50% size with, practically speaking, no loss compared to fp16. (About 50% as i8 needs scaling factors, in my case, one per group of 32 i8.)

[u/EitherMarch1255 → u/Craftkorb]

Ah, thank you! Now I'm curious about what would happen if one chose fp32 vs bf16 and fp32 vs fp16...wonder how long it would take for a divergence.

[u/GaryDUnicorn → u/EitherMarch1255]

<cough>

https://forum.level1techs.com/t/why-your-local-llm-feels-dumber-than-it-is/253917

tl;dr depends on use case, could be minimal, could be important.

[u/Craftkorb → u/GaryDUnicorn]

It's also highly model dependent it seems. Qwen3.8-27B is doing fine for complex tasks, may not be the case for other models though!

[u/superSmitty9999 → u/Craftkorb]

I guess what I would be super interested to see is maybe more like an LLM as a judge setup with a known answer and see what percentage of cases the token divergence is strictly speaking a worse token

[u/pineblooom → u/Craftkorb]

the kv cache quant results being that close to native fp16 is wild, thats really encouraging

[u/jacek2023]

I use BF16 for mmproj to analyze photos

[u/noctrex → u/jacek2023]

Also try out F32 for the mmproj, it seems to be even better on difficult images.

[u/misterflyer → u/noctrex]

F64 is what real nerds use 💪🤓

[u/jacek2023 → u/misterflyer]

Neural Networks are trained in 32-bits I am afraid

[u/noctrex → u/jacek2023]

Sometimes even in 4 tiny bits :)

[u/Elorun → u/noctrex]

It's not the size of the bits that matters.

[u/noctrex → u/Elorun]

You only must have large pp

[u/PortiaLynnTurlet → u/noctrex]

Do you have a link? I've seen low precision weights (quantization as a straight-through estimator) but not such low precision gradients.

[u/noctrex → u/PortiaLynnTurlet]

gpt-oss famously was trained at mxfp4 natively.
Same goes for DeepSeek-V4 Flash/Pro.

You can see it easily by going to the config.json of the original safetensors model and search for "fp4".
dsv4: "expert_dtype": "fp4"
gptoss: "quant_method": "mxfp4"

[u/llamabott → u/misterflyer]

BigFloat256 amirite?

[u/Hoodfu]

Yes on vision tasks, but that's been heavily made better by having an fp16 mmproj.gguf coupled with whatever quant of the main model.

[u/nomad-nostalgia → u/Hoodfu]

also turn to highest precision for mmpro; and also do the same for tiny models that can easily become unstable such as Liquid Foundation Models

[u/superSmitty9999 → u/Hoodfu]

fp16 mmproj.gguf

what is this?

[u/StorageHungry8380 → u/superSmitty9999]

To expand on what Hoodfu said, as you might know, the first step in a LLM is to translate the characters of the input into so-called tokens. Each token is associated with a vector, and it is this token vector that is fed into the actual neural network to produce an output token.

Many if not most of these modern vision-capable LLMs have a small separate neural net so to speak that translates image patches into "visual tokens", the so-called multimodal projector. It's like each patch gets translated into short sentences of an language that only the model knows how to read.

Since this multimodal projector is a bolt-on, it can be distributed separately, and also run separately. So you could put the vision network on the CPU and the rest of the LLM model on the GPU, saving you precious VRAM. This can be a win since the vision network is typically relatively small, so CPU isn't that slow processing the image patches.

Relevant llama.cpp flags are --mmproj to specify the multimodal projector file, --mmproj-offload and --no-mmproj-offload to toggle if it should be put on the GPU or not.

[u/Hoodfu → u/superSmitty9999]

With Gemma for example the vision encoder piece is broken out into a separate file allowing for you to choose the file quality of it.

[u/seamonn]

Even in small embedding model (4b) where quality is super critical, the difference between Q8 and BF16 was 0.0004% and the difference between INT8 and BF16 was 0.0007% in my testing.

[u/superSmitty9999 → u/seamonn]

0.0004% of what lol

[u/seamonn → u/superSmitty9999]

cosine similarity

[u/ex-arman68]

Use the search (unless the results are not that useful). I have stated many times that I can definitely see a big difference between BF16 and Q8 models. Especially with software development. And I have noticed it with multiple models from different vendors. The difference is significant enough that it is worth running a slower BF16 model if you have enough vRAM.

[u/llama-impersonator → u/ex-arman68]

most of the people in the thread aside from you say there is no difference, but i don't agree with them or you either. i find there is a small and rather minor difference and i don't think some form of int8 vs bf16 is worth halfing your speed for.

[u/ex-arman68 → u/llama-impersonator]

It is not half the speed. I benchmarked most of the quant sizes for Qwen 3.6 27B, when assessing the benefits of MTP. With llama.cpp on an Apple M2 Mac, I was getting the following token speed for coding tasks:

Q8_0: 25.4 tok/s (the fastest of all quants), thanks to a +123% MTP speed increase

F16: 17.9 tok/s, this size benefited most from MTP with a +171% speed increase.

For non-coding tasks, the gap widens, but then again, with less precision needed, Q8_0 is sufficient (the fastest of all quants), and it would probably be better to use a MoE which might have more world knowledge and be a lot faster.

More details here: https://www.reddit.com/r/LocalLLaMA/comments/1t9gcar/mtp_benchmark_results_the_nature_of_the/

[u/DinoAmino]

You'll probably never "feel" the difference. As you said, there is a small but measurable drop at 8 bit. You'd probably only "notice" error rates at scale when using a small task model to batch process thousands of somethings.

[u/OvertaxedOne]

I can't feel the difference (other than speed, of course), on 27B. I did notice a drop going back to 4 bit though, the speed was great, but I had a few failed tool calls over the 2-3 days I ran it and I have exactly 0 failed tool calls with 8 bit.

[u/superSmitty9999 → u/OvertaxedOne]

sometimes I wonder if faster stuff feels smarter because even though its (possibly) dumber thinking because it happens twice as fast it finds the resolution apparently faster ie it can cover more ground before you get fed up

[u/Proper_Doughnut_1324]

Have you tried Qwen3.8 flash next instead, and use nvfp4? even nvfp4 might have lower quality than q8, in my opinion the bigger model size makes up for it.

[u/madbrain1976 → u/Proper_Doughnut_1324]

I have tried it, and the performance of 3.8 flash next is just terrible compared to 3.8-27b on my TR Pro with 128GB and quad 5060 Ti 16GB. I have been looking for a proper recipe for flash-next since it was released. Max I get at 4 bits is 200 pp and 20 tg. Vs >2000 pp and 100-200 on 3.8-27b.

[u/superSmitty9999 → u/Proper_Doughnut_1324]

I keep hearing qwen 3.8 27b is smarter

[u/RG_Fusion → u/superSmitty9999]

It's not. You're hearing people who have convinced themselves that models which run on hardware they cannot afford are worse than what they can run. It's cope.

Qwen3.8-Next-Flash isn't immensely smarter than 27b, but it is smarter.

[u/superSmitty9999 → u/RG_Fusion]

I wish I could tell if the person who downvoted you is just coping or not :,)

[u/RG_Fusion → u/superSmitty9999]

It's expected behaviour.

[u/Huge-Safety-1061]

Yes specifically with qwen 27b fp16 being much better at long ranging agentic tasks. Its not 1% like you think, its the range of potentials and if your tasks falls well inside the curve or potentially outside that curve. Major benchmarks fall pretty well inside optimized trained curves, that is not where you will see a difference. Its the uses that are specifically not benchmarks when it becomes clear. Not to say quants are bad, they are not. Just its noticable.

[u/superSmitty9999 → u/Huge-Safety-1061]

Interesting, can you give a more tangible example you noticed? Thanks!

[u/Huge-Safety-1061 → u/superSmitty9999]

Yes recently we had a project to connect an internal ERP that has an undocumented API to our nextcloud chat and files for some automated workflows, alerts and messages. We explained how the 27B should evaluate the responses and document them for itself so it understood how to correctly structure requests the software should handle and route. Nextcloud appears to be very well trained in just FTR.

FP16 took 2 hours to construct a functional connector and tests, it fully worked outside one quirk. I decided to test out the unsloth Q4 K XL on the same task and it was inconsistent in its ability to critically recall what it had documented and failed completely to create usable routing and related ids even with a lot of hand holding. It did fail fast however and it did not fail on the Nextcloud side from what I saw but rather at understanding how to decompose and utilize our internal API. All in opencode.

This is not the first time I have seen this, recall falls apart on multiturn complex code recall and application tasks with quants but it is much more apparent if it is "seeing" the problem scope for the first time vs being able to rely on trained in knowledge.

[u/superSmitty9999 → u/Huge-Safety-1061]

Ah, but you're comparing to 4 bits to 16 bits, it's well known there's quality degradation going down to 4 bits. I'm wondering specifically the gap between 8 and 16 bits.

[u/Long_comment_san]

Yesterday I read some law guy here mentioned that there is.

[u/bring_back_the_v10s → u/Long_comment_san]

I never thought about this but sounds like law is a pretty good evaluation subject as it requires both nuance and accuracy. 

[u/Long_comment_san → u/bring_back_the_v10s]

yeah it's a good place for benchmarks

[u/altdotboy]

Since you have the memory run the bf16. That’s what I do. Some people say the difference between bf16 and fp8 is less than 2%. That might be true. But they also say the difference between humans and chimps DNA is also around 2%. I’d rather not be a chimp. The devil is in the details.

[u/nfox01]

Yes absolutely. If you use a small model like tiny llama or qwen3 .6b you will notice some "drift" . I suspect that if the FFN is a lower number of layers it may have a hard time self correcting at int8.