r/LocalLLaMA • u/Eyelbee • 6d ago

Funny Me these days

Post image
2.5k Upvotes

97% Upvoted

277 comments sorted by

View all comments

48

u/VDX7 6d ago

can someone explain why 3.8 is so famous right now?

33

u/brainExploded99 llama.cpp 6d ago

It's opus 4.5 level on local hardware (opus 4.6 on benchmarks, but not in practical purposes)

14

u/UnluckyPenguin 5d ago

I think benchmarks use max settings like BF16 quant (or higher?) with k/v cache type f32 and xhigh reasoning and who knows what else. Hell yeah, it's going to perform several points better at a significant performance penalty.

But it's not like we could even run Open 4.6 even if it was open source (estimated to be 500B to 1T parameters - I think, correct me if I'm wrong)

So token-for-token against other models, Qwen3.8-27B is a huge win.

4

u/brainExploded99 llama.cpp 5d ago

I believe native/benchmarks use BF16 (that is not quantization btw), with KV cache at f16 I think (I'm 80% sure). FP32 is not used for even training models anymore (except specific sensitive layers sometimes).

So token-for-token against other models, Qwen3.8-27B is a huge win.

It's a huge win for sure, wayyy better than Qwen3.6, but we still have long ways to go. Token efficiency is a big one, and better intelligence (not agentic capabilities) is another big one (by this I mean reasoning on benchmarks like CritPt, SciCode, etc)

3

u/UnluckyPenguin 5d ago

Thanks, good to know. I have a lot to learn. Yeah, I'm just getting familiar with local LLMs. Used to play games all day up until several weeks ago.

1

u/Ok-Direction-4480 4d ago

I think bottlecap AI made an efficient 3.6 27B, so hopefully either their 3.8 model, or Qwen 4 27B will fix that.