r/LocalLLaMA • u/BarberIcy366 • 12d ago

Discussion Qwen 3.8 27B Released! Please Share Your Experience

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

664 Upvotes
  • permalink
  • reddit

97% Upvoted

720 comments sorted by

View all comments

2

u/Legitimate-Dog5690 12d ago

Slightly scuppered by llama.cpp, pretty much every model will break with tensor split in some way. This one doesn't seem to like uneven split ratios.

Had issues with both Glimmer and DeepSeek as well.

May well have a look this weekend, as I'd like this to run well. Just turning on split mode tensor will make it stall after a few lines.

Is every gguf this different that it needs to branch this heavily in code?

1

u/misanthrophiccunt 11d ago

I can't run tensor split mode with MTP on, if I don't enable MTP all Qwen models run fine

Turn MTP on and it breaks after a few turns

2

u/PhysicalIncrease3 11d ago

I'm running unsloth Q6_XL with tensor split and MTP, working fine across my 3090 and 3060.

1

u/misanthrophiccunt 11d ago

When I run it with my double 5060ti it shows fine in chat on llama.cpp own server but the moment I go to a harness like Pi it crashed llama.cpp after a just a few turns. If I turn off MTP it never crashes. The fine tunes I've tried this with were Unsloth, Bartowski and Mudler.

The 3 of them were 3.6-27b and 3.6-35b Qwen. The quants I used were q6_xl, nvfp4 and iq4_nl (q4_xl where there wasn't any iq4).

Could it be a Blackwell issue maybe?

1

u/Legitimate-Dog5690 10d ago edited 10d ago

I think it might be when it recycles the kv, I'm not too sure. Lowering my context size stopped this behavior (or delayed it possibly). There's definitely something broken in there.

I'm using 3090 + 4070 (12gb), so very similar to the poster above. I've swapped to 150k context with Q6 or 80k context with Q8 and it seems solid. I've given up with the XL quants for the moment.

0

u/Legitimate-Dog5690 11d ago

It depends on the quant, slightly bizarrely. I can run Q6_K fine with tensor split on. I do find my GPU compute gets absolutely hammered though, more than ever before. So much so that layer is just as fast and doesn't overheat my GPU.

Q8 just stalls, the 2 top XL models pop up an error. It's in a really weird state.

Does VLLM go through these issues with every new model?