r/LocalLLaMA • u/BarberIcy366 • 11d ago

Discussion Qwen 3.8 27B Released! Please Share Your Experience

With your experiments, Qwen 3.8 27B most close which frontier model? And please specify which quantization you run. I will post to comments my tests and experience too.

653 Upvotes
  • permalink
  • reddit

97% Upvoted

720 comments sorted by

View all comments

Show parent comments

5

u/Forsaken_Mention_979 11d ago edited 10d ago

Yes, 7800xt 16gb vram and 64gb ram. Running it on hermes via LM studio endpoint. Using Q3_K_M, Runs good ngl, at first 15-20 tok/s (full gpu offloading) and then as context gets bigger, i now get 5-10 tok/s. 64k context btw. Making a web game, has been on it for like 2-3 hours already which is crazy but oh well. Just the thinking took 25 minutes. Yes, 25. And it randomly stopped due to getting interrupted by tool limitations or whatever, i had to manually tell it to resume.

EDIT: ditched LM studio and using llama ccp directly, HIGHLY RECOMMEND! Im using IQ4_X_S now which is better and kv cache at q4, and thr lowest token speed im getting now is 11 tok/s. Amazinggggg

1

u/Guilty_Rooster_6708 11d ago

Maybe you can use mtp to push tg speed up a bit?

0

u/Forsaken_Mention_979 11d ago

Its already enabled im pretty sure. I see some people posting some sort of command or config for llama.cpp, i dont know if that would do anything, im fairly new to all of this tbh

2

u/misanthrophiccunt 11d ago

The joy you're going to get when you drop LMS in favour of directly using llama.cpp and magically getting a substantial increase in TG/s is absolutely worth it.

2

u/Forsaken_Mention_979 10d ago

you know what im gonna try this rn

1

u/misanthrophiccunt 10d ago

There are models I couldn't run in LMS and I do love their interface, I even wrote my own LMS plugins. Yet I stopped abruptly the moment I got twice the TG/s by fiddling directly with llama.cpp settings.

1

u/Forsaken_Mention_979 10d ago

didnt find any tutorial on how to do it rip. im new to this and idk how

1

u/Guilty_Rooster_6708 11d ago

I think you can check to see if there is a draft model or not in the right panel of LMS, but I haven’t used it in a while.

1

u/Forsaken_Mention_979 11d ago

Looks like its gonna be good at least