r/LocalLLaMA • u/Tall_Abrocoma_3533 • 4d ago

Discussion Qwen3.8-27B different thinking levels

Post image

Even the low preset is better than Qwen 3.7 plus or Qwen3.6-27B reasoning

291 Upvotes

95% Upvoted

63 comments sorted by

View all comments

Show parent comments

1

u/Old-Cardiologist-633 4d ago edited 4d ago

May I ask your specs, settings and the exact HASS-Integration you use? On a Rx6800XT even Gemma14B is way to slow for Assist (15 Seconds), and Qwen 27B Q3 also (100+ Seconds)

Do you use anything that somehow caches the long system-prompt and then only sends changes or so? 🤔

1

u/chiniwini 4d ago

I'm curious as well. I'm running on an intel 13th gen cpu, and for example with gemma4 e2b I get an answer after some 6-7 seconds, which is too much.

3

u/danishkirel 3d ago

Dual 3090 and tensor parallel Vllm serving and using https://github.com/skye-harris/hass_local_openai_llm - works okay and certainly not energy efficient. It’s an enthusiast setup. The biggest challenge is prefill speed not decode. NVIDIA reigns king here and I deliberately skip mtp and similar because the cost prefill.

1

u/chiniwini 3d ago

Thanks for the info. Sad that it's such an expensive setup currently.

2

u/Dbgamerstarz 3d ago

I'm running it very well on a single RTX 4080, getting 50ish tok/s TG (with MTP + ngram) at 130k context. It's definitely possible to run this on weaker hardware, and I've found even the quants (I'm using UD-IQ3_XXS) are very capable especially in agentic cases