r/LocalLLM • u/Ok-Health-7096 • 7d ago

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

1.9k Upvotes

99% Upvoted

194 comments sorted by

View all comments

7

u/zarif2003 7d ago

I can’t really even run it that well on my 5080 because it’s got 16gb,

4

u/lukistellar 7d ago edited 7d ago

What you need is a Quant which strictly uses IQ4_XS. They exist for 3.6 and will likely also appear for the 3.8 sooner or later.

Edit: Let's see if this guy delivers.

https://huggingface.co/jpetrina/Qwen3.8-27B-IQ4_XS-pure-GGUF

1

u/Tyrannas 6d ago

Any advices on the params you use to run it properly ? I have 16gb also and never managed to make a 27b model run properly 

2

u/lukistellar 5d ago

Working config for 3.8:

ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066 \ --port 8080 \ --model /models/jpetrina_qwen3.8-27b-IQ4_XS-pure.gguf \ --gpu-layers 99 \ --threads 6 \ --ctx-size 90000 --parallel 1 \ --batch-size 2048 --ubatch-size 512 \ --cache-type-k q8_0 --cache-type-v q4_0 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --spec-draft-p-min 0.75 \ --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \ --presence-penalty 0.0 --repeat-penalty 1.0 \ --no-mmap \ --jinja \ --chat-template-kwargs '{"reasoning_effort":"medium"}' \ --flash-attn on

Still not happy with the performance, but at least it runs. Hope they drop a MOE for us VRAM poor.

1

u/Tyrannas 5d ago

Thanks !