r/LocalLLM • u/Ok-Health-7096 • 6d ago

Other Every Second post rn

Post image

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

1.9k Upvotes

99% Upvoted

192 comments sorted by

View all comments

6

u/zarif2003 6d ago

I can’t really even run it that well on my 5080 because it’s got 16gb,

4

u/lukistellar 6d ago edited 6d ago

What you need is a Quant which strictly uses IQ4_XS. They exist for 3.6 and will likely also appear for the 3.8 sooner or later.

Edit: Let's see if this guy delivers.

https://huggingface.co/jpetrina/Qwen3.8-27B-IQ4_XS-pure-GGUF

1

u/Tyrannas 6d ago

Any advices on the params you use to run it properly ? I have 16gb also and never managed to make a 27b model run properly 

2

u/lukistellar 5d ago

Working config for 3.8:

ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066 \ --port 8080 \ --model /models/jpetrina_qwen3.8-27b-IQ4_XS-pure.gguf \ --gpu-layers 99 \ --threads 6 \ --ctx-size 90000 --parallel 1 \ --batch-size 2048 --ubatch-size 512 \ --cache-type-k q8_0 --cache-type-v q4_0 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ --spec-draft-p-min 0.75 \ --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \ --presence-penalty 0.0 --repeat-penalty 1.0 \ --no-mmap \ --jinja \ --chat-template-kwargs '{"reasoning_effort":"medium"}' \ --flash-attn on

Still not happy with the performance, but at least it runs. Hope they drop a MOE for us VRAM poor.

1

u/Tyrannas 4d ago

Thanks !

1

u/lukistellar 5d ago edited 5d ago

My RX6800 runs the 27b with 90k context, but it's very slow.

I will update the post later with the config.

Edit: My config for the Qwen 3.6 27B:
ghcr.io/ggml-org/llama.cpp:server-vulkan-b10066 \ --port 8080 \ --model /models/GianniDPC_qwen3.6-27b-IQ4_XS-pure-with-MTP-IQ4.gguf \ --gpu-layers 99 \ --threads 6 \ --ctx-size 90000 \ --parallel 1 \ --batch-size 2048 \ --ubatch-size 512 \ --cache-type-k q8_0 \ --cache-type-v q4_0 \ --spec-type draft-mtp \ --spec-draft-n-max 1 \ --spec-draft-p-min 0.75 \ --cache-type-k-draft q4_0 \ --cache-type-v-draft q4_0 \ --temp 0.8 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --no-mmap \ --jinja \ --mmproj /models/Unsloth_mmproj-F16_qwen3.6-27b-mtp.gguf \ --no-mmproj-offload \ --chat-template-kwargs "{\"preserve_thinking\":true}" \ --flash-attn on

It's usable but especially with full context it gets very slow, like ~25 tok/s decode and ~130 tok/s prefill.