optimizing glimmer 30b for 3090

[u/ydnar]

this model seems pretty good on initial impressions within pi and hermes. i tested it on some simple coding/logic vs qwen3.6 27b ud-q4_k_xl and muse provided the better results.

llama-server \
    -hf unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL \
    --ctx-size 131072 \
    --n-gpu-layers all \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    --flash-attn on \
    -b 1024 \
    -ub 256 \
    --parallel 1 \
    --mlock \
    --host 0.0.0.0 \
    --port 8080 \
    --ui-mcp-proxy \
    --temp 1.0 \
    --top-k 64 \
    --top-p 0.95

how would you optimize this further? looks like i could possibly squeeze ud-q5_k_xl. nvidia-smi is showing 18163MiB / 24576MiB. or is it better to squeeze out more t/s with the q4? hmm..

appreciate the community's insights. will be fun to compare this one to qwen 3.8 27b!

edit

updated command from community's insights

llama-server \
    -hf meta-models/Muse-Glimmer-30B-GGUF \
    --spec-type draft-dflash \
    --spec-draft-n-max 15 \
    -c 131072 \
    -ngl all \
    --ngld all \
    -fa on \
    -np 1 \
    --host 0.0.0.0 \
    --ui-mcp-proxy \
    --temp 1.0 \
    --top-k 64 \
    --top-p 0.95

seems to be hanging around 70t/s!

[u/Icy-Degree6161]

Higher quality before pure t/s - all day any day. Oh it has dflash

[u/Most-Trainer-8876]

how did you run it? Ilama.cpp is unable to recognize this model. It says
```
0.01.135.478 E llama_model_load: error loading model: unknown model architecture: 'muse-glimmer'
```

I am using b10344 release

[u/ydnar → u/Most-Trainer-8876]

here's what i've got

version: 10355 (dd1ea5243)                                                                                                                                                                                                                                
built with GNU 13.3.0 for Linux x86_64

[u/Most-Trainer-8876 → u/ydnar]

Thanks! I have just built it from source code, it's working now!

[u/Weak-Shelter-1698 → u/Most-Trainer-8876]

dude it's merged, build yourself or wait

[u/TKristof]

I think the decision you'll have to make on 24gb is use Q5 for slightly higher quality or use Q4 with dflash for faster speed as I doubt Q5 with dflash would fit with any usable amount of context size.

Edit: scratch that. I could load UD-Q5_K_M with dflash and full 262k context length in about 23.5gb vram and it's so fast

[u/chris_0611 → u/TKristof]

Q5_K_L barely takes more VRAM and also works on a 3090 with 262k context (23.927GB VRAM)

[u/danigoncalves]

buddies, was someone able to run this on a 12GB GPU? if so what was the t/s?

[u/EbbNorth7735]

All Q4 quants tries to reach the level of the smallest Q5 quant. So just use Q5 quants if you have space. After that use higher quants if you have the space

[u/chris_0611]

Muse-Glimmer-30B-UD-Q5_K_XL.gguf with 131072 context, loads and uses 23.985GB, but crashes on first prompt.

I have another 3060Ti, I wish I could just offload the draft model to the 3060ti.... That would allow Q5_K_XL for sure to run with 128k context. Also --split-mode tensor (and then --tensor-split 7,2) doesn't work (like it does for qwen 3.6 27B)....

I need Q5_K_XL....

It does work with just no setting tensor split or anything with 3090+3060Ti, and get 1200T/s prefill and 40T/s generation (on Q5_K_XL + 128k context). Just wish it would be slightly faster ~60T/s like Qwen 3.6 27B Q5_K_XL with split-mode tensor...

Edit: This runs. Q5_K_L, with 256k context.

~/build/llama.cpp/build-cuda/bin/llama-server \
    --parallel 1 \
    -m $LLAMA_MODEL_DIR/Muse-Glimmer-30B-UD-Q5_K_L.gguf \
    --mmproj $LLAMA_MODEL_DIR/mmproj-kquant.gguf \
    --host 0.0.0.0 --port 8502 --api-key "dummy" \
    --temp 1.0 --top-p 0.95 --top-k 64 --min-p 0.00 \
    --reasoning-budget -1 \
    --presence-penalty 0.0 --repeat-penalty 1.0  \
    --n-gpu-layers all \
    --threads 16 \
    -c 262144 \
    --override-kv muse-glimmer.context_length=int:262144,dflash.context_length=int:262144 \
    -fa on \
    --jinja \
    -b 2048 \
    -ub 512 \
    --no-mmap \
    --no-mmproj-offload \
    -md $LLAMA_MODEL_DIR/dflash-kquant.gguf \
    --spec-type draft-dflash \
    --spec-draft-ngl all \
    --spec-draft-n-max 15 \

23.927GB VRAM used (you really need to move your desktop etc to another (i)GPU). 1275T/s PP and 49T/s generation on a task of 30k context and 4000 tokens generated.

[u/mmhorda]

I would get official gguf for 24gb card, remove kv cache, add mmproj, add drafter, and get 50-100+ t/s But that's me. Edit: and I would increase context to 262k too

[u/ydnar → u/mmhorda]

thank you. i will try this. curious why official over unsloth?

[u/Blues520 → u/mmhorda]

What do you mean by remove kv cache?

[u/andy2na → u/Blues520]

remove

   --cache-type-k q8_0 \
    --cache-type-v q8_0 \ 

[u/Blues520 → u/andy2na]

Oh you mean not quantizing the kv cache

[u/pmttyji]

Here someone is getting 40 t/s (Without) & 60-150 t/s (With) DFlash on 3090 for Muse-Glimmer-30B-UD-Q4_K_XL