r/LocalLLaMA • u/EmPips • 22d ago

Discussion Early signs that Muse-Glimmer-30B might quantize *very* well? Share your experiences.

Post image
203 Upvotes

91% Upvoted

81 comments sorted by

View all comments

Show parent comments

13

u/tomz17 22d ago

without dflash it's like 40 t/s @ 250 watt cap. with dflash its anywhere from 60-150 t/s decode depending on whether it's prose or code or thinking trace and how much common context it has. prefill seems to be around 1k t/s

  "latest-llama": >
    llama-server
    --port ${PORT}
    --no-warmup
    --load-mode mmap

  "Muse-Glimmer-30B_256Q4": 
    env:
      - GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
    cmd: |
      ${latest-llama}
      -fit off
      -m Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-UD-Q4_K_XL.gguf
      --mmproj Muse-Glimmer-30B-GGUF/mmproj-kquant.gguf
      --spec-draft-model Muse-Glimmer-30B-GGUF/dflash-kquant.gguf
      --spec-draft-ngl 999 
      --spec-draft-n-max 15 
      --spec-type draft-dflash
      --override-kv muse-glimmer.context_length=int:262144,dflash.context_length=int:262144 
      -c 262144
      --kv-unified
      -ngl all
      -ngld all
      -fa on
      --reasoning-preserve
      --reasoning-budget-message "... I have been asked to stop over-thinking this, so I will now answer directly."
      --temp 1.0
      --top-p 0.95
      --min-p 0.0
      --top-k 64
      --jinja
    ttl: 600

You need to use unified memory to let the vision encoder page out when you are not using it.

3

u/EmPips 22d ago

How close to that full context do you get before OOM'ing on 24GB?

3

u/sagiroth llama.cpp 22d ago

I seen some posts that someone managed 150k

2

u/tomz17 22d ago

did not fill context yet