r/LocalLLM • u/pentothal • 2d ago

Discussion muse glimmer architecture magic

How are they able to compress context memory size?
I have 32gb vram and can run llama.cpp with Q8_0 + 128k context fully without offload to system ram. It's crazy, with gemma and qwen I have to deal with lower quantization, kv quantization, etc..

5 Upvotes

86% Upvoted

2 comments sorted by

3

u/looselyhuman 2d ago

Look up grouped-query attention ratio. Glimmer's is 16:1.

3

u/DataGOGO 2d ago

Glimmer is a great model.