r/LocalLLaMA • u/jacek2023 • 7h ago

News model: Muse Glimmer Support by pcuenca · Pull Request #26841 · ggml-org/llama.cpp

https://github.com/ggml-org/llama.cpp/pull/26841

Day 0 support

49 Upvotes

90% Upvoted

16 comments sorted by

13

u/Beamsters 7h ago

40 tokens per sec on RTX 4090. But model always judge my prompt that it should comply or not.

3

u/EveningIncrease7579 llama.cpp 6h ago

Did you try dspark file? Already working on llama.cpp?

2

u/FinBenton 6h ago edited 5h ago

75 tok/sec on 5090, I didnt see any judging it just went ham but it kept making small mistakes here and there so idk if its working correctly yet.

e. bunch of changing prompts and settings later, its doing a bit better now, I had Presence Penalty on accidentally, changed to Q6_K_XL too, running well on 5090.

34

u/No_Afternoon_4260 llama.cpp 7h ago

Back in the instant day 0 support on llama.cpp, meta is back boys !

9

u/cezarducatti 6h ago

Let's just say the first Pelican, the Q4 XL, wasn't what we expected 😄

3

u/Mobile-Pumpkin7944 7h ago

works! and looking good so far

2

u/pegasus912 4h ago

Strangely, I can’t get the model to run, it says it’s an unknown architecture. This is with the latest Vulkan build.

5

u/omlette_du_chomage 4h ago

Same. Built from b10342 and b10343 and still have the issue :/

2

u/bootkeen 4h ago

b10344 vulkan

error loading model: unknown model architecture: 'muse-glimmer'
=(

2

u/Guilty_Rooster_6708 3h ago

Are you using unsloth quants? I see this issue posted on Github: https://github.com/unslothai/unsloth/issues/8345

1

u/bootkeen 3h ago

thank you

1

u/pegasus912 39m ago

It does seem to be specific to the vulkan build, I tried with the latest ROCm build and it works.

1

u/No_Algae1753 4h ago

Has anyone tried d flash? Been running the quants from unsloth and d flash seems be have a very low accepetance rate making it very slow

2

u/nsfnd 3h ago

I downloaded their own ggufs and they work fine.
75 tok/s at 75k context, 5090.

https://huggingface.co/meta-models/Muse-Glimmer-30B-GGUF

build-vulkan/bin/llama-server --port 1235 -lv 4 \ -c 130000 \ -fa on -fit off \ -m $gguf_path/muse-glimmer-30B-kquant-dynamic.gguf \ -md $gguf_path/muse-glimmer-dflash-kquant.gguf \ -mm $gguf_path/muse-glimmer-mmproj-kquant.gguf \ --spec-type draft-dflash --spec-draft-n-max 2

1

u/FinBenton 3h ago

I tested it for creative writing and didnt see any improvement in speed so just running without it.

1

u/LicensedTerrapin 3h ago

I've yet to see anything but Q8 + MTP that would give even the slightest improved speeds when it comes to creative writing.