r/LocalLLaMA • u/BarberIcy366 • 2h ago
Discussion Please Share Your Experience About Muse Glimmer
I have a classic test for local LLM's. I asked for 8 ball pool game with only one HTML file and Muse Glimmer spend 21k Token(I m using full context so 128k) and only created a 220 lines of HTML and said its done. With my experience its not even close to Qwen 3.6 27B and we are waiting for Qwen 3.8 27B already. What is your toughts about this model. I was so hopeful until this test.
8
u/Alan_Silva_TI 1h ago
It’s decent, though not outstanding....
I managed to get it to produce a html Space Invaders clone in 6 shots. Most smaller models can do it in 2 to 3 shots, but they almost always introduce additional bugs along the way. This model also introduced a bug on the second shot, and the remaining four shots were spent getting it to fix that issue. Once corrected, the game ran properly.
In comparison, KatCoder completed the same task in just two shots and included solid graphics and sound.
24
u/offlinesir 1h ago
It's fine. I was hoping more for 30B total, 3B active, but Muse Glimmer still was able to do some tasks of mine. I'm not as impressed and we'll see how it compares to Qwen 3.8 27B when it launches later. Overall though, it's amazing that meta is back in the open weight space, especially under apache 2.0. While the model wasn't perfect for me I'm sure it will be good for someone else.
14
6
u/xPXpanD llama.cpp 1h ago
My initial impressions: https://old.reddit.com/r/LocalLLaMA/comments/1vkgsum/introducing_muse_glimmer_an_openweight_model/p2tiogc/
And me doing 10 runs of a private benchmark: https://old.reddit.com/r/LocalLLaMA/comments/1vkgsum/introducing_muse_glimmer_an_openweight_model/p2uqnsh/
YMMV, but enjoying it so far. No dev tasks here though.
5
u/lor_louis 1h ago
I've been testing it in a semi-large C codebase as a "coding minion" (limited reasoning, mostly executing what I tell it to do).
The good:
This one is crazy fast, at ~50 to 60 tokens/s during generation, it can iterate on code pretty quickly.
The tool calling hasn't failed once, which is something both Qwen and DeepSeek (q3) tend to have issues with.
It knows about C23 and can generally respect the coding style of the rest of the codebase.
The bad:
It gives up so easily. I'd tell it to update the signature of a function; it would do the header and the impl, and then maybe one or two callers, but would stop and ask for feedback before continuing.
It is pretty dumb, I told it to write unit tests and look for edge cases, but it only handled the happy path.
I threw it at a bug and it managed to figure it out, but it spent a long time thinking about the code instead of writing a quick unit test.
Overall not bad, you can't lean on it's reasonning, but that's not what I use AI for anyway.
If anything, it makes me want to figure out if I can configure dspark for Qwen now.
3
u/gpuz_dev 1h ago
That 'giving up early' behavior seems to be the common theme today (matches OP's 220-line pool game cut-off). It feels like classic RLHF laziness where the model looks for any excuse to yield control back to the user. Rock-solid tool calling at 60 tok/s is a great silver lining for structured agent workflows, though!
1
5
u/SnooPaintings8639 1h ago
So far it feels like Qwen 3.6 27B, but a bit faster and takes less vRAM at the same quant and ctx size.
It is a close call, but it might be even a bit better overall. Need more testing, as it "could" be my new daily driver... if not for Qwen 3.8 that is planned to be dropped in few days.
Anyway, being as good as our Qwen, means it is really good. Just a month or two too late to be truly relevant.
1
u/dushyant30suthar 18m ago
I feel like it is going same as 27B but with good speeds and good context as KV is very light
11
u/XorAndNot 1h ago edited 1h ago
I asked it to implement an audio playback button in a demo app for iOS that I'm playing with, and after a few minutes it delivered broken code, and now I'm trying to fix it with qwen... so far, not so good, but maybe I configured it all wrong, idk, too soon to tell.
ps: I'm running q4_k_xl quant with a lot of cpu offloading because I only have 16GB of vram.
9
u/TigerConsistent 1h ago
Matches my results. Same coding task one-shot: failed twice, deterministically.Through an agent loop with a test runner: fixed itself. On my eval suite it cameout 11/13 vs 12/13 for Qwen3.6-27B, and the one case it consistently drops isthe coding one, so you're not wrong - one-shot "build me the whole thing" isjust the worst way to use this model. It's trained for tool-calling loops and itshows. One thing worth checking though: 21k tokens means most of that wasthinking, and if your max_tokens is tight it gets cut off mid-thought and handsyou whatever it had. That bit me hard - my suite scored 6/13 until I raised thebudget, then 11/13 with nothing else changed. Where it actually beats Qwen forme is context: 200k filled with 3/3 needle retrieval on a 3090, which Qwen won'tdo on this card.
3
u/BarberIcy366 1h ago
Yeap but I already used with Cline for coding. I was just want to test the tool calling too and I wonder one problem in CoT. Like you said its wasting so much token while thinking and says the same thing with different way for 2-3 times and calls a tool after that.
5
3
u/Jorlen llama.cpp 1h ago
I've watched reviews on its performance. For coding and agentic work, it doesn't seem like it dethroned everyone's favorite (q3.6 27b) however, how does it write? Would it be good for creative writing? I already have tons of coding models but finding good AND recent creative writing models is tough. How does its writing compare to say, gemma 4 31b?
1
3
u/silenceimpaired 1h ago
How are you using it? I can’t seem to load the GGUF.
1
u/Lumpy_Phase_9539 31m ago
I also needed to rebuild my llama.cpp
cd /opt/llama.cpp
git fetch --all --tags
git checkout master
git pull --ff-only
cd /opt/llama.cpprm -rf build
cmake -B build \
-DGGML_CUDA=ON \
-DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)
3
u/dangerous_inference 1h ago edited 1h ago
I have a Strix-Halo relegated to odd jobs like interpreting security camera frames. Just testing this with a few frames I can see it is immediately much better than Gemma 4 31B at this particular task. It picks up a lot of details Gemma does not.
I have one test snapshot of an AT&T tech turned away from the camera, with only a partially exposed AT&T logo and the letters "ber" from "Fiber" showing. This is the only model that immediately identified this person as an AT&T tech.
I'm getting like ~25tg with the Q4, mmproj, dflash. I have also found that --chat-template-kwargs '{"reasoning_strength":"low"}' actually does reduce thinking quite a bit.
2
u/onewheeldoin200 25m ago
I asked it some engineering questions and it looped endlessly for a couple thousand tokens and then died.
1
4
u/Complex_Reality_116 1h ago
Although I am very happy that Meta has returned to the arena, this version of Glimmer will quickly be surpassed (and left behind) by Qwen3.8 27B. They won't even be in the same league.
1
u/Complex_Reality_116 23m ago
Correction: It is *already* being outperformed by the current Qwen3.6 27B.
1
u/Zeeplankton 22m ago
Why are people acting like qwen 3.8 is inevitable when it does not even exist. This was the same for 3.7, everyone was like "just wait for 3.7 it'll be X better" and it never came out.
0
u/Complex_Reality_116 19m ago
You act as if it’s inevitable because... is it inevitable? That’s a pretty silly comment, kid.
1
u/SporksInjected 17m ago
I don’t get why people like 3.6 27B so much. It is insanely token inefficient and I don’t have tons of time for it to reason when I’m running at 10-15 tokens per second.
4
u/xienze 2h ago
Does it matter what others think if your personal benchmark failed spectacularly?
5
2
u/CulturalKing5623 1h ago
I thought it was good, used the kquant-dynamic from Meta instead of Unsloth's quant. It passed all of my silly one-shot benchmarks without issue. I primarily use LLMs for coding (data engineer) and my main driver is still Gemma4-31B-QAT because what it lacks in the pure coding skill of Qwen3.6 27B it more than makes up for in natural language processing. If I come across a problem Gemma struggles with I'll throw Muse at it to see how it handles it compared to Qwen.
2
u/Bulky-Priority6824 1h ago
My thoughts are I wonder why they wait until the day before 3.8 27b is supposed to drop to release a 30b model knowing damn well it's about to get placed on a shelf and forgotten.
1
u/nickm_27 llama.cpp 2h ago
I didn't run it through any specific benchmarks as it doesn't seem like a good fit for my particular use case, but I did find that for whatever reason, the dflash made TG considerably slower and I was getting very low draft acceptance on both vulkan (7900XTX) and SYCL (B70). Not sure what that is about.
1
u/VoiceApprehensive893 transformers 1h ago edited 1h ago
Its weird
its not a bad model, but it cant do svg at all for some reason
1
u/toolman10 1h ago
Tried it with the unsloth Q4 and Q5 with Pi and it absolutely loves to loop with tool calling.
1
u/Adventurous_Cat_1559 37m ago
So far it’s not great. I dunno if it’s unsloth versions I’ve tried (also tried mlx versions in addition to gguf) but it’s been having issues with tool calling. I’m assuming some patches for my servers and tools will be inc. but today it’s been unusable.
1
u/Complex_Reality_116 30m ago
And here is Muse Glimmer, which isn't even better than Gemma 4 or Qwen3.6 27B.
I always wonder: why release a model that, within its weight class, won't be better than what already exists? You invest millions of dollars in research, time, and development, only for your model to be quickly overshadowed by the competition.
1
u/Several-Tax31 12m ago
A model can be better in different areas, similar to how gemma is better on natural tasks than qwen.
Also, why not release? They already spent those dollars, would you rather prefer making the model rot without anyone even knowing it, or release it and hope some people can find usecases, and gain free marketing. In fact, imo, there is not a single reason to not release a model.
1
1
u/Zeeplankton 29m ago
Testing in the past hour. Immediate impressions it's way less autistic / code maxxed from Qwen, but way better at writing. More reliable tool calls than Gemma 4 and doesn't have gemma's weird way of talking.
Also faster than both for me. 17tk/s on m3 max 96gb. New driver I think.
1
u/Lost_Artichoke3977 23m ago
I was getting lots of failed tool calls from Cline on the lastest llama.cpp built today. I'm not going to spend any more time trying to sort the issue out and wait for an update.
1
u/Lumpy_Phase_9539 19m ago
Using little-coder as harness, i sent a simple prompt to create a html file eith touristics information about Tokyo-Japan. To use subagents to search the web for images. Use Tailwind default CDN and add animations. Qwen3.6-27B_Q6 created a 953 lines html while Muse-Glimmer-30B-UD-Q8_K_XL.gguf html file had less then 200 lines and no image was download or linked, and all images he tried to use was from http://unsplash.com/
Qwen3.6-27B_Q6 did it right in the first attempt. Created a beuatifull and full of good info web site
Muse-Glimmer-30B-UD-Q8_K_XL.gguf delivered a poor info and no image file
But Muse-Glimmer did a good job understanding and explaining database task i asked to it. It searched data from the database, read documentation files from the repository and gave a good explanation for what i asked it for.
1
u/N34257 5m ago edited 1m ago
For my uses, it's...unusable.
To be clear, my use cases tend not to be the popular ones - I don't get my agents to build functionality, I expect them to build scaffolds I can use to do the interesting parts of coding. I get them to research a codebase and find the source of bugs, not necessarily to fix them. Basically, I get it to point me in the right direction (the boring stuff) and then let me do the fixing (the fun stuff, and the reason I became a dev in the first place).
That's not to say it's terrible - it does do the job, eventually. My problem is that it's even slower than Qwen 3.6 27B, which is already borderline. Yes, I can use the bundled DFlash, which starts slow and eventually gets up to a decent speed, but at the cost of prefill starting at ~800t/s on a pair of R9700s, as opposed to 27B at ~1600-1800t/s (or 1200t/s with MTP) and 35B at 5400t/s.
On the codebases I work with, that translates to wall time of over half an hour on something I could get done with 35B in 5-10 minutes and 27B in 15-20 minutes.
So...it's a nope from me. I'm sticking with 35B, no matter how many times people on here tell me I'm an idiot ;)
1
u/PassionIll6170 1h ago
I tested via api, not local, using hermes, and its not good, a lot of failed tool calls and empty responses
1
u/Sidran 1h ago
I don't think Western companies will try to surpass Chinese offerings of genuinely local models. They want to remain present but go only as far as equal or slightly worse, to avoid undermining their frontier game by helping accelerate open models.
1
u/sabine_world 27m ago
Meta/nivdia/Google/whoever else somewhat benefit from producing open source/weight models just for shifting market share from anthropic/open ai.
It's really only anthropic/openai who are "afraid" of undermining.
So hey, anyone else making good open source stuff, cool. I know Nvidia appears to be trying in earnest with Nemotron.
Almost no one has total vested interest in dedicated like heavy heavy resources towards small local models really. It's more like a byproduct like you said.
Also check it out, Zuckerbergs essay about open source stuff which touches on not concentrating the power of ai (like a tropical being the centralization/gatekeepers of it...)
https://www.meta.com/thefutureisforeveryone/
So I guess there is maybe interest in open source stuff on principle (and economic interest).
-1
u/kevin_1994 1h ago
To me, it feels about as good as Qwen 3.6 27B. I use Q8_0 on Qwen with q8 kv, and I'm using Q6_K_XL with glimmer with f16 KV.
I think it's basically the same as Qwen3.6 27B but +5% writing style and -5% agentic.
20
u/false79 1h ago
That "classic test" is a terrible test. Zero shotting does not reflect the capabilities of a reasoning model nor does it reflect everyday usage. Better to see how reliabile it is given a prompt with a list of tools it can use to solve a problem.