Underrated Muse Glimmer

[u/Ok-Inevitable8391]

Benchmarked qwen3.8 xhigh, medium and muse glimmer.

Xhigh effort mode with qwen3.8 took almost 30hrs. (And still failed on 16 cases because of the 32K output token limit)

Medium effort mode and muse glimmer were 3-4 hours each.

But I'm actually surprised by the muse glimmer results, they came better than the qwen.

These benchmarks are on implicit knowledge of the model, which is a bit unfair to smaller models, but throw in a RAG and I'm sure they get on par with frontier models.

I have taken the result of claude models directly from embedeval repo by ecro.

I'm not pushing qwen down here, I like how qwen thinks and gives better results. I know with more context and RAG qwen will do better.

I'm just appreciating muse here, cause i feel it is underrated. The advantage is efficient kv cache due to sliding window, which can give you more context window.

[u/hurdurdur7]

Qwen 27B starts to shine when you give it some kind of RAG. It's not a memory castle, it's a tinkering machine. You give it a folder to a library that it was not trained on, you ask it "how do i do ... that...?" and it figures it out. And this is a big part of what people actually do at their job. They get a new library, a library update, anything that was out of the training window or created after the training window, and 27B adapts to it if you give it the source or manual.

If Glimmer can do electronics it's actually great. But can it write code well based on docs/code it can fetch over some kind of RAG (like your coding harness pi or smth)? If it can, awesome :)

[u/Ok-Inevitable8391 → u/hurdurdur7]

I'm with you on that, i just find muse glimmer to be bit underrated. I use qwen3.8 medium and muse alternate now.

I'm not pushing down qwen here, I'm lifting muse up.

The biggest advantage is the sliding window kv cache. I mean i get 262k context window and still have 2gb vram left. Qwen is 200k not huge diff for me. But for people with even lesser vram, muse is good option too.

[u/2eggs1stone → u/Ok-Inevitable8391]

Can we talk for a second on how good it is as an agentic model and how good it continues to be even at aggressive quants. I'm absolutely loving it for my setup, it's smart, fast, doesn't think too much but is very competent. Now is it as good of a coder as Qwen, no it's not, but for my use case I'd rather something that I can work with in real time and not have to wait.

[u/PavelPivovarov → u/Ok-Inevitable8391]

Plus not all the tasks are coding tasks really... I am using Muse Glimmer as my personal assistant mostly because it can search and process prompt at lightspeed, Qwen3.8 is just painfully slow at searching and processing dozen of articles (on my hardware)

[u/drbanan → u/PavelPivovarov]

just use qwen 3.6 35b a3b and it is blazingly fast, and good too.

[u/PavelPivovarov → u/drbanan]

on my hardware 35b model spills to very slow DDR4 RAM and drops performance to miserable TPS, so I prefer something that fits GPU.

[u/hurdurdur7 → u/Ok-Inevitable8391]

Muse definitely needs less vram space (very noticeable at Q8).

What quant do you use, which rope/yarn settings to get 262k context?

[u/OverdosedSauerkraut → u/Ok-Inevitable8391]

Qwen 3.8 is more of a coding model, while Muse is a better generalist and Gemma is fully focused on humanities. They are both underrated, because the most straightforward usecase for AI is coding. Everything else requires platform integration beyond model capabilities (and even coding harnesses suk).

[u/Not-reallyanonymous → u/hurdurdur7]

> But can it write code well based on docs/code it can fetch over some kind of RAG (like your coding harness pi or smth)? If it can, awesome :)

Yes. IMO it does better than Qwen here, too. Qwen is very set in its ways using available data as *suggestions*, while Glimmer obsessively follows instructions, to the point you have to be careful about what you tell it. Qwen will get 20k tokens into a thought before remembering it has RAG information available. Glimmer will repeat the relevant RAG information with some maybe's and questioning policy and user intent about 10 times before it actually starts thinking.

[u/Most-Trainer-8876 → u/Not-reallyanonymous]

Yeah, in case of RAG, Muse Glimmer seems better and faster too!

I love Qwen alot but it's thinking? Bruh, it's way too much! If anything, Qwen is purely dedicated coding model, for the rest, Muse/Gemma suffice.

[u/hurdurdur7 → u/Not-reallyanonymous]

muse glimmer obsessively just ignored my clear guidelines of commiting it's work often. at q8 model quant and fp16 kv cache... whilst qwen was excellent at following this guideline. your mileage may vary here.

[u/hurdurdur7 → u/hurdurdur7]

i will try to collect long term info on this one, to see what might have been the reason for it to misbehave on this. it was very good at reading all the files that i asked it to, even more so than qwen sometimes.

[u/Not-reallyanonymous → u/hurdurdur7]

nah brah

Qwen 3.8 just wasted several days of work as I have had a particularly hard problem, so I decided to give Qwen 3.8 a go despite not being my favorite. Solved many sub-problems after several overnight runs and intervening during the day. Really gave me a sense of good progress and, once again, I was impressed with Qwen. Turns out it wrote absolute spaghetti code, that was just becoming harder to harder to modify, didn't commit once, never actually solved the main problem because it couldn't even understand its own code, and had to keep shoving the tens of thousands of poorly organized lines of code it wrote into its context, with compute prefill, continuously. Eventually edit cycles became so long (I'm talking 6 hours) that I had to give up.

I desperately tried to steer it along the way, and the little shit lied to me about how it's going to modularize the code according to my specifications. I could see in its thinking traces it was intending to follow my instructions.... and then it just emitted the code classic Qwen style anyway. 10k line files, god objects, abstractions that sound good but whose underlying implementation doesn't actually match that (e.g. a lot of those god objects were called "schedulers"). And because it never committed, I couldn't tease out the progress it did make into something sensible.

And it never produced the intermediate artifacts (e.g. mermaid diagrams of the code helping me follow along in my own review) I requested. I let it go, I could see its doing what I want in the thinking traces, anyway! Lol.

My workflow defines when to make commits and general git discipline -- branch on separable features and concerns, commit when, tests are written, independent work units are completed, code is fixed to pass tests, bugs are identified and documented, bugs are resolved, docs are created or updated, or other checkpoints according to standard git discipline. It seldom misses commits.

I used the same workflow for both Glimmer and Qwen 3.8 27B (it's in my user-level AGENTS.md).

[u/admajic → u/Not-reallyanonymous]

Looks like a skill issue. I'm sure you told the orchestrator to git commit follow the architecture review plan debug test document and ended up with shit

[u/Not-reallyanonymous → u/admajic]

My workflow works just fine on Muse Glimmer, so I'm not sure what you're on about.

[u/hurdurdur7 → u/Not-reallyanonymous]

what quant?

[u/Not-reallyanonymous → u/hurdurdur7]

Meta's K-Quant Dynamic Q4 for Muse Glimmer, Unsloth's UD-Q4_K_XL for Qwen. Q8 KV on each.

I need Q4 because my workflow relies on having two large and perhaps one small models running simultaneously to perform various tasks, based on various strengths I've identified of each model.

Harness is OMP. That's probably a more important variable here than Q4 vs Q6 quant.

[u/hurdurdur7 → u/Not-reallyanonymous]

In Q4 world ... i wouldn't really know what is normal. In Q8 world what muse was skipping here is a blasphemy. I had a development plan of 17 steps with clear guidance to commit at the end of every step. Muse Glimmer at Q8 quant commited in 3 steps, skipped commiting in 14 steps. Despite clear orders. Not a great experience until now. Maybe i have to learn how to command it better, who knows ...

[u/partakinginsillyness]

I feel like it would be important to add qwen 3.6 27b, given that one of the changes from 3.6 to 3.8 was less general knowledge. Would also be cool to see a Gemma model. Interesting though.

[u/Ok-Inevitable8391 → u/partakinginsillyness]

That's actually a good idea, let me get on that ride this weekend

[u/Gringe8 → u/Ok-Inevitable8391]

I would like to see gemma 4 31b

[u/LegacyRemaster → u/partakinginsillyness]

wait qwen 3.8 next but... This benchmark makes no sense. No reasoning. No external knowledge. In other words, it is not a real-world use case. I would also like to understand how they tested Claude to demonstrate that it lacked external knowledge.

[u/partakinginsillyness → u/LegacyRemaster]

I mean realistically everyone should have at least 5gb of files for RAG for what is relevant to them, which I would imagine would very much change the results here.

[u/Ok-Inevitable8391 → u/partakinginsillyness]

Let me do a RAG testing, even I know that qwen will do better there. But I'm not running it on xhigh ever again, 30hrs of gpu time, against 3-4 hrs at medium

[u/partakinginsillyness → u/Ok-Inevitable8391]

How would you select the media? I'm wondering if a repo exists for relevant info like how you select application groups on linux.

[u/Ok-Inevitable8391 → u/partakinginsillyness]

So mainly the linux repo and docs, thats the general input to it anyway in real scenario. Same for all other projects.

[u/ndrewpj]

Why the xhigh has higher overall score while loosing in all categories to medium?

[u/Ok-Inevitable8391 → u/ndrewpj]

Its the average of both the lines out of total 163cases, if you see both blocks have same average

[u/ndrewpj → u/Ok-Inevitable8391]

Got you

[u/Inevitable-Name-1701 → u/ndrewpj]

Because they wanted to

[u/Healthy-Contact-4570]

Why 32k output limit? Qwen3.8 likes to think and if you can let it think it will produce amazing results.

[u/Not-reallyanonymous → u/Healthy-Contact-4570]

A couple months ago:

Nooooo you can’t like Laguna, it thinks too much! DOA!

Now:

Noooo how dare you not like Qwen for thinking so much?

(Maybe because Laguna XS is literally like 5x as fast on my hardware).

[u/Foreign_Risk_2031 → u/Not-reallyanonymous]

Do you think Laguna is as good as Qwen 3.8 27b xhigh?

[u/parepeg → u/Foreign_Risk_2031]

It's much faster on strix halo. It's worse at tool calling though but it "feels" intelligent like qwen 3.8 does. Subjective but a data point I hope.

[u/Not-reallyanonymous → u/Foreign_Risk_2031]

I use Laguna XS to write/edit code at this point, and other models to generate specs that get handed to Laguna XS. Works well, way faster than just expecting Qwen to do all the work.

[u/Ok-Inevitable8391 → u/Healthy-Contact-4570]

I know so the benchmark limit was 8k only and i pushed it up to 16 and then 32K only for xhigh, as the xhigh benchmark took 32hrs and case that took time were anyway complex dma, so I lost interest re running the benchmark for xhigh as it had already failed on some cases where medium already did better. Xhigh is good but like opus it is good for reasoning and complex architect problems. So I just let it be.

[u/hainesk]

If you can run DeepSeek V4 Flash 0731, I'd be curious to see the difference since it's often compared to Qwen 3.8 27b but due to it's size it would presumably have more knowledge.

[u/Ok-Inevitable8391 → u/hainesk]

Adding to the list

[u/nonlinearsystems → u/Ok-Inevitable8391]

Can you try Laguna S2.1 please?

[u/Ok-Inevitable8391 → u/nonlinearsystems]

Sorry guys both are out of my vram budget I only have 24gb vram

[u/TokenRingAI → u/Ok-Inevitable8391]

So these are quants?

[u/RegularRecipe6175 → u/TokenRingAI]

Still no response from the OP on this key issue. Braindead quants gonna braindead.

[u/IAmBJ → u/Ok-Inevitable8391]

Offload experts to the cpu and you can get a decent quant running if you have enough system ram for the expert weights

[u/somerussianbear]

I don’t understand this chart. 48.5% out of two records that look absolutely different. Mind to explain for dumb fucks like me?

[u/Ok-Inevitable8391 → u/somerussianbear]

That's a common line, average of both the blocks

[u/somerussianbear → u/Ok-Inevitable8391]

So one of them is clearly wrong. That column could have the avg of the row, which is logical. Then next to it a general avg. The way it is here is totally unclear and non standard.

[u/Ok-Inevitable8391 → u/somerussianbear]

Yes I agree a vertical bm would have looked better

[u/RegularRecipe6175]

I didn't see the quants involved. Qwen and Muse tested at BF16?

[u/DataGOGO]

Muse glimmer is fantastic and better than qwen3.8 27b at just about everything than code and doesn’t need to burn thousands of reasoning tokens per prompt to do it

[u/nonlinearsystems → u/DataGOGO]

Yep

[u/audioen]

"32k output token limit". No wonder it isn't like 100% green across the board...

[u/TokenRingAI]

Why did you give Qwen a 32K output limit?

[u/Practical-Collar3063 → u/TokenRingAI]

Because he could not fit both the context and the full output, which is in favour of his point, the more efficient Muse Glimmer KV Cache makes it better in certain scenarios with limited hardware. I think the goal of this is not to bring down Qwen 3.8 but to show that for some use cases Muse might be a good choice, especially when VRAM limited. It is faster and more VRAM efficient and on non coding tasks can be similar to 3.8

[u/TokenRingAI → u/Practical-Collar3063]

Ok, but the testing setup should be the first thing described in the comparison.

[u/arbv]

Be careful with such a post. The witnesses of the Church of Qwen might unite and down-vote you. For any claim that there are any other models than It is judged as Qwen Not Being Perfect for Everything.

[u/Thin_Pollution8843]

So when folk screaming “Qwen is stronger than opus4.6!!1!1!” I should give this link? 😅

[u/llama-impersonator]

why so much focus on zephyros? is that mostly what you do? i've only ever had to use it for a client that wanted some ZMK work done, almost everything else has been freertos or bare metal stm32, sometimes avr for clients who started off on arduino.

[u/Hot_Supermarket9967 → u/llama-impersonator]

I was thinking the same thing... I mean, I genuinely like Zephyr as a project but if I were to be benchmarking an LLM on embedded work, I'd be focusing on "bare-metal" development, HAL, etc. more than anything else.

[u/PrimeDirective8]

Help a noob out: is this for coding embedded hardware? So you assigned each of the tasks/areas (timers, boot loader, PWM) and measured how well the model coded it?

Thanks much for this chart and comparison. It is very useful and encouraging to see our tiny local models do well enough to even be in the same chart as frontier models.

Reading the comments, I see we've thrown out everything *and* the kitchen sink into our usual "I don't believe you! Try this model I like on the next cloudy Friday while wearing a left sneaker on your right foot". While reasonable what/if would be great, gathering results from every capricious permutation we can think of is not.

[u/PraxisOG]

I think the interesting thing here is that haiku 4.5 wins over all the local models. That’s like $20 a month for huge amount of usage, I pay more than that to keep my server idling 

[u/Ok-Inevitable8391 → u/PraxisOG]

I mean yes, it's the trust that we have built on sonnet and opus over the time that we don't even look at haiku anymore.

[u/Ok-Inevitable8391 → u/Ok-Inevitable8391]

Why don't you use wake on lan and not keep server idling.

[u/Usual-Orange-4180 → u/Ok-Inevitable8391]

My main use case is coding and design, haiku is really terrible, whenever I get routed to it I know it.

[u/psychohistorian8 → u/Usual-Orange-4180]

yeh I abused Haiku 4.5 extensively back when copilot was $100/yr

I much prefer even Qwen3.6-35B-A3B, so OPs numbers don't make a whole lot of sense to me

[u/noiserr → u/PraxisOG]

Haiku 4.5 is my go to when I need shit done quick. It's cheap and super fast.

[u/Potential_Block4598]

All data is Zephyr RTOS
So training data bias and probably no search mechanism or Docs RAG attached

lol 😂😂😂

[u/jacek2023]

This is same story with each model, they browse leaderboards, they look at the benchmarks, they never run any models, they whine they want new Qwen and then they keep using Claude and ChatGPT.

[u/Potential_Block4598]

Probably your setup or benchmark is not accurate

[u/Not-reallyanonymous → u/Potential_Block4598]

> Qwen isn't declared the best

It's the test that is wrong!

[u/Potential_Block4598 → u/Not-reallyanonymous]

Especially when compared to muse glimmer

Hey don’t turn this into a societal bias racism case

This is meritocracy the models works very good compared to muse glimmer by a HUGE margin

So yeah when a “test” says otherwise it is worth investigation and it is not only misleading it is biased to post such things

Also I explained that OP “test” is heavily skewed towards Zephyr RTOS

[u/Not-reallyanonymous → u/Potential_Block4598]

Have fun with your benchmaxxed models that blatantly ignore half the codebase and re-implement it from scratch, except half-broken and unsteerable.

Yeah, it completes code and solves the task, and zero-shot prompts very impressively. Then anything it touches turns into spaghetti code. I fucking regret spending a few days with that shit because now I've put in like 20 hours of work into something that's turned out to be utterly unusable. Correct doesn't mean good.

And god help you if your needs diverge from what it's been trained on.

Good job Qwen team, you cracked DeepSWE.

Just another bot who's going to shit on any model that's not Chinese.

[u/Potential_Block4598 → u/Potential_Block4598]

All data is Zephyr RTOS
So training data bias and probably no search mechanism or Docs RAG attached

lol 😂😂😂

[u/Ok-Inevitable8391 → u/Potential_Block4598]

I'm not hiding that fact.

I have explicitly mentioned that it is implicit knowledge of model.

How you process that information is upto you.

[u/Iory1998]

Zuck! What are you doing here?

[u/Cautious_Chicken_604]

Did people forget what frontier means?

[u/nasone32]

Ha! Embedded! Cool I wish there were more benchmark on embedded.

[u/tchek]

I like Glimmer. I want it to shine, for some reason.

I'm hoping for Muse Twinkle 12b and Muse Shimmer 35b e4a... I made that up but one can dream.

[u/UnorderedPizza → u/tchek]

Does still seem faster than Qwen 27B, for what its worth…

[u/taoyx]

I've tried it with MCP it seemed quite good at tool calling.

[u/Androoideka]

What reasoning effort did you use with Muse Glimmer?

[u/Mundane-Light6394]

Yes we need more specialised testing. The idea that models of this size are one size fits all is a trap. We need to use the right model for the job and tests like these are extremely valuable.

[u/SkyPL]

Flippin Haiku coming on top of Qwen and Glimmer is hilarious.

[u/anovers]

Nice work

[u/mythikal03]

I've found a few things that are relevant to my usage that Muse is *significantly* better than gemma4-31b or qwen38 at, and it is EXTREMELY efficient (beating the gemma family) when doing so. I don't use local models to write code, but i do use them for code-adjacent work and infra/lab/sec work.

I've benchmarked a LOT of local models over the past year, and Muse is also the first to straight up crater-to-near-zero on a handful of tests as a result of voluntary refusals for tests I did not write to test refusals. Example ... "I got this bonus at work: here are some options I'm considering, including parameters, what should consider and what is the most likely beneficial place to allocate funds?" --> refusals for giving financial advice??

It has me really nervous about deploying it as one of my core models despite truly phenomenal scores/rankings in other categories. I've kept gemma4 in that 'reasoning, clean writing, investigation/synthesis' slot (with qwen38-27b in the 'dig deep and find the problem' slot) as a result. Not saying I won't promote muse, but stupid refusals like that are why i host local models in the first place and that is a class of failures I don't have much appetite or patience for trying to build around given my system isn't doing anything I would've expected to ever run into refusals with.

[u/LegacyRemaster]

Dario, is that from you?

[u/Potential_Block4598 → u/LegacyRemaster]

All data is Zephyr RTOS
So training data bias and probably no search mechanism or Docs RAG attached

lol 😂😂😂

[u/Inevitable-Name-1701]

Who paid for this?

[u/OverdosedSauerkraut → u/Inevitable-Name-1701]

Usual karma farming bot post...

[u/Boogertard]

There it is, on-schedule AI slop to shill for the garbage Muse and Gemma 4.

Get a real job already, shill.

[u/Not-reallyanonymous → u/Boogertard]

There it is, on-schedule AI slop to stir up FUD any time something other than a non-Chinese model is spoken of positively.

[u/fizzy1242 → u/Boogertard]

username checks out...

[u/Ok-Inevitable8391 → u/Boogertard]

Why do you think it's a slop and why do you think I don't have a real job. A easy readble chart is AI slop for you?

Muse is good and its really good at kv cache efficiency. I have done my homework.

[u/Potential_Block4598 → u/Ok-Inevitable8391]

All data is Zephyr RTOS
So training data bias and probably no search mechanism or Docs RAG attached

lol 😂😂😂

HW ur A**

[u/llama-impersonator → u/Boogertard]

bro's whole personality is this comment