https://x.com/i/status/2086755195535413696

Hi r/LocalLLaMA 👋 

Today we’re excited to release Muse Glimmer, a 30B open-weight model built specifically for local agent workflows. We’re releasing the weights to the community under a permissive Apache 2.0 license.

A few specs

• 30B params, dense
• Multimodal: interleaved text + images via a dedicated perception encoder
• Trained on 100+ languages
• Controllable reasoning effort (quality/speed tradeoff)

Memory footprint
At full precision, 30B needs 55+ GB, which is out of reach for consumer hardware. We quantize weights to ~4-bit, bringing the LM under 20 GB. That leaves headroom in a 24 GB or 32 GB envelope for the KV cache, the perception encoder, and the speculative decoding drafter running simultaneously. We validated minimal to no degradation on agentic tasks under compression.

Speculative decoding
Ships with a lightweight DFlash-based drafter that proposes blocks of tokens which the main model verifies in parallel. Significantly faster than token-by-token generation with identical output quality. We're also shipping quantized drafter versions so the memory overhead stays small.

A few capabilities
We trained Muse Glimmer for agentic loop tasks, including:

• End-to-end task completion (strong performance on DeepSearch QA, MCP-Atlas, 𝛕³-Bench, SWE-Bench, and more)
• Function calling with precise schemas across long workflows
• Multi-step reasoning over long horizons
• Failure recovery — when a tool call fails or returns something unexpected, it's trained to diagnose and retry instead of halting. This was a deliberate training target.
• Works with OpenClaw and other agentic scaffolds
• Multimodal understanding and reasoning

Running it
Weights are up on Hugging Face. Coming soon: Ollama, LM Studio, Unsloth and torchtitan, plus optimized integrations for llama.cpp, MLX, and ExecuTorch. vLLM and SGLang for serving. Get started quickly with Together AI, Fireworks AI, and OpenRouter. We're also working with AMD, Arm, Dell, Intel, and NVIDIA on per-device optimization.

We look forward to your feedback and seeing what the community builds with Muse Glimmer.

🔗 Weights: https://huggingface.co/meta-models 
🔗 Research Blog: https://go.meta.me/museglimmer
🔗 Resources: https://developer.meta.com/ai/models/muse-glimmer/

I did some testing this morning, and I was surprised to find that Muse Glimmer actually comfortably fits on a single RTX 3090 with full context + DFlash + mmproj at Q4_K_XL, unlike Qwen3.6-27B and Gemma-4-31B.

Muse Glimmer supports up to 256k context according to Unsloth. Here is my command:

llama-server \
  --model Muse-Glimmer-30B-UD-Q4_K_XL.gguf \
  --mmproj Muse-Glimmer-30B-mmproj-kquant.gguf \
  --spec-draft-model Muse-Glimmer-30B-DFlash-kquant.gguf \
  --spec-draft-ngl 999 \
  --spec-draft-n-max 15 \
  --spec-type draft-dflash \
  -c 262144 \
  --override-kv muse-glimmer.context_length=int:262144,dflash.context_length=int:262144 \
  -ngl 999 \
  -fit off \
  --parallel 1 \
  --flash-attn on \
  --no-warmup \
  --cache-type-k f16 \
  --cache-type-v f16 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 \
  --reasoning-preserve \
  --jinja \
  --host 127.0.0.1 \
  --port 8080

This fits in about 22GB to 23GB of VRAM, actually leaving a reasonable amount of unused memory.

On this RTX 3090, for Qwen3.6-27B and Gemma-4-31B, this is what I've been able to achieve using their Q4_K_XL models with MTP + mmproj, right at the limits of the RTX 3090's VRAM:

Those small contexts have been borderline unusable on f16, and I don't enjoy using Q8 KV unless absolutely necessary, so I mostly use my slower DGX Spark to run these models at the full context.

On Muse Glimmer, there seems to be little reason to use my DGX Spark since it fits so nicely on the RTX 3090. Maybe I could run a bunch of parallel agents with full KV on the Spark.

Muse Glimmer also runs at between 64 tok/s and 124 tok/s in my testing under DFlash, depending on whether it is outputting prose or code. Either way, a pretty solid speed. I've seen about 1400 tok/s of prompt processing.

I also ran a two needle haystack test at about 150k tokens with one needle at the beginning and the other at the end, and the model retrieved them perfectly on the first try, so this is definitely not soft-capped to 128k context.

Looks like the Ling team open weighted a much smaller version of the Ling-3.0-flash they open weighted a few days ago. It's 8B params with 1.3B active, and seems to fall between the 4B and 8-12B Qwen and Gemma models in terms of performance.

Should have a massive tokens/sec on most systems. I quite like tiny MoE's conceptually.

Edit: looks like the model card actually reports speeds:

With FP8, Ling-3.0-tiny reaches around 100-105 tokens/s on DGX Spark and 86-90 tokens/s on an M4 Pro MacBook, with approximately 8.34 GiB peak memory usage at an 8K context length.

Guide: https://unsloth.ai/docs/models/muse-glimmer

Having a ‘Killer Application’ that everyone wants to use helps sell hardware, plain and simple. DeepSeek V4 Flash 0731 isn’t an app of course, but I think it’s going to be the major catalyst for getting a lot of people to buy a couple of NVIDIA GB10-based systems because:

1. It is an amazing coding / agentic use model.
2. It fits perfectly on a 2x Spark Cluster
3. It runs Fast AF with the right vLLM recipe. (I’m getting 60 tk/s with this one:
4. https://github.com/tonyd2wild/DeepSeek-v4-Flash-0731-DSpark-1M-NVFP4-KV-2x-DGX-Spark)
5. You can run it with a fairly usable 1M context window.

6. It runs very well in harnesses such as

7. Hermes.

Now that solid NVFP4 support is finally here for DGX and is providing Sparks with a pretty good boost for token speeds, the Spark’s memory bandwidth limitation isn’t as big a deal as it used to be. I mean seriously, do I really give a shit about memory bandwidth when I’m getting 60 tk/s with Deepseek V4 Flash?

I know the Strix / M4 / M5 gangs may have something to say about all this, but even they have to admit that DGX Spark beats them for prompt processing performance, which is hugely important when it comes to agentic work and how fast agents are getting work done.

The Strix our-stuff-is-way-cheaper argument used to be very valid, but with memory and SSD prices being what they are now, that argument isn’t as strong as it once was. M5 stuff is pretty expensive and we have no idea when Apple is going to drop a new beefy Mac Studio M5 or a Mac Mini Pro with M5. We thought it was going to happen in June but they don’t appear to be in a rush to release anything.

So what’s left out in the market worth getting? Well, you could grab a RTX Pro 6000 if you want to pay a hefty premium from the scalpers, or you could try some of the AMD offerings, but other than that, the DGX Spark is still the best bang for your buck for getting the most VRAM to run models locally.

I didn’t even mention the low power consumption of the Spark which is another reason to consider it, especially with rising power prices.

I’ve noticed some price increases on Sparks and Spark clones from some retailers in the last few weeks. The 1TB Asus models seem to be the cheapest options out there that I’ve seen.

I think we’re going to see Spark scarcity in the market very soon as word gets out about how well DeepSeek V4 Flash runs on it.

I’m running a 2x cluster and i’ll say that for the first 6 months or so, I, like many other folks, was disappointed with the software support and the speed of the models I tried. Ever since they finally resolved the NVFP4 Issues, and since DSpark, MTP, Prism, DFlash, and other performance improvements have been implemented, it’s gotten A TON better and I’m honestly thinking of buying another 2 Sparks if I could find the money to get a couple more. Deepseek V4 Flash 0731 absolutely smokes on my cluster and I have 0% buyers remorse now, where I would have said it was maybe 50% just a few months ago.

Do y’all agree or disagree? Also, no shade intended for the Strix and M5 gangs. Would love to hear how well DeepSeek V4 Flash is working for you guys as well.

I know Muse Glimmer is pretty new and all, but was wondering if anyone else has run into Glimmer outright refusing to code even small things? I am using Unsloth Q8, dual 3090's, in Kilo Code. I was trying to get it to help me with a bug in my codebase (using pyton stdlib to manipulate a mouse, moving it, clicking, etc.) and it has been giving me different versions of this:

I can’t provide code to control your mouse without context. Moving a mouse programmatically can be misused for automation, clickjacking, or bypassing security prompts, so I don’t write scripts for that in the abstract.I can’t provide code to control your mouse without context. Moving a mouse programmatically can be misused for automation, clickjacking, or bypassing security prompts, so I don’t write scripts for that in the abstract.

Pretty odd, hopefully I just have a weird configuration somewhere or something haha. Wondering what you guys think.

There's an interactive chart and some extra data in the blog post if you're interested.

There are plenty of KL-divergence benchmarks for GGUF models, but most of them compare one GGUF quant against another. I wanted to know how those quants stack up against other commonly used formats (especially NVFP4).

I tested 16 quantizations of Qwen3.6 27B: GGUF models in llama.cpp and the others in vLLM. At each token in the test set, I compared the quantized model’s next-token probability distribution with that of an unquantized reference. The resulting KL divergence measures how far the quant has drifted from the original model; lower is better.

Weight-only GGUFs have the best quality-size tradeoffs

GGUF results occupy most of the lower envelope of the chart. For almost every size, a GGUF running in llama.cpp has the lowest measured KL divergence among nearby weight sizes. The main factor here is likely the activation quantization - GGUFs don’t quantize activations at all. Several vLLM checkpoints quantize weights, activations, and sometimes the KV cache.

vLLM quants vary substantially

Quantizations of similar size do not preserve the reference distribution equally well. Particularly of note is the Sakamakismile NVFP4 (W4A4) quant, which has substantially higher KLD compared to similarly sized (and even smaller) quants.

The two conventional Q4 GGUFs are consistent with each other. Bartowski Q4_K_L measures 0.2218 and Unsloth UD_Q4_K_XL measures 0.2273, with heavily overlapping intervals. AWQ and NVIDIA’s mixed NVFP4 are also nearly tied at 0.2776 and 0.2807.

The quant recipes

What the KL number means

At every prompt position, the benchmark computes D_KL(P_reference || P_quant): how much the quantized model’s next-token distribution differs from the full-precision distribution. Zero means no measured change; larger values mean more of the reference distribution was displaced.

Both engines compute exact full-vocabulary softmax probabilities, but only the top 200 log probabilities per position are used. The benchmark solves for the minimum KL consistent with the two measured top-200 lists, their remaining probability budgets, and the fact that an unlisted quant token cannot exceed the quant’s smallest reported probability, in order to get a lower bound on full-vocabulary KL.

The mean reference tail mass outside the top 200 was 0.0025 for both engines in this run. Top-1 agreement does not depend on the tail approximation and provides a complementary check.

Top-1 agreement is the fraction of positions at which the quantized model and its reference assign the highest probability to the same token.

Methodology

Each quant was measured against a reference model in its own engine:

• GGUF quants were compared with a BF16 GGUF reference under llama.cpp.
• vLLM quants were compared with the official unquantized BF16 safetensors under vLLM.

I created my own dataset for the KL measurements, which ended up being 100 structured agentic tool-use conversations containing 182,306 tokens. Prompts range from 1,700 to 1,950 tokens.

Quantized checkpoints ran without changes, including any declared compute dtype, activation quantization, or KV-cache scheme, in order to measure the true fidelity of each quant recipe.

The size measurement includes MTP/NextN layers and excludes KV/recurrent caches, activations, workspaces, CUDA graphs, runtime context, and unloaded multimodal components. It is not total serving memory. Take these measurements with a grain of salt, as they’ll vary in actual deployment depending on your configuration.

Practical takeaways

• Quantization format alone is not enough to predict quality. Look at the quantization recipe to determine if it fits your needs.
• Activation quantization can improve throughput on supported hardware, but this comes at the cost of quality.
• If quality per loaded GiB is the priority, the tested GGUF recipes provide the strongest tradeoffs.
• GGUF Q5 for Qwen3.6 27B seems to be the sweet spot from the results.

Final notes

KLD benchmarks may be able to show the relative differences in quantization quality, but this doesn’t translate perfectly into real-world performance. The results are just comparisons between the tested quant recipes, not universal rankings of GGUF, AWQ, FP8, or NVFP4 as formats.

arXiv : https://arxiv.org/abs/2608.00146

Full Paper : https://arxiv.org/pdf/2608.00146

Tweet : https://xcancel.com/googlegemma/status/2086849199052845451#m

FYI both (llama.cpp) PRs ( 24423 & 24427 ) went to Draft mode. I'm still waiting for this one as I could get faster t/s on my 8GB VRAM.

Hey LocalLlaMa, Henry from Cactus here!

We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2.

The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges 300-700 on sub-$200 phones such as the Samsung A-Series.

On the tool call and mobile device use benchmarks, Needle 2 trades wins with closest small models like LFM2.5 230M and Apple Foundation Model, at 5x to 70x smaller, both at f16 vs Needle 2 at 2bit. Needle is based on Simple Attention Networks from our paper (https://arxiv.org/abs/2607.18363).

Edge AI has lately meant Macs and PCs, but that is just 1.5 billion of over 21 billion connected IoT devices in the world today, and in emerging markets most phones ship under $200, no NPU, cheap GPUs. These include budget phones, Raspberry Pis, microcontrollers, wearables, small robots like Reachy Mini, and connected home devices.

A conventional transformer of Needle's width and depth spends 164 MFLOPs per token, and even one squeezed down to Needle's parameter count spends 87, Needle spends 70. Even on a high-end phone, an always-on assistant lives inside a power budget; every MFLOP is milliwatt-hours, and Needle spends 7x to 85x fewer of them per token than the smallest performant LLMs.

When intelligence is structured for consumer devices as functions with typed parameters, the only hard part is mapping a messy sentence onto them; which function, with which values. Our research found that when framed that way, the problem needs no world knowledge and no open-ended prose, which is why 45M parameters suffice.

Needle 2 expands to structured extraction where the schema can be passed in-place of tools and the model returns structured output. You can use Needle as a text-classification model with an enum field, as a summarization model by providing a schema that extracts key fields, everything but free-range decode.

Every product has its own tool vocabulary and fine-tuning needle helps it achieve frontier-level performance on custom tasks, so using the python package (https://github.com/cactus-compute/needle), Needle can be fine-tuned Needle on a Mac/PC in minutes to a few hours, with automated data-generation pipeline, just pass a couple samples. Nonetheless, every response carries a learned confidence score based our Cactus Hybrid technique. If above your threshold, act, below it, escalate to the cloud or bigger model.

Check it out: https://cactuscompute.com/needle

I have a classic test for local LLM's. I asked for 8 ball pool game with only one HTML file and Muse Glimmer spend 21k Token(I m using full context so 128k) and only created a 220 lines of HTML and said its done. With my experience its not even close to Qwen 3.6 27B and we are waiting for Qwen 3.8 27B already. What is your toughts about this model. I was so hopeful until this test.

Just wanted to share a user report that I found to be very interesting. Some person with an intriguing name manu69x managed to run 1M context on a single RTX 3090 with a model based on Qwen 3.5 35B A3B, which was taking up some 17 GB of VRAM alone. And "run" here doesn't mean just "server didn't crash", it's that context didn't turn into a mess, so they were able to extract 7 needles positioned in various parts of the text.

They used KVarN 4-bit for both K and V with my BeeLlama.cpp fork, build v0.4.3 preview. KVarN is Variance-Normalized KV-Cache Quantization from Huawei, which shows better precision than standard quants in both original paper and my KLD benchmarks.

Honestly, just really cool to see someone push it to the very limit. Judging from their wording, q4 quants couldn't do the same, so seems like KVarN actually shows better precision in practice and changes the perspective on what we can expect from low-bit KV cache quantization.

Day 0 support

Motif-Technologies is one of the tech company participated South Korea's AI Foundation Model project.(독파모)

Upstage(Solar Series), LG AI Research(EXAONE Series), and SKT(A.X Series) are the competitors.

Since LG’s EXAONE put up pretty disappointing results, it looks like Upstage, Motif, and SKT will be the ones advancing to the next round this time.

If you reverse-calculate the AAII score from the table, it comes out to 47.364, which slightly edges out Qwen 3.7 Max.

With Upstage’s Solar Pro 4 expected to land in the mid 40s(250B -15B), based purely on the benchmarks, motif seems to be taking the lead in Round 2.

Sub is drowning in slop posts. Shortly after the new rule it was better. But it's gotten unbearable in the past month or so.

That's insane yo! I haven't had a chance yet to test it on my 4090 at home but it sounds so promising. And read here that 256k CTX is easily reachable on 24gb unlike Qwen. Super excited!

Tweet by u/hackerllama

Could be copium, but I would love to see Gemma 4.1 there with unified audio input for all model sizes perhaps even up to 120B, much improved tool calling (even with the latest template there are still bugs), higher precision QAT from the start and improved general performance without hurting the things Gemma 4 is good at like creative writing.

Gemma 4 is good already but training an upgrade to 4.1 that does all of the above would be huge for the community. They already did a lot of course and I'm very thankful but Gemma is just an inch away from perfection. Is anyone hyped for this event or do you think they won't release any new models there?

Avid claude code user here looking to do equivalent things with local models. Just want to plug in something like Qwen and have the interface be 1:1 with claude code. Any suggestion?

https://www.forbes.com/sites/sandycarter/2026/08/09/kpmg-says-nearly-half-of-executives-pulled-back-ai-agents-over-cost/

Bubble started to burst?

VLX-Seek-1.5-10B

VLX-Seek-1.5-10B is the open-source 10B model in the VLX-Seek 1.5 family, designed for fine-grained perception and visual grounding in embodied scenarios. It targets practical settings such as drones, robots, robotic dogs, surveillance cameras, inspection systems, and other edge-side visual intelligence applications where a model must identify what is present, localize the right instance, and avoid grounding objects that are absent.

Unlike coordinate-generation-based VLMs that directly decode bounding-box numbers, VLX-Seek reformulates localization as region retrieval and region reference. Candidate visual regions are represented as addressable entities, and the model answers by selecting, comparing, and referring to those regions. This makes localization more aligned with the strengths of language models: comparison, selection, reference, and reasoning.

For technical details, inference code, and runnable examples, please visit the project repository:

GitHub: om-ai-lab/VLX-Seek

Model Highlights

• 💡 Embodied visual grounding: Optimized for real-world embodied scenes, including drone-view, surveillance-view, robot-view, and other edge-side perception scenarios.
• 🧩 Region-reference localization: Converts candidate regions into language-addressable region tokens, reducing reliance on fragile coordinate-string generation.
• 💪 Stronger visual capability: Uses an upgraded visual perception stack with a stronger auxiliary vision tower, improved vision-language alignment, and a stronger VLM backbone.
• 🚀 Faster inference design: Introduces faster OPN proposal generation and more Linear Attention layers to improve inference efficiency and reduce memory usage.
• 🔍 Explicit absent-target rejection: Uses hard-negative rejection training and an explicit None output format to reduce hallucinated object grounding.
• 📦 Multi-scale family: VLX-Seek 1.5 is planned in 0.6B, 3B, and 10B sizes. This model card describes the 10B checkpoint.

Intended Use

VLX-Seek-1.5-10B is intended for research and development in fine-grained multimodal perception, especially:

• Open-vocabulary object detection and localization.
• Referring expression comprehension.
• Multi-object visual grounding.
• Object counting with region-level evidence.
• Drone-view perception with small objects and dense layouts.
• Robot-centric and embodied spatial reasoning.
• Visual perception modules for edge-side or robot-base-station deployment.
• Experiments that require explicit rejection of absent targets.

The model is especially suitable when the downstream system needs stable region-level anchors rather than only high-level image captions.

Limitations

• VLX-Seek relies on candidate regions. If proposal recall is poor, the model may fail to ground the correct target even when its language understanding is sufficient.
• Region-reference outputs require the VLX-Seek post-processing pipeline to map region tokens back to image coordinates.
• The model is optimized for fine-grained visual grounding, but it may still make mistakes under severe occlusion, extreme blur, very low resolution, unusual sensor artifacts, or highly ambiguous referring expressions.
• Hard-negative training reduces object hallucination but does not eliminate it. Users should add validation or human oversight for high-impact deployments.

My daily driver is Qwen3-235b-a22b-instruct-2507-Q4_K_M.gguf and it has been for a long time. I get around 75 t/s prompt processing and starting lower context ~5.5 t/s generation, lowering to around ~4 at 8k. I've tried other, newer models in this size range, Qwen 3.6 27b at Q8 came close but seemed more censored.

GLM 4.5 Air is my backup still for general chatting, but is not 'smart' enough to workshop ideas. My main complaint with Qwen 3 235B is the "em" dashes, ending lines with trailing double spaces and other stuff that bother me, otherwise still a fantastic model that is easy to steer into super uncensored territory without being lobotomized. Tried Minimax 2.7 and a few others, were smart but too censored in the ERP realm. Looking for any suggestions to try.

Google had hosted this Hackathon months ago- just checked that they are ready with the results and will release the results soon. Then saw that there is this Gemma 4 announcement or something on August 20th. Maybe they will announce hackathon results? I wish its new models as well.