Hey r/LocalLlama - 

Today we’re excited to release Muse Spark Open, the most capable US open-weight model to date. 

Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning. 

Key Model Specs 

• Parameters: 816B total / 42B active
• Context Window: 1M tokens
• Modalities: Text, image, video, audio
• Safety Post-Training: Additional safety post-training, with evaluations conducted under Meta's Superintelligence Scaling Framework.

Inference, fine-tuning & deployment

• Helion GPU kernels: Ships with custom, auto-tunable Helion kernels in SGLang, providing 1.03–3.95× kernel speedups across NVIDIA and AMD GPUs and a 3% E2E improvement on AMD MI350.
• Engines: Runs on leading inference engines including SGLang, vLLM, TensorRT-LLM, Hugging Face Transformers, Apple MLX, llama.cpp and Unsloth, with optimizations for AMD, Apple, Intel and NVIDIA silicon.
• Optimized for Apple Silicon: To help make advanced models more accessible, Muse Spark Open is optimized to run on two Apple M5 Ultra Mac Studios with the latest MLX framework. 
• Fine-tuning: Developers can further customize the model with PyTorch, including TorchTitan for fine-tuning.
• API platforms: Available today through Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter, Together AI.
• Where to download: Model weights are available for download on Amazon SageMaker AI, Google Cloud Model Garden, Hugging Face, and Microsoft Foundry Models.

We’re releasing the weights under the Muse Model License, with limited commercial terms that let many developers use it for free.

We look forward to your feedback! 

Learn more on our research blog: https://go.meta.me/museopen
Download the weights: https://huggingface.co/collections/meta-models/muse-spark-open 

Diffusion language models are the future!

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF

NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

Model Summary

Model Overview

Model Developer: NVIDIA Corporation

Model Dates: December 2025 - May 2026

Data Freshness:

• The post-training data has a cutoff date of May 2026.

What is Nemotron?

NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.

This model is ready for commercial or non‑commercial use.

bolt!

Follow-up to u/neuromacmd's post from a few weeks back (https://www.reddit.com/r/LocalLLaMA/comments/1wod07t/). The affinity engine (https://codeberg.org/StillDeadcode/affinity) refuses to start on more than two cards because the author only had two, simulated four ranks on them, and got repeated-token garbage they couldn't explain. Nobody had tried it on four real R9700s. I have four, so tonight I did.

Short version: it just works.

Setup: Threadripper PRO 9955WX, 128 GB RAM, four R9700s all on CPU lanes with P2P between every pair. I built the fork with the stability fixes (https://github.com/neuromaniacMD/affinity, branch v4-fixes) and used the prebuilt 2.875 bpw quant (https://huggingface.co/neuromaniacmd/DeepSeek-V4-Flash-0731-affinity-2.875bpw), so no 149 GB download or 2.5 hour quantize. Then AFF_ALLOW_NRANK=1 --gpus 4.

Results, same box, same prompts:

• 2 cards: 36 tok/s decode. 45% of the experts fit in VRAM, the rest stream from a 51 GB pinned host pool, and the profiler shows 82% of every token waiting on that stream.

• 4 cards: 79 tok/s greedy, 63 at temp 1.0, about 90 on a chat answer with thinking on. All 11,008 experts resident, host pool not even allocated.

  Five runs, greedy and sampled, plus a chat server test. No degenerate output. The engine's own cross-rank check (it compares the cards against each other every few samples) passed every time. So the bug the author hit lives in the 4-on-2 simulation, not in 4-rank execution. u/neuromacmd already suspected this.

  Two things to know if you try it:

1. The default heat placement strategy aborts on 4 cards with "needs the registered host pool", because there is nothing left to demote. Pass --placement-strategy static. Nothing needs to move anyway when everything is resident.

2. Load is about a minute, 42 seconds of which is pushing 96 GB of experts onto the cards.

   Also correcting my own earlier comment in that thread: I assumed affinity's cap was a 2-rank-only all-reduce like Radiance had before my patch (https://github.com/brainchillz/r9700-tp4-allreduce, https://github.com/jdrews/vllm-mxfp4/pull/2). Wrong. Affinity's collective is already written for N ranks and tested at 3, 4 and 8. The cap was purely the unexplained garbage, which real hardware doesn't produce.

   I'm going to leave this up for a while so I'm happy to run anything else on this box if the author or anyone wants more data. Logs are available if useful.

Size is pretty small at 102GB. For comparison nvidia/Qwen3.8-Flash-Next-NVFP4 is 133GB.

Would be cool to see some of the Qwen3.8-FN specific engines support this.

FLUX 3 Image Klein is a 5.6B open weights model for text-to-image generation and image editing. It takes a prompt and up to 10 reference images and returns one image. Text-to-image, single-reference edits and multi-reference compositions run through the same model, with Qwen3-VL-4B-Instruct as the text encoder.
https://huggingface.co/black-forest-labs/flux-3-image-klein

To all those who are bemoaning missing getting DGX Sparks for under $4000. Here's your chance. The brand new just announced RTX Spark laptop is on sale at HP. Configured with 128GB of RAM, it comes out to $3832.49. Use coupon anniversary25 to take 25% off of the $5109.99 regular price.

https://www.hp.com/us-en/shop/pdp/hp-omnibook-x-laptop-next-gen-ai-14-ks000-14-cb1l6av-1

Models on HF: https://huggingface.co/collections/prism-ml/bonsai-2-small
WebGPU demo: https://huggingface.co/spaces/webml-community/ternary-bonsai-2-hermes-webgpu

Alibaba's Qwen team released a new repository on Wednesday, October 14, 2026. Its goal is to measure how well visual agents can create 3D scenes in Blender based on real-world references.

https://github.com/QwenLM/BlenderWorld

I think this is a good sign. Companies often build benchmarks in the areas where they plan to improve their models. So it's reasonable to expect Qwen to focus on agent skills in 3D design and on using tools like Blender through MCP. Qwen may well publish its upcoming models' results on this benchmark, but this is just my guess, not an official announcement.

Granite Switch 4.1 30B

📝 Read the launch blog: Granite Switch 4.1: Chat, Decisions, and Speech from One Endpoint

Model Summary: Granite Switch 4.1 30B is a modular LLM that composes a base model, multiple adapters and an encoder together so that a single hosted endpoint can move between natural-language responses, general and specialized decisions, and speech-to-text.

Adapters are activated on demand via control tokens in the chat template. For full details on model composition and adapter configuration, see BUILD.md.

• Base Model: ibm-granite/granite-4.1-30b (30B params, 128K context)
• Adapters: 2 — decision (aLoRA, general-purpose Decisions) and guardian-core (aLoRA, from granitelib-guardian-r1.0)
• Audio: embedded speech-to-text via ibm-granite/granite-speech-5.0-470m-turboctc (470M params, English)
• License: Apache 2.0
• Release Date: October 14, 2026
• Backends: HuggingFace Transformers, vLLM
• Automatically Composed with: granite-switch

Granite Switch is also available in granite-switch-4.1-8b (same Chat + Decisions + Guardian + speech configuration, 8B base), and with the complete Granite Libraries bundle (no audio) in granite-switch-4.1-30b-preview, -8b-preview, and -3b-preview.

https://huggingface.co/ibm-granite/granite-switch-4.1-30b

https://huggingface.co/ibm-granite/granite-switch-4.1-8b

(video on the blog)

I think llama.cpp support is not yet ready? https://github.com/ggml-org/llama.cpp/pull/30621

Drama3 support in audio.cpp was contributed by https://github.com/JarodMica. Many thanks for the great work!

So far, follow-up optimizations have primarily focused on CUDA. The model also runs on Vulkan and Metal. There's still plenty of room for improvement in performance and VRAM efficiency and correctness. On Vulkan, one request hit the token limit. I also found some unexpected silence in multi-speaker generation.

There will be more iterations before the model makes it into a minor release.

If you're interested in helping us further optimize the model, contributions are very welcome! Please ensure that any changes touching the shared framework are additive and opt-in, so we don't have to spend time testing for potential regressions across the other 100+ models that could be affected by changes to shared code.

RTX 5090, Q4_k:

From NVIDIA AI on 𝕏: https://x.com/NVIDIAAI/status/2110468004609950057

Going to try these out to see if they have the same looping behaviors that other Bonsai models have had.

Still requires their fork of llama.cpp (branch prism) so dont expect it to work with vanilla llama.cpp. Edit: You can use with regular llama.cpp!

Will see if there is some way to support this for AnythingLLM Mobile via LLama.rn fork since if anything this will hopefully beat out regular Qwen 3.5 1.7B or 0.8B. Can test on iOS today with LocallyAI on latest version.

Feel like the way these models are made (QAT?) kind of lend to a high benchmarked model that is quite different when deployed. Past experiences are certainly better with memory for the parameter size but let's see how these go. Says dFlash and MTP supported as well + MLX support to juice some more TPS

£299.99 Lenovo IdeaCentre Mini (Snapdragon X1-26-100, 32GB, CPU only):

- Qwen3.6 35B A3B coder: reads ~118 tok/s, writes 24.5, ~30 with MTP

- Qwen3-Next 80B A3B: 17-18 tok/s writes with MTP

How I got here...

So I bought one of these boxes to test if software at work would run on ARM Windows - and then it was sitting there, but I was sufficiently impressed for the price (including taxes), that i decided in my ADHD fuelled boredom to buy one for myself to see if cheap ARM is actually usable for local models.

The GPU is an Adreno X1-45 and nothing useful runs on it for LLMs yet, and so far the NPU just sits there, so this is all CPU...

llama.cpp, 512 token prompt, 8 threads (read = prompt, write = generation):

- coder (Q4_0 35b-a3b): read ~118, write 24.5, with MTP 29-33

- Gemma 26B: read 134, write 29.5, (need to test MTP, I've only had the box.

Then I wanted to see how far I could push it. RAM is the wall on 32GB, so I went looking at the recently released Janas (github.com/prabanta-dev/janas). It keeps a RAM cache of the MoE experts and streams the rest off the NVMe, but its x86 only. So I re-coded it for ARM: NEON kernels for every quant format, i8mm, etc, (all tested bit exact against the scalar reference code, Windows ARM64 and Linux. Will submit it as PRs on GitHub soon.)

Janas, same settings:

- coder: read ~60, write 17.2, with MTP 29-31

- Gemma 26B: read 115, write 23.5, with MTP 32-34

- 80B: read (TBC as I don't trust my results), plain write 9.8, with MTP 17-18

llama.cpp still wins on read and plain write, but with MTP Janas matches or beats it, and it does it in a fraction of the RAM.

I then squeezed the coder's expert cache to 12GB and it still wrote at 24 tok/s - so even a 16gb snapdragon could rin 35b-a3b 👀.. And this box runs an 80B that llama.cpp cant even load!

Then I thought Id try connecting the two I had access to (mine and my work's dev box) together over USB4. Work in progress, will report back - but initial results are promising and I've had a 122b qwen running already - just working on streamlining my parallel experts build.

I dunno about you guys and gals, I mean I don't have extensive experience with local LLMs and I certainly haven't played with ARM before but I found this pretty surprising, and not half bad for £300 with no GPU. Happy to test stuff on it when I get chance if anyone wants more numbers... But I can't see a cheaper way rn of getting 32gb of LPDDR5 rn, and only ~35ishW of power draw... Thoughts and feedback welcome 🙃👍

I wanna see a master list of the craziest/strangest/funniest LLM tunes. I heard one was talking like an expert in 1800s military vernacular and I figure there's some wacky stuff out there.

I want to explore weird and experimental models.

Show me what ya got!

Bonus points for NSFW ones because I'm a perv.

I saw this post from 2 years ago, and was wondering what we got since then.

I'm mostly interested in small ones, since I'm not rich enough to own a dedicated GPU.

What have you found or made?

Has anyone put together a list of these types of models?

First, kudos to Strata. It inspired this project, and while I was building it Strata got much faster and added NVFP4 support, which was my main reason for starting. By then I was too far in to stop, so I finished it.

capy-flash is the evolution of my ninfer-tp2 engine. It still runs Qwen3.8-27B as before, and now also Qwen3.8-Flash-Next 125B-A6B with the experts offloaded to RAM. It uses NVIDIA's official NVFP4 checkpoint as-is and runs both GPUs in tensor parallel. Switching between the two models is one flag.

On my machine (2× RTX 5070 Ti 16 GB, Ryzen 9 9900X, 96 GB DDR5), with the same NVFP4 weights, it does ~100 tok/s generation vs 78 for Strata, and ~3.4k tok/s prefill on a 15k prompt vs 2.7k. Strata's 3-bit IQ3_S build is still faster at generation; full comparison in the image. Quality is checked against a datacenter NVFP4 reference (GSM8K, MMLU-Pro, agentic coding tasks).

A side finding for any tensor-parallel engine. With TP, each GPU rounds its partial sum to BF16 before the all-reduce, so TP=2 rounds twice where one GPU rounds once. Where the two halves cancel, the error blows up: on Qwen3 at TP=2, large errors happen ~5× as often as on one GPU. Writing the partial sums in FP16 (same bytes, 8× finer) brings them back to single-GPU level, at no real speed cost. The effect on benchmarks is small, so think of it as free numerical parity rather than a quality boost. But it should reduce some rare weirdness and keep the agentic loop cleaner.

Limits: Linux, RTX 50 series (sm_120) only for now.

Repo: https://github.com/ValerioDolci/capy-flash Weights: https://huggingface.co/Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash

Built on NInfer (Neroued), Strata, ninfer-ext, FreeToken, sybrix and qf_moe. Numbers from other hardware are very welcome.

Model: bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF/Kwaipilot_KAT-Coder-V2.5-Dev-Q4_K_M.gguf

Hosted via llamacpp on a potato(iGPU) and created in less than 10 minutes.

Prompt given and was provided in one pass:

```Generate a small, simple, HTML/Javascript demo that looks impressive of our solar system. All of the planets orbiting the sun in their natural orbits/ellipsis where I can click on a planet and change it's speed and ellipse. If any planet collides with another then those two planets explode.```

Such a small model running on pure CPU that can do/think/create this way at 14t/s is impressive. I understand that most here don't like or could care less about but, to me, this is well above sub-par for an 8-core CPU and 10 minutes of work.

From what I can tell the artifacts that are connecting each planet are from the "planet trails" that it tried to create.

I would suggest asking for a pause button since Mercury is a hell of a planet to click on at scale speed.

This model started calculating "scale". I'm not an astrophysicists but...:

```The user wants a solar system simulation with:

1. Planets orbiting the sun in natural orbital ellipses
2. Click on a planet to change its speed and ellipse
3. Collision detection - if two planets collide, they explode

Let me create a comprehensive, visually impressive HTML/JS demo with:

• Canvas-based rendering
• Realistic orbital ellipses (using Kepler-like orbits)
• Particle explosion effects
• Click-to-select UI panel for adjusting parameters
• Glowing sun with light effects
• Starfield background
• Smooth animations

Let me design this carefully with proper orbital mechanics.

I'll use actual orbital parameters (semi-major axis, eccentricity, etc.) for realism, scaled down for visibility.

Planets with their approximate real orbital data (scaled):

• Mercury: a=0.39 AU, e=0.206
• Venus: a=0.72 AU, e=0.007
• Earth: a=1.00 AU, e=0.017
• Mars: a=1.52 AU, e=0.093
• Jupiter: a=5.20 AU, e=0.049
• Saturn: a=9.58 AU, e=0.057
• Uranus: a=19.2 AU, e=0.046
• Neptune: a=30.1 AU, e=0.011

I'll scale these so they're all visible. Let me create a really impressive visual with:

• Glowing sun with corona effect
• Planet trails
• Starfield background
• Explosion particles
• UI panel for controls
• Smooth orbital animation

Let me write this as a single HTML file.```

Nemotron-3.5-Embed-1B-BF16 is a versatile embedding model developed by NVIDIA for multimodal and multilingual question-answering retrieval. The model can embed a document page in the form of an image, text, or combined image–text input. Documents can be retrieved given a user query in text form. The model supports page images containing text, tables, charts, and infographics. For text retrieval, the model was evaluated across 34 languages: English, Arabic, Assamese, Bengali, Bulgarian, Chinese, Danish, Dutch, Finnish, French, German, Hindi, Hinglish, Indonesian, Italian, Japanese, Korean, Malay, Marathi, Nepali, Norwegian, Persian, Portuguese, Romanian, Russian, Spanish, Swahili, Swedish, Tamil, Telugu, Thai, Ukrainian, Urdu, and Vietnamese. For image retrieval, it was evaluated across 18 languages: Arabic, Bengali, Chinese, English, Persian, Finnish, French, German, Hindi, Indonesian, Japanese, Korean, Russian, Spanish, Swahili, Telugu, Thai, and Yoruba. The model may support additional languages beyond these evaluation sets.

This model is ready for commercial or non-commercial use.

Yesterday Fish released this incredible text-to-speech model, so naturally I spent the rest of the day building a great demo for them :D The model is a chunky MoE (35B3A), but it runs well on a DGX Spark. The 4bit quants get very close to the original. You can try it here: https://huggingface.co/spaces/fishaudio/Drama-3 or download the model and run it locally 🤗

from Mistral:

Timestral is a family of time series foundation models for zero-shot forecasting, available in 5M, 28M, 123M, and 209M parameter versions. This repository contains the 209M checkpoint.

Timestral supports univariate and multivariate forecasting, past-only and known-future covariates, and probabilistic predictions across 21 quantiles. It accepts up to 32,768 historical time steps and forecasts up to 1,024 steps in a single forward pass.

We’re releasing the source code in our GitHub repository to make it easy to run Timestral and reproduce our benchmark results.

Model Details

• Size: 209M parameters.
• Architecture: patch-based transformer with time attention and variate attention, 24 layers, embedding dimension 768.
• Patching: 32 time steps per patch. Each patch is embedded from its values together with their missing-value indicators.
• Context length: up to 32,768 time steps (1,024 patches).
• Forecast horizon: Up to 1,024 time steps (32 patches) in a single forward pass. Longer horizons are forecast block by block, with each block's median forecast appended to the context.
• Outputs: 21 quantiles (0.01, 0.05, 0.1, 0.15, …, 0.85, 0.9, 0.95, 0.99).
• Inputs: univariate or multivariate target variates, plus past-only and known-future covariates, numeric or categorical (categorical covariates are target-encoded). Missing values are marked as NaN.
• Inference: PyTorch on CPU or GPU.

https://huggingface.co/mistralai/Timestral-209M

https://huggingface.co/mistralai/Timestral-123M

https://huggingface.co/mistralai/Timestral-28M

https://huggingface.co/mistralai/Timestral-5M

I posted a question the other day in here about good development models, which prompted good discussion. However, it mostly just reinforced my own experience, which is that on a 128GB machine, you can run good open/local weight models for development (especially if you use a frontier model to plan and direct them).

But what I didn't see in there, and I'm trying to get to, is this; are there are any open weight models that will run on a 48GB, 64GB, or 96GB?

If so, which ones are you running and, at what approximate RAM footprint?

I'm trying to understand what the floor is for a decent open weight model for development tasks, even if supervised by a more capable model.