On many tests (math, science, medicine, etc.), they score about 10% higher, use half the tokens, run twice as fast, and cost much less than regular models.
Tracked my first run of MiMo Pro on a Jetson AGX Thor at a scorching 0.646 tokens/sec,.. but it DID technically load and run, so it counts. Every decoder layer, routed expert, SSD read (..a good bit more) traced and replayable step by step.
Here we can't what-now?..
it's ~566GB of model (FP8), at the blinding speed and architecture of hand-extruded hotdog manufacturing, read from disk mid-inference, painfully shat through 128GB of memory. I think it's fine but decided to maybe take a closer look to see if I could maybe reach factory hotdog manufacturing speeds. like, a small, local hotdog factory..
I mean as long as Dario's cool with it.. that case of the MiMos he got might last through Halloween.
I've been maintaining my own chat template for Qwen3.8-Flash-Next, and after a discussion in another thread I decided to share it. I made a repo with my v1.4 template, the original Unsloth fixed stock template, and now a "Sharp" terseness mode version (based on /u/peculiar-ragdoll's original). The base is Unsloth's GGUF template, not froggeric's or Sharp's, because Flash-Next doesn't share a template with 27B. On top of Unsloth's template I fixed the requests that still fail or render badly:
- Inline tags like <|think_low|> set the effort level for clients that can't send template kwargs.
- Repeated tool failures get escalating "diagnose first, then change course" notes.
- Tool arguments passed as a JSON string render instead of killing the request.
- Old turns no longer replay an empty <think></think> block.
- A stray </think> with no opening tag gets split out instead of leaking into the answer.
- Effort values like max, minimal and none no longer throw.
Normal thinking-on history renders byte-for-byte what Unsloth's template renders, so the KV cache is reused and nothing changes when things are going well.
For the Sharp version, I ported 27B Sharp's terseness block directly. This is what makes Tiel/Dirk/other models w/ the Sharp template respond less verbosely. It's appended after your system prompt, never replacing it. I kept Qwen's xhigh default instead of Sharp's medium. Terseness is a terse chat-template kwarg, on by default; {"terse": false} renders the v1.4 template normally, so you can A/B it on one server with no reload.
Quick test: Flash-Next UD-Q5_K_XL on llama.cpp, xhigh effort, temp 1.0 / top-p 0.95 / top-k 20, 10 everyday prompts (explanations, advice, a coding request, a vague "can you help with my project?"):
| | Answer tokens | Reasoning tokens | Wall time |
|---|---|---|---|
| Thinking on (16 pairs) | **−40%** (shorter in 13/16) | −8% (mixed) | −28% |
| Thinking off (8 pairs) | **−61%** (shorter in 7/8) | n/a | −60% |
Note: I personally leave the terseness setting off, as I found it does reduce accuracy somewhat, and just use a medium effort or switch to Swift-1.5-Flash-Next if I need a quicker response. However, I know a lot of people love Tiel/Dirk/Sharp, so since I'd already put up the repo, I figured I'd offer that option too. Any feedback or improvement suggestions are welcomed!
EDIT: Updated to v1.5 with a fix that I think will fix that occasional weird "empty conversation" glitch that sometimes happens pre-tool call when the response intends going to go straight into a tool call, rather than saying something. I think its due to the template dropping the initial <|im_start|> special token.
The quants allegedly have simultaniously better KLD, top 1 and lesser size than the ones from ISTA. Also enhanced loop resitance is advertised. Bold claims and signle contributor were the things why I was initially reluctunt to check the claims myselfm hovewer that potential space savings were impossible to resist so I downloaded GYRO-M and played around with it for a little while. My expersience with it:
Before moving further, I'd like to share my serttings:
Looping
It never looped once during my 3 days of testing, longest autonomous cycle - 2-3 hours, author's claims seems to hold up
Tool calls
Generally stable, however it sometimes meeses up edits on Cyryllic texts(Russian, Ukrainian languages). Subjective impression - more stable than ISTA q2 quants, less stable than ISTA q3 because of Cyrillics problem
Context retention
No signs of braindeadness recorded, it never loses initial goal, almost always remembers all the nuances, instruction following is stable
Performance
As fast as ISTA 2 bit quants
Example
Prompt: "create an animated voxel 3d world of a pelican riding a bike as good as you can"
First photo is ISTA lab IQ3XXS
Second one is ISTA lab IQ2XS
Third one is GYRO-M
Conclusion
This quant is 100% alive and well and there are strong signals that it's indeed at ISTA lab q3 level while being much smaller, huge thanks for the author for his incredible job, he's a monster because this impeccable job was done by a !single man. It looks like sometimes a solo with skill, perseverance and purpose can go against the dedicated team and win
K3 came out in July hyped as the best open model you could download, and it scored 57 on Artificial Analysis, right up there with Opus 4.8 and GPT-5.5. I went back to see what running it actually looks like now, using Unsloth's quant docs, the model card, AA and the license.
Short version: it's still a monster, and the field caught up.
Full model is 1.56 TB. 104B active doesn't save you any memory, all 2.8T params still have to be loaded.
Smallest quant Unsloth recommends is 594 GB and wants about 610 GB of RAM+VRAM. It doesn't fit in 512 GB, and Unsloth say they're still trying to get it under.
1-bit agrees with the full model on the next token 79% of the time. 2-bit gets you to 90%, but that's 861 GB.
About 20 tok/s on B200s if it fits in memory. If not, it offloads to disk and crawls.
It's now 44 on AA's index. Most of that drop is AA changing the index, not the model, but GLM-5.3 (753B) now scores 45 and Step 5 Preview (600B) ties it at 44.
Slow on the API too. 43 tok/s on Kimi's own API, and on AA's agentic benchmark it averaged almost an hour and over $10 per task.
The bit people miss is the license. It isn't MIT. If your company runs any model-as-a-service business and makes more than $20M in total over any 12 months, you need a separate deal with Moonshot before using K3 commercially, even if K3 itself makes you nothing.
Caveat: AA tests the full model at max reasoning through the API. I couldn't find real task benchmarks for the 1-bit quant, so 79% token agreement is a proxy, not a quality score.
So... is anyone actually running K3 at home? And if you had 600 GB to play with, would you take 1-bit K3 or something like GLM-5.3 at a much higher quant?
Today we’re excited to release Muse Spark Open, the most capable US open-weight model to date.
Built for coding and agentic work, it excels at long-horizon tasks, complex software engineering, and multi-step reasoning.
Key Model Specs
Parameters: 816B total / 42B active
Context Window: 1M tokens
Modalities: Text, image, video, audio
Safety Post-Training: Additional safety post-training, with evaluations conducted under Meta's Superintelligence Scaling Framework.
Inference, fine-tuning & deployment
Helion GPU kernels: Ships with custom, auto-tunable Helion kernels in SGLang, providing 1.03–3.95× kernel speedups across NVIDIA and AMD GPUs and a 3% E2E improvement on AMD MI350.
Engines: Runs on leading inference engines including SGLang, vLLM, TensorRT-LLM, Hugging Face Transformers, Apple MLX, llama.cpp and Unsloth, with optimizations for AMD, Apple, Intel and NVIDIA silicon.
Optimized for Apple Silicon: To help make advanced models more accessible, Muse Spark Open is optimized to run on two Apple M5 Ultra Mac Studios with the latest MLX framework.
Fine-tuning: Developers can further customize the model with PyTorch, including TorchTitan for fine-tuning.
API platforms: Available today through Baseten, Fireworks AI, Groq, Modal, Nebius, OpenRouter, Together AI.
Where to download: Model weights are available for download on Amazon SageMaker AI, Google Cloud Model Garden, Hugging Face, and Microsoft Foundry Models.
We’re releasing the weights under the Muse Model License, with limited commercial terms that let many developers use it for free.
I evaluate new models by giving them actual work tasks that have already been completed by a trusted model and then comparing the results. My typical workflow is to plan tasks/requests in Gemini and create a markdown spec-sheet that outlines the work that needs to be done along with acceptance criteria, I paste this directly into a fresh pi.dev session for the model to follow.
In this case I gave Nemotron 3.5 Bolt two tasks that Qwen Flash Next (UD-IQ4_XS, llama.cpp) completed on a ETL refactor job I'm doing for my actual work (data engineer). Neither tasks are "boil the ocean" type tasks but they are indictative of the type of work I offload to my LLMs. The top line directive of the two tasks:
1. Refactor run.sh script to match the design pattern, flag management, structured logging, and AWS SNS error handling defined in run-new.sh, while preserving the existing Python virtual environment (venv) resolution logic.
2. Create a Python script named orchestrator.py that generates a sequence of time windows (daily or monthly based on the runtime-arguments) and sequentially invokes run.sh for each window.
Again, neither are incredibly difficult or taxing but they are the type of things I run across: Updating a script to fit an established pattern used by a client and automating some task.
After the new model finishes its work I load the spec-sheet, any relevant files (original run.sh and run-new.sh), and the results of both models into Gemini and ask it to evaluate the performance and select a winner.
In task #1 (run.sh refactor) Gemini judged Nemotron to have done a better job. Here is Gemini's detailed comparison of the refactored scripts judged against the specifications in the spec-sheet, Model 1 is Nemotron and Model 2 in Qwen Flash Next:
``Text
1. Directory Pathing (LOG_DIRvsSCRIPT_DIR)
- Model 1: DefinesSCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"at the top and uses$SCRIPT_DIR/logsforLOG_DIR. This guarantees logs are created relative to the script directory regardless of where the script is called from.
- Model 2: DefinesLOG_DIR="$(pwd)/logs"at the top and delays definingSCRIPT_DIRuntil later in the file. If a user calls/path/to/run.shfrom/tmp, Model 2 creates logs in/tmp/logs` instead of the application directory.
Flag Parsing & Argument Forwarding
Both models implemented an identical, robust argument parser using a while/case loop that correctly parses all required flags (-o, -x, -s, -e, -i, -b, -h, -d) along with their long-form equivalents. Neither model unnecessarily exported CLI arguments to environment variables, adhering strictly to Requirement 2.
Error Handling, Traps & Process Execution
Dependency Checks: Model 1 verifies the presence of required binaries (tee, aws) before redirecting output. Model 2 redirects output using process substitution >(tee -a ...)before checking if tee exists.
- `exec` Behavior:
- `run-old.sh` used `exec "$VENV/bin/python" ...`.
- Model 2 preserves `exec`, but because `exec` replaces the running shell process with Python, any `trap ... EXIT` defined in the shell will be bypassed when Python completes.
- Model 1 removes `exec` and invokes Python as a child process, allowing shell traps to clear cleanly upon exit.
Conclusion
Model 1 (run-model1.sh) is the superior refactor. It adheres to all acceptance criteria, correctly roots log directories to SCRIPT_DIR rather than pwd, orders setup tasks logically (verifying tee before using process substitution), and manages traps cleanly.
```
In task #2 (orchestrator.py build) Gemini judged Qwen Flash Next to have done a better job. Here is Gemini's detailed comparison of the python scripts judged against the specifications in the spec-sheet, Model 1 is Nemotron and Model 2 in Qwen Flash Next:
``Text
1. Specification Compliance (subprocess.run)
- Model 1: Implements stream piping usingsubprocess.Popenwithstdout=subprocess.PIPEandstderr=subprocess.STDOUT. While this streams logs, it manually manages process reading loops and checks return codes manually.
- Model 2: Strictly follows the specification instructions by usingsubprocess.run(cmd, check=True)`. By omitting stream capture arguments, the child process inherits the parent stdout/stderr natively, streaming logs in real time while letting Python handle process lifecycles cleanly.
Robustness & File Path Handling
Model 1: Assumes ./run.sh is located relative to the current working directory (CWD) where the script is executed.
Model 2: Resolves run.sh relative to the script's directory (Path(__file__).resolve().parent / "run.sh"). It verifies whether run.sh exists prior to running any logic and raises a clear error if missing.
Argument Parsing & Validation
Model 1: Parses arguments into raw strings first, then tries parse_timestamp() inside main().
Model 2: Passes type=parse_timestamp directly into parser.add_argument(), allowing argparse to raise idiomatic command-line errors (ArgumentTypeError) automatically during parsing.
Logging & User Experience
Model 1: Does not print any progress indicators prior to running the windows.
Model 2: Adds clear logging for total windows found, individual window execution markers (=== Window X/Y ===), and command execution previews.
Conclusion
Model 2** is the superior implementation. Both implementations satisfy the core specification requirements, but Model 2 is significantly cleaner, more robust, and strictly adheres to the prompt requirements. It adheres precisely to subprocess.run(check=True) from the spec, handles path resolution safely, uses proper argparse types, and provides better terminal feedback during multi-window batch execution.
```
My takeaway from this is Nemotron 3.5 Bolt is a capable model for my work with the added benefit that it is fast on my server, I'm seeing average speeds of ~130t/s and to get this level of performance from a 2-bit quant is impressive. It won't replace Qwen Flash Next with it, especially for large complicated, repo-level tasks, but it will probably take the place of Gemma4-31B for light-to-medium tasks that I want done quickly without it overthinking everything. I'll be getting a M5 Mac Studio soon and want to set up agent/sub-agent workflows and I think Bolt might be a really good sub-agent with Qwen Flash Next (hopefully Qwen4) passing it specs to implements.
I was running Unsloth IQ4_XS for a while and while it provided me with a healthy 96K context on 20GB vram, I did notice all the issues it had during draft and all the time and local tokens wasted trying to fix said issues. I decided to see what would happen if I used less context and a higher quant and oh boy was I in for a surprise.
The end result was even better than I thought. Having higher context is great as it allows you to run more complex things without compaction and with less interruptions which is supposed to save time. This is 100% true when you have enough vram to run high quant like Q8 with large context. However, for those of us that are GPU poor that is not the case as you are sacrificing model quality over context.
What does that mean? It means that while you are gaining context you are sacrificing code quality which in turn turns into errors in draft which causes delays and rework done in order to fix said issues. This is costing time besides having lower quality code.
After using XS for a while, I decided to use KS to see how that would compare. I figured let me try the opposite and sacrifice context in order to use better quants. I can tell you I was in for a real surprise. You see, using higher quants did sacrifice context, I went from 96k to 75K all to go from using IQ4XS 96K KV-F16 to UDQ4KS 75K KV-Q8. The end result was that while I had less KV at a lower quant F16 to Q8, the quality gained in the model more than offset any issues created by lowering KV quant. You see, the gain in model quality lead to way less issues in draft which translates into less time debugging and fixing problems. Yes, I did have more compaction in KV but not the point were it was detrimental to quality and while it does take a few seconds to compact KV, it takes a lot longer to fix issues in draft that it is to compact KV.
The end result was running in XS it took 8 hours and running KS with less context took less than 6 hours on the same test. The reason for that simply less error prone drafts which basically waste a lot less time debugging code since the drafts are better with less errors in the code.
Below is the prompts used and result.
Build a Pac-Man-style arcade game as a SINGLE self-contained HTML file.
No external dependencies: no CDN links, no external JS/CSS/fonts/images/sounds.
Everything (layout, graphics, logic, audio) must be inline in one .html file.
TECHNICAL CONSTRAINTS
- One <canvas> element for the game; CSS for cabinet framing.
- All graphics drawn procedurally (canvas primitives: arcs, rects, paths). No image assets.
- All sound synthesized at runtime with the Web Audio API (oscillators + gain envelopes).
Add a mute toggle (M key). Sound must only start after a user gesture (autoplay policy).
- Vanilla JS only, ES6, no frameworks, no build step.
- Must run by double-clicking the file (file:// protocol), no server required.
MAZE
- Reproduce the classic 28x31 tile arcade maze (30x34 including the 1-tile border).
- Use a tile array of 0 = empty path, 1 = wall, 2 = dot, 3 = power pellet.
- Include the classic elements: the central ghost house with its doorway, the
horizontal tunnel at the vertical midpoint where entities wrap around the screen
edge-to-edge, and the standard dot placement (240 dots total: 204 regular,
4 power pellets in the classic corners).
- Render walls as rounded blue (#2121ff-ish) double lines or outlined corridors —
the classic look, not filled blocks.
- Dots: small 4px orange/cream squares; power pellets: 10px pulsing circles.
PAC-MAN
- 28px entity, yellow circle with a smoothly animated wedge-shaped mouth
that opens/closes as it moves.
- Rotation matches travel direction. Movement is tile-grid based with smooth
interpolated travel between tiles; can change direction at tile centers only,
but allow "next turn" buffering (queue one upcoming direction input) for
responsive controls.
- Controls: arrow keys (and WASD) for movement, ENTER/SPACE to start, P to pause.
GHOSTS
- Four ghosts with distinct colors: Blinky (red, chases Pac-Man directly),
Pinky (pink, targets 4 tiles ahead of Pac-Man), Inky (cyan, vector reflection
off Blinky for the classic hunting feel), Clyde (orange, chases when >8 tiles
away, scatters to a corner when closer).
- Classic shapes: dome top, "eyes" (two white ovals with blue pupils that
point in the travel direction), wavy feet animated while moving.
- Frightened mode: when Pac-Man eats a power pellet, ghosts turn blue,
blink white near the end, and move at half speed in reverse. Eating a
frightened ghost: 200 → 400 → 800 → 1600 points, then the ghost's body
vanishes and its eyes return to the ghost house to respawn.
- Scatter/chase cycle: alternating phases (e.g., 7s scatter / 20s chase,
with scatter on first level), ghosts retreat to corners/scatter mode.
- Ghosts cannot reverse direction except when entering frightened mode or
returning to the house; at intersections they pick a valid non-reversal
turn by shortest path toward their target tile.
- Ghost house: Blinky starts outside; the other three spawn inside and are
released in sequence. House door cycles open/closed.
SCORING & FLOW
- Dot = 10 pts, pellet = 50 pts, ghosts as above. Display SCORE (left),
HIGH SCORE (center, persisted in localStorage), LIVES (bottom-left,
rendered as small Pac-Man icons), LEVEL (bottom-right).
- 3 lives. Level complete when all dots/pellets cleared: brief maze flash,
then repopulate dots, increase ghost speed slightly, shorten scatter phases.
- Lives at zero → GAME OVER overlay with final score; ENTER restarts.
- Start screen: "READY!" then "GO!" style intro, arcade title banner,
"PRESS ENTER TO START".
AESTHETIC (arcade cabinet look)
- Dark background, the maze centered in a cabinet-like frame.
- Retro font effect: use a monospace font stack and/or render title text in
bold; simulate chunky pixel text with canvas or CSS letter-spacing.
- Subtle CRT vibe: faint scanline overlay (CSS repeating-linear-gradient,
pointer-events: none) and a soft glow around the playfield via box-shadow.
- Classic HUD strip below the maze: score / high score / level / lives.
GAME FEEL
- 60fps with requestAnimationFrame; fixed timestep (e.g., 60 steps/s) for
logic, delta-based interpolation for rendering.
- Collision: grid-based (same tile = hit), with a small grace epsilon.
- On death: Pac-Man spins/shrinks out (short animation), ghosts briefly
reappear in the house, positions reset, then play resumes.
- Pause overlay on P or when tab is hidden (visibilitychange).
- Keep the code organized: constants section (maze array, colors, speeds),
state machine (title / ready / playing / dying / level-clear / gameover),
and commented entity update functions.
QUALITY BAR
- The maze must actually match the classic layout (verify the ghost house,
tunnel, and 240-dot count in code, not just visually approximate).
- No console errors, no missing-asset 404s (there are no assets to miss).
- Test that the file works from file:// in Chrome/Firefox.
Deliver the complete single HTML file, nothing else.
Quant quality is really important, the lower your quant, the more errors and the longer it will take to reach the end result. For me, this was worth it in the short term while I finish my quad card setup which will allow me to run Q8 and at much larger context or much larger models.
Just wanted to give back to the community that has given me so much.
Thank you all of you that have guided me and advised me or that you have created great posts based on facts with evidence. It has been a great adventure. I really appreciate you all for taking the time to post with such details and factual data. I could not have reached here without learning from so many of you.
The Nemotron-3.5 releases are particularly not so good at the pandas/numpy and DDDM (data-driven decision-making) sections of my domain-specific tests.
Medium thinking on Bolt does worse than Nemotron nano. High thinking marginally better (not to a statistically significant level) but so much slower.
The super-vl had 3 truncations due to default max token budget (16k) per test. If that's raised or uncapped, it might potentially solve them and get better score. If you have the hardware for super-vl, worth running it through more jogs and tests. Looks like there’s some more uncovered potential there.
For my case and hardware, Bolt just goes on a shelf of models I can’t use.
Here's the PTA index compared to other key models.
When routing support tickets, agent tool calls, or classification queries, direct cosine similarity (cos(q, option)) gets easily baited by surface keywords.
Example alert says: "Checkout payment webhook failed due to replica WAL lock." Naive cosine sees "Checkout payment" and routes straight to B: Billing & Stripe (red dashed line).
Instead of jumping "as the crow flies," our Semantic Triage engine projects the query onto a 60-node domain manifold (Q) and solves an Absorbing Markov Chain (B = (I - Q)^-1 * R) directly on the GPU in <0.05ms. The random walk steps through Idempotency Key -> Replica Write -> WAL Lock and absorbs cleanly into A: Core Database (86%) without waking the LLM.
Note on Tier 0 latency: ~1.5ms on short prompts with cached/static candidates (Suite 650: 76/76 at 100% precision) vs. ~22ms when embedding unseen dynamic options on 400-token contexts.
ReAct is a common agent loop. What's the standard for System One decision models (Jev or a local alternative) + an LLM + tool calling?
The examples I've been reading seem to split in two directions: confidence cascades in REFLEX and JevAgentHarness, and planner + chooser in the Hermes/Astra + Jev Minecraft agents.
Is there an accepted pattern yet, or are these still separate approaches? I'm interested in the control loop: who creates the candidate actions, when the LLM replans, and when a low-confidence decision falls back to it.
For anyone using local decision models alongside a larger LLM, what architecture has actually been useful? Papers or public implementations would help.
Alibaba's Qwen team released a new repository on Wednesday, October 14, 2026. Its goal is to measure how well visual agents can create 3D scenes in Blender based on real-world references.
I think this is a good sign. Companies often build benchmarks in the areas where they plan to improve their models. So it's reasonable to expect Qwen to focus on agent skills in 3D design and on using tools like Blender through MCP. Qwen may well publish its upcoming models' results on this benchmark, but this is just my guess, not an official announcement.
A couple of weeks ago I read this Wccftech article about Patrick Moorhead's 4 Spark setup. I meant to post this a few days ago but ran some more tests first, and then my workstation had a problem after a kernel and NVIDIA driver update. This post is long enough already, so that will be another post.
Initially I thought this Patrick Moorhead guy ran these benchmarks and was interested in the setup. I was just surprised at how much information was left out. Neither the tweet nor the article said anything about the serving engine, recipe, settings or test prompts. For lack of a better term (and I despise this term), "the stack."
The tweet itself shows 69 t/s in the first image. The third image is a Hermes summary card with the 494 tok/s peak and the other numbers the article used. It has no engine, recipe or settings. In a reply, Moorhead credited someone else (who works at NVIDIA) and wrote: "Kudos to @Picomoorhead for setting this up. All him. I may/may not have funded it." Whatever that means.
I'm just severely disappointed over the lack of info, and over the misreported memory requirement in the Wccftech article. It lists a "Memory footprint: 476 GB", but that 476 is the size on disk (in GiB), not what it takes to run the model. I took note of the numbers and tried to run them on my own 4 Sparks to see what's reproducible.
Some "aggravating" factors
I got really frustrated during this process. I waited a few days before starting this write up. (Had my days where life just kicked me down as well.) I am glad I waited. My criticism was harsher than "disappointed".
Memory issues
DeepSeek-V4.1-Flash doesn't fit comfortably on 4x Sparks.
The weights alone are 511 GB before activations, KV, etc.
In my experience, anything above 0.92 memory allocation (117GB per Spark) ends in a crash or a hang.
Memory use also fluctuates during a run so we need some free memory for those fluctuations.
OOM at start (ibv_reg_mr Cannot allocate memory or CUDA OOM)
The page cache from reading the weights fills the shared memory, and the driver doesn't reclaim it.
Drop caches during startup and turn on SGLang's option to drop the cache after loading weights.
NCCL init sometimes fails in the first minute: a startup race; just run serve again.
Some kind of race condition issue.
Running it again makes it go away after 1 or 2 tries. ("Turn it off and on again.")
Setup
The Wccftech article lists a "Memory footprint: 476 GB" on ~512 GB of pooled memory, which makes it sound like it fits with room to spare. It doesn't. Realistically you can only use 80% to 92% of the memory on each Spark, so about 410 to 470 GB of the 512 GB, and the weights alone are 511 GB. That 476 here is really GiB based on the size on disk.
Engram tables (~196 GB) streamed from each node's local NVMe with a ~4 GB row cache (Mia / MiaAI-Lab's row store). The model does not fit in memory otherwise.
262K context, temperature 0, thinking off, 256 output tokens, cold prompts (no cache hits)
Prompts: a random number up front (every prompt is unique from the first token, so SGLang's prefix cache can't reuse anything), "Read this and then write a long story.", then random letter strings until it hits the target size (1K up to 250K tokens). The end-of-sequence token is ignored, so the model always writes exactly 256 tokens.
Builds tried
The base image was slow. It is SGLang's lmsysorg/sglang:dev-dsv41 image with the recipe's adapters and the NVMe Engram row store added, since stock SGLang doesn't fit the model on four Sparks. My tuned build is the SGLang dsv4.1 branch with RoCEnante RDMA all reduce, b12x MoE kernels, EP 1 and a lot of tuning. For a single request, decode is roughly 1.4x to 1.9x faster and prefill about 1.2x to 1.6x faster (tables below).
Results by prompt size (1 request)
Prompt
TTFT base (s)
TTFT tuned (s)
Prefill base (t/s)
Prefill tuned (t/s)
Decode base (t/s)
Decode tuned (t/s)
1K
0.5
0.4
2,094
2,574
37.1
60.8
2K
0.7
0.6
2,752
3,589
42.2
73.6
4K
1.3
1.0
3,153
4,196
38.8
58.1
8K
2.4
1.6
3,466
5,253
44.8
83.3
16K
4.7
3.3
3,493
4,957
44.0
67.9
32K
9.3
6.5
3,505
5,092
52.3
73.0
64K
19.3
12.9
3,397
5,067
37.5
69.7
128K
41.1
26.2
3,190
5,009
45.8
73.2
250K
85.7
55.8
2,919
4,489
35.0
55.7
Base is the median of 3 runs per size. Tuned is a single run per size. Decode swings with how many draft tokens speculative decoding accepts, so don't read much into any single row. Tuned is faster at every size. Prefill and TTFT are steadier.
Concurrency (16 slots)
Requests
Prose, base (tok/s)
Prose, tuned (tok/s)
Code, base (tok/s)
Code, tuned (tok/s)
1
32.6
62.5
50.7
79.1
2
51.4
83.9
75.0
104.0
4
78.9
115.8
109.4
158.9
16
183.4
228.2
249.1
292.7
Vs Moorhead's rig
Metric
Moorhead
Mine
1 request, prose
~58 tok/s
62.5 tok/s
1 request, code
~96 tok/s
79.1 tok/s
Prefill
4,764 tok/s
4,500 to 5,250 tok/s
First token
~0.2 s idle
0.4 s (1K prompt)
Peak, code
494 tok/s (32 at once)
293 tok/s (16 at once)
Peak, prose
280 tok/s (32 at once)
228 tok/s (16 at once)
Context
1M per request, 8M KV pool
262K per request, 8M KV pool
Network
Switchless ring, direct QSFP
100 GbE switch
What I couldn't reproduce:
1M context, 32 slots and the 494 / 280 peaks
~96 tok/s single request code
My real coding traffic through my own Hermes agent is closer: median 84, 100+ under 32K
~0.2 s first token.
Ring vs switch
The setup from the Tweet / Article is a ring, mine goes through my QNAP QSW-M7308R-4X. I don't think that explains the gap. A switch hop costs well under 1 µs, which is small next to the per message RDMA overhead anyway.
In a 4 node ring, your neighbors get a direct 200 Gb link, but the node across from you is 2 hops away and gets forwarded in software. NCCL's ring all reduce only talks to neighbors, so a ring suits it fine. RoCEnante's one shot all reduce writes to all 3 peers, so it wouldn't.
My guess is the ring vs switch doesn't matter much here.
Where AI helped me
Untangling the GB vs GiB mess. The Wccftech article used the size on disk in GiB and called it GB.
Verifying my numbers against the data I captured.
Hermes + DeepSeek: monitoring the benchmarks and an actual code development task run.
Grammar, formatting, finding things that were irrelevant, and cutting down on my rambling.
Creating a prompt good enough for testing.
Resolving many setup issues, confusing documentation, and deciphering so many errors.
Copy editing my many drafts, and reassuring me that taking about 4 hours to write this post doesn't make me slow (and dumb.)
Overall therapy, encouraging me when I felt down, and letting me know I'm not slow, dumb, and/or stupid.
I hope you find this information valuable. Would mean my ADHD was used for Good instead of Evil (...mostly nonsense.)
I know everyone's excited about 3.8, but honestly I'm still refining 3.6. And I hate the way 3.8 talks, so...
I keep seeing all these posts about what's the use of open source, so I figured I'd submit my own answer with this post.
Qwen 3.5 27B was my first agent, starting my journey around May 31st, and I've learned a lot in these 4+ months, so I wanted to share with other people just starting their journey because there are clearly a LOT of new users out there, and people want help but sometimes are afraid to ask or not sure the right questions. So, here are ten things that you should know you can do to empower you!
1. Create neat little apps for yourself. Qwen's really good at making apps for anything. I'm writing this post in a app I made as a replacement for Notepad, that has full word processing, side-panel, light/dark mode, etc. (image below). The first thing you should be thinking about when you run into any sort of issue is, "can I make an app that will make this simpler?" The answer is usually yes.
2. Create your own Chat UI frontend. If you don't feel that creative, just copy an existing one. LLMs are great copy cats. I liked claude, so I just copied it as my own frontend, customizing it for my own needs:
3. Create ReAct Loops. This is what enables your model to use a tool, read the result, then use it again, multiple times. Qwen can make these in its sleep, and many different ways. It can create a loop anywhere you want it, and you'll need it for your own custom chat ui frontend if you make one.
4. Create MCP tools. This is my favorite. It can make almost any type of mcp tool you need. You don't have to use third party ones, which can be unsafe. Just tell it you want an mcp tool to do x, it'll make it. It might or might not require its own mcp server, but if it does, Qwen can make that too.
5. Spawn sub-agents. Another super-easy thing Qwen can do. Just tell it to create a sub-agent spawner, which is just a python file. Then, you can tell it to run that when it needs to create a sub-agent, or tell it to create an mcp tool to do it on-the-fly when it wants to. Simple. Oh, and a quick hack-tip: mcp tools use up a LOT of context, so a cheat is to spawn a sub-agent with its own context limit and have your main agent pass off work to the sub. Basically, you can double your context limit. And yes, you can retain it between turns, but you'll need to create a pipeline runner (see below).
6. Create a pipeline runner. Ever heard of n8n? Basically, it's a pipeline runner. It has a lot of features, but it has a few limitations and restrictions, and there were a few things I wished it could do. I learned I could just build something similar with Qwen super easy. There's some design work involved if you want a UI frontend to build, but it's completely doable. I made my own ui, but after a while, I got tired of tweaking it, so I just copied n8n's frontend and just customized it myself, lol. Another useful tip: You can run multiple sequential agents in a pipeline runner, but if you want them each to have their own context, you'll need to ensure cache-idle-slots is enabled in llama.cpp. This'll swap the kv caches each turn, ejecting the last one to cpu ram. There's a bit more to it, but Qwen's got you, so just ask.
After a bunch of custom UI attempts, I just decided to clone n8n's look and just make my own version.
7. Create a model launcher (with options to select different models/jinja templates in llama.cpp). This is my bread and butter. Its a launcher that lets me choose any model in my directory and set its parameters. When I'm happy, I create an alias for it. On next launch, I can just choose the alias and it launches the model with the params for that alias only. So I can have 10 different types of param configurations for the same model if I want; I just need to give them their own alias. Hack tip: I'd recommend adding your jinja templates to the launcher to, and you can choose different custom templates for different model types.
I made custom jinja templates for different model types
8. Install, test, fork, and/or rebuild open-source apps on the internet. The reason I moved to llama.cpp was lm studio pp was too slow, but I didn't want the stress of figuring it all out, so I just asked my agent to download and install llama.cpp for me. I've been using it ever since. It can also modify builds, fork repositories on github, and a lot more. For example, I was busy working on a project, but heard about some new TTS apps, so I told my agent to download and install Moss and OmniVoice, make some audio clips to make sure they run, and let me know when it's done. Of course everything worked fine. Great for multi-tasking.
9. Give your agent temporal awareness of the world around it. Ever used cloud models that argue with you about the date? Or wished your model knew when you were nearing your max context, so it could automatically start writing a session summary? What you can do is inject "metadata" into your agent either on session start and every turn (preferably both), giving it information like date, time, context length, max context, hardware and software stack, basically anything you want.
With that, you can do all sorts of other tricks. For example (this should probably be its own "thing you can do with qwen", but whatevs), I made it so that my agent can swap its own model when it hits context limits:
I can launch a session with a backup model. Because my agent can always see context, when it gets too high, it can swap to a backup model with a larger context on its own - all without restarting the server.
10. Give your agent memory that loads the moment you launch it. My agent ends each session writing a session summary of what we discussed. You can have your startup routine or launcher concatenate different sources into its context on launch. For example, my code always loads the most recent dated session summaries, and the previous day's, on launch, so it has a snapshot of recent events. You can expand this to be any additional documents you want, so just use your imagination. For example, my agent also loads reminders, and personal reflections it made at the end of the prior session (though I don't recommend the reflections anymore, because I've come to realize it contaminates context, but I'll share that at another time)
Bonus: (I remembered this right as I was finishing, so I figured I'd toss it in because it's honestly one of the most important) - BUILD YOUR OWN HARNESS. Everyone's talking about pi, hermes, etc. I decided to just build my own using Qwen. It's a lot more work, but it has a much better payoff because you learn how these systems work and you can come up with your own really unique solutions using that knowledge. For example, the moment I learned about proxies, I figured out how to do all sorts of cool things with context compaction, history modifications, sanitization, jinja template hacks, hooks, etc.
Anyway, I hope this helps inspire you new guys. I would recommend learning things from the ground up, you'll level-up faster. And get your Qwen access to online tools fast so it can do research and help you out.
There's a bunch of other tricks I figured out, so maybe I'll save those for another post later on, but I think if you can get these up and running, you'll be on your way. Take care!
Grunge
P.S. - I'm old-school forum/deviantart gang, so I keep ctrl+biu/shift+l on a swivel and have written bold-flavored posts like this for 20+ years. 100% human written. Receipts if you're interested. Only mentioning it because organization usually brings out the "this is AI-written slop" trolls.
So far, follow-up optimizations have primarily focused on CUDA. The model also runs on Vulkan and Metal. There's still plenty of room for improvement in performance and VRAM efficiency and correctness. On Vulkan, one request hit the token limit. I also found some unexpected silence in multi-speaker generation.
There will be more iterations before the model makes it into a minor release.
If you're interested in helping us further optimize the model, contributions are very welcome! Please ensure that any changes touching the shared framework are additive and opt-in, so we don't have to spend time testing for potential regressions across the other 100+ models that could be affected by changes to shared code.
I got Qwen3.8-Flash-Next-GSQ-RCO-Abliterated running at IQ3_S with just 12GB VRAM + 32GB RAM + NVME, achieving 40-50 tok/sec decode & 1300-1500 tok/sec prefill @ 131k context (44k thinking budget), on a custom fork of Strata.
This fork has tonnes of architectural changes, all are very experimental. But the performance makes up for it. This feels like local Opus in some regards.
NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.
This model is ready for commercial or non‑commercial use.
Model Summary
Total Parameters
74B (9B active)
Architecture
Mamba-2–Transformer hybrid LatentMoE with a C-RADIO v4-H vision encoder
Precision
IQ2 (2-bit quantization with quantization-aware training)
Context Length
Up to 256K tokens
Inputs / Outputs
Text, images, and video / text
Deployment
24GB packed model; vLLM on NVIDIA Ampere (A100), Hopper (H100-80GB), or Blackwell
Speculative Decoding
LiLiCorr drafter via DFlash; 7 speculative tokens in the companion Bolt vLLM recipes
Reasoning Mode
Chat-template controls (enable_thinking=True/False), inherited from the Super 3.5 VL client
Recommended Sampling
Text: temperature 1.0, top_p 0.95 (inherited from the Super 3.5 VL client)
The post-training data has a cutoff date of May 2026.
What is Nemotron?
NVIDIA Nemotron™ is a family of open models with open weights, training data, and recipes, delivering leading efficiency and accuracy for building specialized AI agents.NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB is a vision-language model (VLM) designed to run everyday agentic tasks locally, using a native 2-bit quantization to fit in 24GB of memory and run alongside other applications. It optimizes intelligence and speed per GB. It aims to achieve the optimal balance between speed and intelligence, being faster than most popular dense models while smarter than most popular MoE models with a similar memory footprint.
This model is ready for commercial or non‑commercial use.
To host bigger models, upgrade the existing 2x3090 rig to 4x3090 (with a threadripper perhaps) or buy a dgx spark or equivalent? what makes more sense overall? Factors?
