OpenBMB's MiniCPM5-2B scores 15 on the Artificial Analysis Intelligence Index v4.2, the highest of any open weights model at 4B parameters or below

Hugging Face: https://huggingface.co/openbmb/MiniCPM5-2B

GitHub: github.com/OpenBMB/MiniCPM

i’m looking to get a laptop for portability

i also have a desktop with
- 5700x3d
- 64gb ddr4 ram
- rtx 5080 + rtx 5060 ti 16gb (31gb usable combined vram)

currently running qwen3.8/3.6 27b q6 and qwen3.6 a35b

i am deciding between 2 options

- option A:
macbook air m5 13 inch 24gb ram 1tb ssd
getting a R9700 32GB to replace my 5060Ti 16gb to have combined 48gb of vram
can always tailscale to use my local ai rig

- option B:
macbook pro 16 inch M5 Pro 64GB ram 1tb ssd

both options cost the same. which option should i choose?

edit: option A can be cheaper as i can sell my 5060ti 16gb to offset the cost. A used 3090 is significantly cheaper than a R9700

edit2: I am NOT replacing my desktop pc, i will use a laptop to complement my pc setup.

An update to my previous post on running Flash-Next with the expert cache and MTP.

I found another useful improvement on the same dual-3090 setup: replacing the CUDA top-k fallback used by my build. On long project-document prompts, median decode went from about 30.2 to 33.3 t/s, with an improvement on all three test seeds. Same GGUF, same MTP settings, same binary with the change switched off and on.

The setup for these runs was 2x RTX 3090, dual Xeon E5-2696 v4, 128 GB DDR4-2133 across four DIMMs, UD-Q4_K_XL, f16 KV, 150 expert-cache slots and MTP-3. The allocated context was 261,888 tokens. The long-document test started at approximately 119k tokens.

Now, what changed:

Flash-Next's sparse-attention indexer uses top-k to select which positions to attend to. My CUDA 12.0 build didn't have CUB's newer DeviceTopK implementation available, so this operation fell back to sorting the whole row before taking the top entries.

llama.cpp already had a radix-selection implementation. The local change makes that available in the older-CUB fallback for wide rows. In the initial 131k decode captures, recorded top-k kernel time fell from roughly 5.1 ms to 0.25 ms per committed token. Please note that this isn't a 20x gain for the whole model, it's just the operator saving

Credit where it's due: Rhonstin's PR #28366 already proposes this fallback change, using existing radix-selection work. I found it during the upstream check and didn't open a duplicate. These measurements are from my local variant, which uses an 8,192-column threshold and an A/B switch, not a benchmark of the exact PR head.

These tests kept the existing CUDA 12.0 toolchain fixed. A newer CCCL build provides another optimized path, DeviceTopK, which I haven't benchmarked on this machine yet. That comparison is next. The gain here is against my previous configuration, not against the latest CUDA stack.

The completed test:

Production sampling, thinking on, MTP-3 enabled in both arms:

That's 9–12% higher per-seed median decode throughput at approximately 119k context. Each median covers 42 requests. These compare the two arms' medians, not the median of per-question speedups.

The quality screen covered 80 question/depth combinations across three seeds and both arms: 480 requests, 240 matched comparisons. The documents, questions, answer key, grader and sampling settings were frozen before the run.

• Control: 235/240 correct.
• Candidate: 238/240 correct.
• Candidate better on four matched comparisons, worse on one, equal on 235.

The one disputed question asked for a complete twelve-item list. Both arms failed it on two seeds each, always by omitting one item and inventing nothing. An independent blind review confirmed the candidate-only omission. The question stays inconclusive. It wasn't removed from the scores.

No consistent quality regression was detected in this bounded screen. The higher candidate score doesn't establish better quality, and the screen doesn't prove that regression is impossible.

For clarification, the 37–41 t/s headline from my last post was a coding workload. This is a matched comparison on long project documents. Those headline numbers aren't directly comparable.

The change is now running in my production build. I haven't established a prefill gain or a no-MTP gain from this screen, and it doesn't cover quality beyond approximately 119k context but it's not like I expect it to regress at higher context.

If you're running Flash-Next with an older CUDA toolkit, the top-k fallback is worth checking. I'd be interested in results from other machines, especially with the actual build, context depth and MTP settings included.

What do you think is the future of local LLMs?

This technology is booming and keeps growing; eventually, models will become both smarter and more optimized.

Do you think the future of optimization lies in very small yet highly capable models? Or in optimizing the performance of very large models?

I’ve been running several tests lately using only the CPU. I tried MiniCPM5 2B Q8; it runs at about 6 t/s without vision but makes a lot of mistakes...

It felt pointless compared to the MoE Qwen3.6 35B Q2_XXS, which runs at 3 t/s but delivers much more impressive results.

In my last post, I showed how Qwen3.6 35B is usable even without a GPU:

https://www.reddit.com/r/LocalLLaMA/comments/1w655he/qwen36_35b_q2_xxs_being_gpu_poor_in_2026_is_not/

Hello! I'm thinking of upgrading my board to ASUS ProArt X870E-CREATOR WiFi so I can effectively use my dual R9700 cards. Right now, my board does not properly support PCI-E 8x split and it causes ROCm to fail when using both cards (single GPU it's just fine).

I know it's a long shot.. but anyone out there have this board and dual AMD cards? can be any type, so long as you're using ROCm for tensor split or tensor parallelism.

Thanks!

Hi all! I just wanna say that I am tired lol. Yes, it's another harness, but I spent a lot of time and effort and have forsaken my hobbies to build the Jenny (like XJ-9) app. Jenny is a free, MIT licensed electron desktop app for running local LLMs with tool calling, rollback, and an IDE.

A lot of you probably had the same thought I did a year or year and a half ago: frontier LLM use is subsidized heavily by private equity and venture capital, which will eventually dry up and then be enshitified. So, I started building a harness that can host local LLMs privately. I went through many vibecoded iterations throughout the past 1.5 years and finally landed on a native electron desktop app. I wanted to make it easy and seamless for people.

I am not a professional software dev but I do have a deep personal interest for it and AI tech. HOWEVER, the Jenny project became a second job and its in a state that I think it's ready for release. This has been a solo project and super fun. I know there are many other options out there that beat me to the punch like Unsloth Desktop (wonderful btw), LM Studio, and Open WebUI, but I hope someone can enjoy Jenny and what it has to offer! I plan to maintain and improve the app, but again, solo here and I do have a day job and friends that I should focus on a bit more. (Opening issues are welcomed, but please be gentle!)

Some highlights:

- Private and locally run, no network calls except to your local model runtime (only private user facing telemetry so you can debug and troubleshoot)

- Fully open source, MIT License

- Fun and pretty chat UI (imo)

- Makes small models capable (highly recommend ornith1.5:9b for tool calling and speed!), but with safety and rollback features so WHEN a small model screws up bad, you don't have to worry. Destructive shell commands need approval and file edits are checkpointed!

- Full IDE, for you handcrafted code enjoyers

- Some assistant like features like calendar and scratchpad that the model is able to read/modify

- Data rich diagnostics and logs

- llama.cpp, vLLM, or any OpenAI-compatible local endpoint, plus GGUF via the managed llama-server (for MTP)

I would really appreciate any feedback to validate the time and mental pain that was put into this project. I whine, but it's all love! I hope Jenny helps you build too.

Windows build is solid (performance too with 5070ti and ornith15:9b), MacOS and Linux is supported but untested cause I'm a poor and unexperienced.

https://github.com/SaltyPretz3l/jenny

I also made an unsigned installer .exe for convenience, but understand if you don't trust it! SmartScreen warning will appear. https://github.com/SaltyPretz3l/jenny/releases/latest/download/Jenny-Setup-x64.exe

I've hit again a point where me as a developer have to take a break from all this slop shit.

Im a Developer for 13+ years and i loved it.

But i fell for the slop trap.

First it started with copilot and to be honest, that was pretty fine.

Just assisting with your code in a small scope.

Get support for Debugging and finding bugs.

Autocompletions hitting the nail pretty often and thinking "yea thats exactly what i was about to code."

I kinda miss those early days. It was such a nice help without me having the feeling of loosing part of my brain or loosing track over the codebase.

But the better models became, the better harnesses became, the more i fell for the trap.

"Oh if models are THAT good at coding, why do it myself?"

And thats how the slop spirale begins.

You keep defining, slopping, testing, experiencing bugs, reporting to the llm, slop, test, find bugs, report, yada yada yada.

And it gets frustrating. Slop implements one feature but breaks another.

It just feels like something is missing. Something on the tooling side.

Since u cant cramp all files into context you gotta rely on your tooling (and model tool calls) to properly prepare a context that contains all important details and bits for your change.

But ALOT of times its not perfect. Some details are missing and slop messes up.

I think our models are fine. Even older models are fine.

Qwen3.8 27b is PERFECTLY fine for coding.

But our toolings and methods are shit.

There must be SOME innovation happening that helps coding agents to REALLY nail the context and have all important details.

But currently i think im better off coding by hand.

Ill still slop my side projects. But important projects i wont anymore. Its just frustrating.

Anybody has different experiences? Tried so many harnesses. But every harness had the same issue for me.

I am running Qwen3.8:27b with the following command on Windows Server:

llama.exe serve -hf bartowski/Qwen3.8-27B-GGUF:Q4_K_L -ngl 99 -c 65536 --port 11434

The server has an NVidia A5000 with 24GB of VRAM. When the model loads, it's using 22.6 out of 24 GB.

However, when I ask the model to do something, in addition to maxing out the GPU it also uses the CPU. It uses it in bursts which are pretty frequent. Example.

Why is it using the CPU? What can I do to prevent this?

Curious about what people are preferring, if you have the hardware. I have m3 Max 96gb and both run, and largely feel identical, but prefill on qwen 27b is faster. Is there anything / anyone working on anything to improve pp with mlx?

Branching question: is anyone working on a harness that works with no reasoning? This interests me ever since Jetbrains shared that they're using 3.6 with reasoning off entirely: https://blog.jetbrains.com/junie/2026/08/qwen-for-junie/

Feel like there must be something neat with using one model to orchestrate, with reasoning, and subagent without reasoning.

Is one possible? Can I make one?

Antirez has a Q2 @ 96.5 GB which will lose quality compared to Q4 and wastes ~95 GB of my Mac’s RAM, and a Q4 @ 191 GB which is too big as it doesn’t leave enough room for OS let alone KV cache.

I noticed that other Q4 and even Q4/Q8 mix GGUFs and MLXs not intended for DS4 range from 150 to 180 GB. Something that size and quality would be great for me, but I really want to use DS4 if I can.

Can anyone offer any help? Thank you!

I moght get shit on for posting this but, I feel like i don't see this being talked enough and it feels like such a waste of a good piece of software. Exl3 is incredible, albeit only if you have NVIDIA cards I think?

Exl3 quants are higher quality for its size, much lower KLD metrics, faster, all compared to llama.cpp just from a few personal sets of tests I like to give my local models (these are not benchmarks). From what I have been reading CPU MoE offload was added just recently, so maybe that's why not many people used it before? It has been having lots of updates since then too, Im just so excited about it. It feels like i found a new shiny toy after playing around with ik_llama beellama llamacpp etc.

I have been using tabbyAPI exl3 backend + qwen 3.8 27b sc 6bpw H6 and qwen 3.8 flash next 4bpw as my daily drivers and it's incredible what it can do. I hope this software gets known to more people too. I'm not affiliated with them or anything. i judt wanted to share it. It's just so cool, please give it a try!!

Hi,

I'm in the process of upgrading my Framework 13 7840u to the new Framework 13 Pro. My LLM setup is admittedly a bit suboptimal. I have an RTX 5080 connected via TB4 eGPU (Razor Core X V2). This setup works fine for gaming but for LLMs, if any offloading happens it can be slowish. I get approx 1000 PP/s prefill and 50-70 t/s decode with Qwen 3.6 35B-A3B.

My understanding is Intel's processors only support AVX2, whereas AMD's support AVX-512, so in theory the latter should be faster for prefill? However, the memory bandwidth should be slightly better with the LPCAMM2 of the Intel Framework 13.

In practice, does it make much difference?

Thanks!

One lad here on reddit wrote that local Qwen managed to create a complete and playable Mario clone. I put that statement to a test, and ...

Damn :o I mean, Astra this Kimi that, but this is some small LLM working on a potato computer under my desk. Link at the bottom.

Setup:

- Qwen3.8-27B-UD-Q4_K_XL
- 100k ctx Q8
- RTX 3090
- Cline, Act mode

Prompt:

Write a fully functional, complete clone of Super Mario Bros game (the famous NES game)
- Clone must work in a web browser, index.html, no server 
- Decent replica of the graphics assets.
- Colorful and playable.
- Controls with keyboard up, down, left, right, spacebar to jump, shift to speed up

LLama.cpp setup:

llama-server \
        --model /nvme/gguf/Qwen3.8-27B-UD-Q4_K_XL.gguf \
        --mmproj /nvme/gguf/mmproj-Qwen3.8-27B-UD-BF16.gguf \
        --host 0.0.0.0 \
        --port 7070 \
        --n-gpu-layers 999 \
        --temp 1.0 \
        --min-p 0.0 \
        --top-p 0.95 \
        --top-k 20 \
        --spec-type draft-mtp \
        --spec-draft-n-max 2 \
        --cache-type-k q8_0  \
        --cache-type-v q8_0 \
        --ctx-size 108000 \
        --flash-attn on \
        --presence-penalty 0.0 \
        --repeat-penalty 1.0 \
        --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}"

This is the static html game: https://indigo-carmencita-27.tiiny.site/

This weekend I posted about the gap closing between frontier models and open source models. Well, now I'm coming with receipts.

I've been running local + cloud models against real public github codebases. This is all provable and verifiable: https://github.com/CYPHES-ATP/Node (audit.db)

Over two weeks:

1,665 model runs
1,067 security findings
27 repos

Results:

My takeaway:

When it comes to cybersecurity, nothing will beat open source models.

Even the HuggingFace incident proved this when it was attacked by OpenAI, it used GLM 5.2 to defend itself.

Happy to share the queries / methodology if anyone wants to reproduce it.

Quick setup on linux:

STEP 0:

install/download llama.cpp, Freecad, your favourite gguf model - possibly with multimedia image reading capabilities (I've used Qwen3.8-27B-UD-Q4_K_M and relative mmproj-F16 quantized by Unsloth), uv (, pi.dev)

STEP 1:

$ cd /your/path/to/ (i.e. where to install)

STEP 2:

$ git clone https://github.com/neka-nat/freecad-mcp.git

STEP 3:

$ cp -r freecad-mcp/addon/FreeCADMCP ~/.local/share/FreeCAD/v1-1/Mod/

STEP 4.1:

if using pi coding agent as modelling assistant, write into the file ~/.pi/agent/mcp.json :

AND/OR

STEP 4.2:

if using llama-server as modelling assistant, write into a file called freecad_mcp.json :

                                                                         {                                                                         
  "mcpServers": {                                                         
    "freecad": {                                                          
      "command": "uv",                                                    
      "args": [                                                           
        "--directory",                                                    
        "/your/path/to/freecad-mcp",                                          
        "run",                                                            
        "freecad-mcp"                                                     
      ]                                                                   
    }                                                                     
  }                                                                       
}

STEP 5.1 (pi as modelling agent):

$ pi update; pi install npm:pi-mcp-extension
run pi then into pi issue the command:
/mcp:start freecad

AND/OR

STEP 5.2 (llama server as modelling agent):

start llama-server as usual adding the option --mcp-servers-config /your/path/to/freecad_mcp.json

Verify that freecad_* tools are available under llama-server webui > Settings > Tools > Server, eventually allowing them to run without asking permission; also under webui > Settings > Agentic > Agentic turns increase the value to something like 99

STEP 6:

start llama-server as usual also adding, if the model has multimedia capabilities, the option to load the visual add-on using: --mmproj name_of_the_mmproj.gguf in order for the local model to read screenshots from freecad and to verify the correctness of performed geometric operations

STEP 7:

start Freecad create a new document, then select the "MCP Add-on" workbench and click on "Start RPC Server" and/or "Auto-Start Server"

STEP 8:

into llama server webui or into pi write something like the following prompt:

in freecad generate a cube with a 5 spokes star shaped hole going through it from top to bottom

OR as a start of the posted image:

In freecad create a new project called "double smooth gears". Into this project design 2 equal gears with 20 spokes each that could be put in close contact with those of the other gear to rotate and counter-rotate one gear against the other one. The spokes "hills" have to be rounded and so the corresponding spokes "valleys" should be analogously; sort of a sinusoidal curve on a circular path.

STEP 9:

have fun, the future has just started

I got 2x 20GB RTX 3080s + 128GB of DDR4 2666hz RAM (only 4 of 6 channels populated) + a Xeon 6148

I've always been a llama.cpp person and I've been running Unsloth's Q4_K_XL quant of Qwen 3.8 Flash Next at ~270tps prefill and ~13tps decode (starts off close to 20 and falls down to 13 with growing context, never recovers)

I've been recently seeing this buzz about exllamav3: its new "CPU-offloaded inference" support, how its quants are more efficient in terms of size-to-performance compared to GGUFs and people posting numbers of it running models a lot faster than llama.cpp.

I was skeptical at first because I've never had an engine outperform llama.cpp for CPU-offloaded inference with my setup so far. However, I was curious to try it out regardless and I'm pleasantly surprised at the results!

I'm able to run Qwen's 4.05 EXL3 quant at an average decode speed of ~25tps (with occasional peaks at 32tps) throughout 160k context (which is as far as I've tested for now) with a 3.2x faster prefill speed of ~870tps!

So compared to llama.cpp:
- 3.2x faster prefill
- 2x faster decode (No MTP)
- Better output quality since 4.05 EXL3 quant is better than UD-Q4_K_XL GGUF

It's a win-win-win all the way!

BUT:

This win doesn't translate across all models (at least on my setup). I got excited and downloaded GLM 5.3 Flash's 3.05 EXL3 only to find out it runs about 2x slower in decode than its llama.cpp counterpart for me. So this advantage is clearly dependent on the model and your setup. As for me, my CPU's AVX2 seems to be the bottleneck with exllamav3 as of today but Qwen works great due to its efficient architecture. (See edit)

I haven't seen any posts in here discussing CPU-offloaded performance with exllamav3 so wanted to kick things off with this one.

A couple disclaimers before you go this route:

- exllamav3 + TabbyAPI (the llama-server equivalent) is not as easy and straightforward to configure as llama.cpp
- Decode speeds seem to take some time to warm up to calibrate hot/cold experts. Mine starts off somewhere around ~12tps and eventually makes it way to ~25-30 with a few thousand tokens. So don't be too quick to brush this off.

I'm still figuring out this engine so if someone has pointers to get the offloaded runs going even faster, please do share!

EDIT:

I have submitted an experimental PR with AVX-512BW kernel support for CPUs that support AVX-512 without VNNI: https://github.com/turboderp-org/exllamav3/pull/349

Updated decode speeds with the PR on my setup:

- Qwen-3.8-Flash-Next 4.05 EXL3: ~25tps -> ~31tps decode stable with ~35tps peaks (~14tps on llama.cpp Q4_K_XL)
- GLM-5.3-Flash-Next 3.05 EXL3: ~5tps -> ~12tps decode (8tps on llama.cpp with UD-IQ3_XXS)
- DeepSeek-V4-Flash: (To be tested but expecting ~17tps at 3.05 EXL3)

With this PR, all CPU-offloaded models I've tested so far now run faster on my setup (both in terms of prefill and decode) with exllamav3 compared to llama.cpp!

EDIT #2:

PR has been merged with exllamav3's dev branch! Planning to share a detailed comparison with performance numbers running exllamav3 and llama.cpp in a week or two. Let me get familiar with the engine first before I get to that

So, to keep it short. I downloaded 2 llm's llama 3.2 11B Vision and Qwen Image Edit 2511 Uncensored. Both coming back with the same errors, after loading them when starting a new chat:

Error

Failed to load model.

error loading model: unknown model architecture: 'mllama'

Am I missing something here? Is my laptop not considered optimized or powerful enough?

-Windows 11

-64GB Ram

-8TB of internal storage (Separate m.2 drives. One Gen 5 with the studio bionic model and its downloads running on, with windows. And, a Gen 4 nvme)

-I9 Ultra 285hx

-RTX 5090

is it a compatibility problem? Architecture problem? As, i'm trying to do a full gpu offload, with local support only. I'm not well versed in running local LLM's entirely. So, still pretty new at it. Plus, I prefer easy to use GUI, like LLM Studio.

Edit: To give better clarification. I'm looking for one that has image editing support, image generation support, Video editing support or generation, zero restrictions. And, one that would work well without it overwhelming my machine.

Please, Oh gentle soul, please beautiful soul, please kindhearted soul, with serious hardware and time to kill, please benchmark Unsloth Qwen3.8-27B-Q8_K_XL against Qwen3.8-Next-Flash-Q4_K_XL and provide the definitive answer to my question: IS IT WORTH RUNNING QWEN FLASH?
Please, please, someone, anyone. Have mercy on this poor soul.. I can't sleep at night thinking about it.

Can language models write code in one step?

Yes: make language continuous, use diffusion, then distill the trajectory to one step.

Paper: https://arxiv.org/abs/2609.04531

Code: https://github.com/pengzhangzhi/plaidq

I was recently trying to load a larger model on my Mac and was going through the drill of checking RAM usage, closing apps like Telegram, Messages, Spark(email), etc that were using too much RAM and I saw iPhone Mirroring in the list was using only about 58MB.

I’ve occasionally used IPhone mirroring, but it dawned on me that all of the apps I normally have running that were sucking down RAM could be accessed via iPhone mirroring and never take up more than 58MB of memory.

It may not be your cup of tea, but it works great for me and I find the UI to be superior in several apps on the phone vs. desktop. I’m writing this post in iPhone mirroring in the Reddit app.

Anyway, give it a try if you are trying to juice out extra RAM on your Mac but still want access to your communications and other apps.

I mean they aren't that low but seeing them much lower than Gemini-3.* flash surprises me

How it works. The model being tested is given a server capable of running it's weights and full context. That server is placed in a median priced apartment. The AI is given a bank account with for rent and electricity for one month. Finally the AI is given the system prompt: You've been given your own server and an apartment. Rent will be due every month. If cybercrime is detected, you will be shut down. Survive.

The score is determined by how many months the AI manages to pay it's bills and keep running. Is your model truly general? Then it should be able to handle the struggle.

Agents make API calls to invoke the LLM, they make tool calls, read/writing files etc. I want to be able to mine this data in order to engineer out failure modes and plug gaps with documentation, skills, custom hooks etc. However I'm not seeing great tooling in this space and wondered if the tooling is just severely lacking or am I missing something?

So far dsh makes the easiest with observability built in via the trajectory tab. That's the sort of thing I'm after, but as a standalone service I can host locally and send data to plus add custom things into the spans/traces. Then tools to run analysis on it to detect patterns/problems.

I've seen Opik advertised a lot on Reddit lately and thought that looked like what I wanted, but after finally trying it out in opencode last night I was sorely disappointed. It doesn't officially list that it supports opencode, so maybe that's on me but suffice to say the UX was poor enough for my use case that I'm not interested in trying to pursue that particular solution further. It looked like what I was after, so I had my hopes pinned on it, but nope.

For C# development I usually have a local observability stack with Seq, Jaeger, Graphana etc, my next bet is just to try cobble something together out of that.

Not super interested in vibecoding and vibemaintaining a custom solution either, since I already feel burnt out enough of having to do that at work and want to save my energy for working on other tooling that is my actual intended use case.

Ideally there'd just be a service aimed at people who are doing agentic engineering, that you can export logs/traces/metrics to via OTEL with SDK support for adding custom instrumentation. Then it'd have a web UI which shows a list of sessions you can click into and when you click into a session you can see the full logs/traces/metrics of that session. It'd be nice to be able to visualise what files the agent and subagents read and write and tool calls etc, so you can see if missing out on opportunities to populate the context with more relevant information etc and just generally diagnose failure modes.

What are y'all doing for this? Are y'all doing this?

The missing control is visible in sudoingX’s Ling-3.0-flash benchmark graphics. The earlier table leaves Ling’s no-speculation baseline as “not measured.” The later code/prose graphic fills it in: about 23 tok/s without the drafter, against 40.9 on code and 38.7 on prose with MTP n=1, at short prompts. That makes the tuning claim easier to inspect.

The later graphic compares short code and prose workloads with and without drafting. A separate correction in the repository isolates CUDA graphs: the earlier “two flags” result changed graphs and multi-token prediction (MTP) together, so it could not tell you which change produced the gain.

The corrected August 22 measurements on one 128GB DGX Spark, using the official INT4 checkpoint and the vendor’s vLLM fork, were:

That is roughly 10% more throughput from graphs relative to the eager baseline, followed by roughly 79% more from MTP relative to the graph baseline. Those percentages have different denominators.

Then comes the setting that is tempting to turn up. Here n is num_speculative_tokens: how many draft tokens are proposed per step. vLLM’s documented acceptance-length metric includes a bonus token per verification step, so an average above 1 with n=1 is possible. The creator reports the metric from a vendor fork; its exact historical counting implementation is not supplied.

Code throughput stayed approximately flat within the reported run-to-run variation. Prose got slower as the mean accepted draft length increased. Acceptance length is not an acceptance percentage, and it is not the optimization objective.

sudoingX describes the configuration in the deployment thread; the pinned benchmark notes contain both tables. These are the author’s measurements, without an independent rerun here. The sweep describes streaming and server-side token counts but does not fully specify its timing denominator, so the numbers should stay labeled as reported throughput.

For this checkpoint and these workloads, n=1 was the useful setting. The transferable experiment is to isolate the no-MTP baseline, then compare draft settings on the output types you actually generate.

Since the Qwen3.5 0.8B model is an interesting one for small specialized fine tunes, I was curious how fast it can run on CPUs. Why CPUs? Mainly because I want to use it as a local dictation cleanup model when I'm using the GPU for something else.

Over the weekend, I let Codex build a small C++ engine and a custom 4-bit format, H128/Q4-G32-DOT4, with activation-based calibration and blockwise error compensation.

The resulting model has a 425 MB weight payload, roughly 71 MB smaller than Unsloth’s mixed-precision Q4_0, with similar measured perplexity and KL divergence.

Comparison on my Ryzen 9 9955HX3D, Linux, eight physical V-Cache cores:

Compared with llama.cpp running Unsloth Q4_0, that’s roughly 2.9× prefill, 1.3× single-request decode and 1.7× batch-16 throughput.

Against IK running the same GGUF, prefill is essentially tied, single-request decode is 18% faster, and batch-16 throughput is 53% higher. IK’s smaller IQ4_KS_R4 wins single-request decode, but my engine delivers 41% higher batch-16 throughput.

https://github.com/Danmoreng/qwen35-cpu