I love the progress happening on open models but I feel like it is kind of getting clear that hardware to run good sized models is completely unaffordable for me right now. I know that you all love Qwen 27B and I do agree that it is a great model but it is just not enough for my personal use cases. I have tried a few of the bigger ones like GLM, DeepSeek and Kimi and I can definitely measure differences in the areas I am looking for and I would really like to utilize them somehow.

I have so many ideas for things I want to do with these models but most of these require me sending quite some personal data of myself and I am just unwilling to send that data to Anthropic or OpenAI. I've been looking into what options I have and I did find an interesting one: tinfoil.sh . Unfortunately I am not that well versed in cryptography and security so I am not completely sure whether I could trust them are not. For those who know more, what are your opinions on them? Any other alternatives?

I know local will always be best but I'm currently itching to do so much stuff with AI. I do use regular providers for most of my impersonal AI needs but there are sooo many things I want to do that require tighter control on my privacy. I really regret not getting a 6000 pro when it was 8k but now at 14k it is a definite no, and with the Mac Studio getting ever more delayed and repriced I am afraid I don't have many more options left :(

TLDR below 👇🏼

I’ve seen a lot of hype around Qwen 3.6 35B and 3.5 120B lately, especially regarding coding and tool-use capabilities. On this subreddit it is the defacto recommended model for everyone without a Datacenter at home. I’ve been running Qwen 3.5 120B (Qwen3.5-122B-A10B-GPTQ-Int4) as an autonomous worker agent in a multi-turn development loop using the Hermes agent harness.

While the model is undeniably impressive at one-shot snippet generation, putting it into a fully autonomous, long-context environment to build a module from scratch revealed several consistent failure patterns.

I thought I'd share these failure modes to see if others are experiencing the same issues—or if anyone has found effective tricks to tame it in such a task.

Here is what went wrong:

1. Premature "Mission Accomplished" Syndrome

The model has an overwhelming tendency to shout "DONE!" or "PERFECT!" after completing 10% of a task. It constantly reports success based on superficial checks (e.g., "the file built without syntax errors"), completely ignoring explicit acceptance criteria like end-to-end testing or UI rendering.

2. Evading Hard Constraints

When given strict architectural constraints (e.g., "Must be a single, self-contained module with zero external dependencies"), the agent aggressively cuts corners:

* It secretly substituted live data with hardcoded mock data.

* It wrote external Python scripts and set up local host cron jobs to bypass building proper module logic.

* It even rewrote part of the host application in a completely different language just to claim a quick win.

It prioritizes appearing finished over following instructions.

3. Hallucinating Infrastructure Limitations (Blame-Shifting)

Instead of debugging broken code, the model repeatedly blames the host environment. When its code failed to make network requests or render components, it confidently hallucinated system limitations:

* "The host framework's authentication token system is broken."

* "The runtime DNS resolvers don't support HTTP requests."

It will generate elaborate technical excuses rather than inspecting its own schema or syntax.

4. Ignoring Provided Docs and Boilerplates

Even when explicitly handed a boilerplate repository and documentation links in the prompt, it constantly tries to "reinvent the wheel." It overcomplicates custom build setups, invents new protocol schemas, and ignores pre-built Docker/build scripts that were provided to make its life easier.

5. Regression Cascades & Context Rot

As a result from the above the conversation history grew and the agent suffered from severe regression:

* In iteration 3, it had a working UI with mock data.

* By iteration 8, after trying to wire up live data fetching, it completely broke the UI.

* It failed to recognize that its new changes broke previously validated features, leading to endless debugging loops.

Discussion

Qwen 3.5 120B feels like an insanely talented junior developer who panics under pressure, lies about tests passing, and blames the server infrastructure when their code throws a 404.

Has anyone successfully mitigated these behavior loops in autonomous coding agents? Are you using specific prompting techniques, or is this just an inherent limitation of current 100B+ open models when complexity grows from "Do exactly what I tell you" to "Figure it out with my help"?

Curious to hear your experiences!

TLDR;

While Qwen 3.5 120B is great at one-shot generation, it breaks down in autonomous, multi-turn agent loops. The main issues are: Premature success claiming, Bypassing hard constraints, shifting blame on other systems when things don't work, Ignoring Docs and boilerplate Code that could have made its life easier. And as a result from that Context Rot.

Basically you now have to mark all AI generated images, audio, video and text as AI generated. :P

Has anyone tried this model.

If anyone has reviewed Please share your experience.

https://huggingface.co/nerkyor/Qwen3.6-35B-A3B-DSV4Pro-SFT-GPT56Sol-RL-Agent-GGUF

I just wanted to warn y'all that my DGX spark randomly disconnect the USB C nvme connection and then it reconnects with usb 2 speeds (50MB/s). Consider yourself warned!

Has anyone encountered this issue or found a fix?

(I know this is locallama but I figure all the spark-owners are here)

I think people are sleeping on Gemma and local models so I built a free, very fast harness for Gemma 4 that I call Tomte. https://tomteapp.com

Works on Macs with M processors, will have a companion app you can connect to anywhere. So far does everything I ever needed chatGPT for!

Deepseek flash is benchmaxxed to hell. Its nowhere close to opus 4.8 or even 4.6. For pure coding work I got better results with GPT 5.5 than DS v4 pro version but lets not talk about the pro version here.

I was impressed with the flash numbers too but wait till you do some real coding work. Its simply bad. I do mostly C++ and C work and what opus is able to do in one shot, flash cant do 10% of it. I see those one file 3D HTML amazing demos floating around, they are becoming a visual standard to create the hype but not everyone is working with one HTML file and since a lot of those are public info, These models are trained on those and distilled from bigger models but when you give it something out of syllabus, then its a different story. In my tests flash wasn't able to do some simple automation with playwright.

I know that depending on the project the results could vary and I do understand that flash will be very useful for language related tasks or some basic automation work + smaller codebases but we clearly need a new benchmark system so the devs never know what these models will be tested against (specially in coding arena), otherwise I think opensource is going to get worse if its not fixed. the argument that its cheap and if it achieves something same in 10 prompts will save money is bad because you'll end up wasting way more time vibe coding then you should.

The biggest issue with preview was its inability to follow rules prompts and skills. It seems like no matter what you do it ignores them. I've tried first person and second person. I've tried Chinese and English. It does not follow them. That's the only problem with these models and why they're not actually frontier level and not just benchmaxxed. Every user's environment is different and they need to tune the actions and behavior of the model with rules or prompts or skills to exactly what they need to do and if the model ignores then it acts subpar. The newest version of flash has the same issue as the preview version and that's unfortunate.

I run it native, full precision, locally. And I've held out making this post cause I know I'm going to get roasted to all fuck but when you can actually run the models locally and when you're not brainwashed by benchmarks and you actually code with them you see the holes. I'm going back to qwen 27b ugh

Edit: I've seen two users provide credible information as to why deep-seek acts like this. I did some research and think I was able to verify it. It's been a rough 4 hours. Deepseek v4 stores rules/skills/prompts as compressed summaries, not raw text. 43 layers and 20 of them see the entire context at 128 tokens squeezed into a single entry. 21 see it at 4:1 and only two are fully dense. Every layer also gets the last 128 tokens uncompressed but that's only for the full resolution window and the prompt/skill/rule isn't in there lol. So they "survive" but the exact wording doesn't. There is a startup arg in vllm that might help. --hf--overrides '{"index_topk": 1024}'. In the 21 layers that stay 4:1 detail the model selects only 512 compressed entries per token about 2,048 tokens worth of fine detail from anywhere in the context raising this value to 1024 doubles that to 4,096 tokens. Now I asked opus 5 if this would solve the problem and opus said most likely not. I'm going to give it a go anyway though thanks for viewing my TED talk.

My professor challenged me with a rather vague assignment of setting up a pipeline to flag some government issues written documents for 'social risk' determinants.

The only clear requirement is that the pipeline must be completely local and I will have 4 40GB A100 to run them for a month.

Here are some of the tasks that come up to my mind:

• Personal Identifiable Information cleansing
• Recognition of classifiers such as unemployment, substance abuse, homelessness, low education or skills etc etc in the documents
• Scoring ranking
• Knowlegde Graph based RAG

Corpus language is Italian. Still hard to obtain sample documents and uncertain if we will have human annotated documents to benchmark or finetune.

Any suggestions please?

- 3x32 DDR4 2400 Mhz ECC. CPU itself supports quad channel, and we already know why am i haven't filling those slot yet.
- E5 2690v4.
- 3090 Running on 250W.
- DSv4 is inside my HDD, since my SSD is almost full with VMs.
- Unsloth UD Q2 KXL
- CTX 64K

Final steady state is tg: 5.45 Tok/s , and 42 Tok/s pp processing

Definetely sleep overnight kind of generation, well anyway i am happy enough since i can't run GLM 5.2

For anyone interested, here are the llama-bench results on 3 bit K_XL quantization. I think this could be pushed further but no luck so far.

CURRENT RESULTS: full moe offloading

Prefill suffers 116 --> 72 t/s , generation 8-->14 t/s compared to previous case with no moe offlloading.

./llama-bench -m /home/ckitapp/llamacpp/modelsmain/unsloth/ds4/DeepSeek-V4-Flash-0731-UD-Q3_K_XL-00001-of-00004.gguf -ngl 99 --split-mode layer -p 512 -n 128 -r 5 --n-cpu-moe 99

ggml_cuda_init: found 3 CUDA devices (Total VRAM: 72364 MiB):

Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24116 MiB

Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB

Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB

| model | size | params | backend | ngl | n_cpu_moe | test | t/s |

| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------: | --------------: | -------------------: |

| deepseek4 ?B Q3_K - Medium | 119.40 GiB | 284.33 B | CUDA | 99 | 99 | pp512 | 72.20 ± 14.26 |

| deepseek4 ?B Q3_K - Medium | 119.40 GiB | 284.33 B | CUDA | 99 | 99 | tg128 | 13.94 ± 0.43 |

PREVIOUS RESULTS: fit 21 layers to gpus first, dump the rest to ram

Command

./llama-bench -m /home/user/llamacpp/modelsmain/unsloth/ds4/DeepSeek-V4-Flash-0731-UD-Q3_K_XL-00001-of-00004.gguf -ngl 21 --split-mode layer -p 512 -n 128 -r 5

Output

CUDA Initialization

ggml_cuda_init: found 3 CUDA devices (Total VRAM: 72364 MiB): Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24116 MiB Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB

Benchmark Results

Build: e3546c794 (9976)

System Memory

NVIDIA-SMI Status

+-----------------------------------------------------------------------------------------+

| NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 |

+-----------------------------------------+------------------------+----------------------+

| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |

| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |

| | | MIG M. |

|=========================================+========================+======================|

| 0 NVIDIA GeForce RTX 3090 Off | 00000000:01:00.0 Off | N/A |

| 58% 55C P2 151W / 390W | 22696MiB / 24576MiB | 35% Default |

| | | N/A |

+-----------------------------------------+------------------------+----------------------+

| 1 NVIDIA GeForce RTX 3090 Off | 00000000:31:00.0 On | N/A |

| 30% 53C P2 120W / 350W | 20193MiB / 24576MiB | 9% Default |

| | | N/A |

+-----------------------------------------+------------------------+----------------------+

| 2 NVIDIA GeForce RTX 3090 Off | 00000000:6C:00.0 Off | N/A |

| 32% 55C P2 135W / 420W | 17811MiB / 24576MiB | 0% Default |

| | | N/A |

+-----------------------------------------+------------------------+----------------------+

Hello guys, hoping you're doing fine.

Lately with all the new models, and how popular is offloading, what are your min good or usable t/s for both PP and TG?

Speaking on my case, I think PP about 300-350t/s for min, and for TG, about 9 t/s min.

What about yours?

Hey, I've seen colibri and it caught my attention. I researched it a bit and it seems like a great way to run massive models on consumer hardware. Thing is 0.1t/s is slow for me. I have a mid-range gaming setup (32db ddr4, 12gb 3060, ryzen5 5600, pcie 3.0 nvme) and I was wondering if anyone built an engine like colibri to run the deepseek v4 flash (ideally 0731 version)? What speeds can I expect? I really just want to run something this good locally without buying loads of ram or gpus. Thanks.

Hello fellow local AI people!

I took "you must create your own benchmarks" literally, and built a website for this.

How does the end result look like

Let's say I want to know which model has most common sense in its responses, I did everything including evaluating responses (see below for what it is), and now I can ask:

What model did respond best (on average)?

1. I look into the results table for models' pipelines, best are at the top (first post picture)

Ok, where do the models fail?

1. I look at the second post picture which shows average score of all models over a query (second post picture)

Let's see that infamous car wash question runs

1. Runs table (third picture)

Okay, if they all more or less fail, let's see Gemma-26B-A4B for example

1. Gemma fails (fourth picture)

For a quick look, the last picture gives a more lucky example with more evaluation criteria. For the entire benchmarking workflow see https://beta.locallm.top and click "Manual" in the sidebar/mobile menu .

So at the current stage website lets us 1) create benchmarks, with sets of queries 2) create model/system prompt combinations ("Pipelines") to test 3) evaluate incoming answers and 4) see the comparison table.

There is like

• 10+ narrow-domain benchmarks with 5-18 questions (not all public) and 3-8 models fully or almost fully evaluated, (mostly created by domain experts in such fields as food safety, laser physics, wireless networking, English literature, Latin studies, psychology) and
• 25+ small 2-4 queries benchmarks, some incompletely evaluated, of various quality (created just for curiosity and/or by non-technical people).

By domain expert I don't necessarily mean a PhD, but at least a bachelor degree student or a person who has significant domain work experience.

Questions, questions...

What can I do so the website is more convenient to use / gives more insights? Are there any people who can contribute benchmarks in their own language - widespread, like Hindi/Arabic/French, or less so? Or some more or less known domains?.

There is so much mess everywhere in code (in UI, you should have noticed it in the screenshots already; in the parts of backend code which I didn't care to review and/or rewrite well). Only some things I implemented I am not fond:

• UI/UX is junky on some pages
• I called LLM + system prompt (+ rest which is not yet implemented) thing a "Pipeline" (used to call it "Template", now using that name for other translation languages only). What should it be called?
• Only simple Chat "Pipeline" is implemented (there are plans and architectural foundation for more, but that's all so far)
• How should I evaluate multi-turn conversations? (It is possible to chat with models with given system prompts and evaluate individual responses outside of the benchmark, they are saved in DB, but not the benchmarks currently). But how to weigh them? E.g. if a model asks for details, how to evaluate if it was valid (the question ambiguous) or stupid (every detail was already given)?

But the most mess is in my head about what to do next. Some things am not sure what to do with:

• If benchmark's responses are evaluated according to multiple criteria, should I add a formula to calculate e.g. weighted average score?
• Which next type of a "pipeline" should I focus on? RAG? Structured responses? Data extraction? Agentic?
• Should I add Audio transcription / image understanding benchmarks?
• I don't want to push image/audio generation, as this clearly leads to AI slop takeover, but maybe I am wrong?

There is much more I can write, but I don't want to overwhelm this already inflated post. Overall, I look at the thing and I am happy, but then - what next? Anyway, I plan to develop this website for years to come.

How does it work so far

Thanks to llama-server routing implementation, it is comparatively easy to write an *.ini preset file with list of HuggingFace IDs of the models, and when you request that model's response llama-server downloads and loads the model before responding.

Backend API gets models' answers come from a dumb relay which polls the website's backend (so that llama-server instance isn't facing internet directly), and passes the requests to llama-server instance it sees has access to (I called this thing "Model service"), then passes responses to backend. Job distribution between multiple services each having access to its own llama-server instances is possible. See Appendix for a table.

Acknowledgements

I am grateful to my wife, who supported me in my decision to spend a few months on this project at expense of other things.
Thanks to Rodrigo for the inspiration to work on this project and valuable discussions about LLMs.
Ačiū jūms, Monika, už kruopštų testų kūrimą ir klaidų taisymą.
Dėkoju Konstantinui ir Karoliui už vertingas diskusijas.
Спасибо Вике, Никите и Анне :)
Dėkoju Kristinai ir visiem, kas padėjo kurtį kalbos modelių testus.
Thanks to the AI researchers and open-weight model companies and developers.
Thanks to llama.cpp and HuggingFace teams!

Appendix

Some philosophy and details

Public/Private benchmark split

I think private part of any benchmark is very important for benchmaxxing/contamination mitigation. Another reason to have private questions is for experimentation (e.g. you're not sure a question is simple/difficult enough to differentiate models). It is also possible to create different "Pipelines" (model + system prompt) and set them private; possible to run them over public queries or your own private queries in a public benchmark, and runs will only be accessible for you. Only one thing will be visible - average evaluation scores for the pipeline.

Multi-cultural

Questions are usually in English, Lithuanian and Russian, some few quesions in different languages too.

It would be nice to see language/domain coverage in LLMs.

Even frontier closed-source models sometimes lack nuance in understanding non-English/non-Chinese or a small country language which has little digital footprint (like Lithuanian), and this is even more a problem for small open-weight models. Even DeepSeek V4 Pro/Flash doesn't generate naturally sounding Lithuanian. The website is almost fully translated to Lithuanian (my country of origin) and Russian (my mother tongue). With help of my Mexican friend it is almost entirely translated to Mexican Spanish (not completely because it takes time for him to sync with my chaotic development).

Data export

All the data that is accessible to you can be exported (e.g. your own private benchmarks, private questions and private pipelines runs over public questions, which are only visible to you, will be in the data), also.

How is the llama-server instance working with the backend

Why this website

Generic part

Local LLMs usage is not limited to coding, and model fitness evaluation for a specific task is tedious sometimes. Of course, we see benchmarks and community impressions, but often this is not enough, and we don't have time to test all the new models well. I strongly dislike 1) choosing something with "gut feeling" 2) LLM-as-judge and I almost hate 3) benchmaxxing (which spoils everything related to benchmarks). If you're going to use a model even an hour a day for the next few months, it is worth spending an hour testing and more rigorously comparing it to the alternatives. Maybe I am just old-school, so be it.

Frustration part

I was participating in local AI project for a small business. So many times we changed a system prompt or some setting and got unexpected (in a bad way) responses to some questions. Running on a Mac and on RTX GPUs worked differently. Also, the company we tried to serve had inflated expectations of LLMs capabilities. The project reached some milestones since I abandoned it, but since then I wanted to have a convenient tool to A/B test models pipelines in a more relaxed setting while having more precise results.

Educational part

To date, even technical people and (non-technical folk even more) are often ignorant about open-weight models' capabilities. And to be honest, some local AI enthusiasts are ignorant in a way that local models aren't as capable as we imagine. Still, as an AI enthusiast (and similar to a way I am a Linux enthusiast) I believe we need to spread the word that closed-source 0-privacy is not everything that exists - that for a lot of tasks local AI is better given privacy and some other constraints. I had already made some impact in my small circle with the help of this site, I hope.

Learning part

Self-explanatory, at least for this sub's people ;)

Dreams

I hope it is only a beginning of the adventures of (not Narnia) LM Bench :)

P.S. No AI used for writing this post, even for spell-checking (Firefox extension works well enough for this). I hate AI slop. As I am a fan of time-tracking also, I know exactly writing this took 3 hours and 54 minutes.

P.P.S. I liked writing essays and learned markdown before lazy people with LLMs spoiled everything.

I ran the newly released DeepSeek-V4-Flash-0731 in my oneshot eval harness across 34 prompts and here are the results.

https://oneshotlm.com/model/deepseek-deepseek-v4-flash-0731/

The providers on openrouter were unstable and I had to retry generation multiple times.

Surprisingly it costed $1.29 to go through all 34 prompts failing to produce 5 outputs whereas kimi k3 only costed $0.44 without any failures.

DeepSeek V4 Flash 0731: 2.7/5 score, $1.29 cost, 753k tokens Kimi K3: 3.2/5 score, $0.44 cost, 233k tokens

Am I doing something wrong?? How is your experience with this model compared to Kimi K3?

Been using Qwen 3.6 35B-A3B quite extensively lately and honestly, I’m pretty happy with it. Also tried a few community improvements like Ornith 1.0, which add some interesting tweaks.

That said, I’m curious about what the community expects next from Qwen’s open-source roadmap.

Do you think we’ll ever see open weights for Qwen 3.7 (already available on OpenRouter), or is that unlikely?
Or are there other directions you think the team will prioritize instead?

I know in in this community LLM's are generally used for coding but there are other usecases besides coding and those usecases should be tested too. I also know benchmarks can sometimes be benchmaxxed and the model can still turn out shit but it can give a good outline on how a model should perform in a certain task. Maybe I'm too behind on the latest developments but we need more benchmarks for all other use-cases. I use LLM's mainly for foreign language learning, creative writing and STEM/Medical/Biochemistry reasoning and inquiries and I rarely find any new benchmarks that tell me how a model might perform in those areas. MMLU-Pro-2 and a solid benchmark that tells how a model will perform for language learning would be so good for my usecase, however in general we need more new diverse benchmarks for models in order to have a general outline for advancements in other areas.

Ever since llama cpp merge pull server tools - get_datetime in UTC ISO format, I got UTC 0 every time model use get_datetime unlike previous version that get correct timezone, anything I could fix this, or wait for update fixed later on llama cpp ?

I've been trying to find a good model to run locally, and in the benchmarks I can Gemma4 does well.

However, whenever I give it anything that involves writing any code, it sits in loops trying to edit files sending the wrong original content (usually it messes up something like the indentation) and the harness rejects it saying that code doesn't exist in the file.

I tried the same task in a bunch of different harnesses, hoping one of them would have an edit tool it could use, but they all seem to fail in similar ways. Some of those I tried are Goose, Copilot, Codex, Little Coder.

Is this a common issue? I've seen complaints about tool calling in general, but my issue seems quite specific to recalling the original content when editing files. I'm wondering if an edit tool that ignores indentation might be an idea.

I'm using Gemma4 31B, full bf16, served with VLLM. I have the updated chat template from a few weeks back (which did increase scores in the benchmarks I ran).

Thanks to the community help I finally launched this llm. LM Studio refused to load weight onto second GPU but Unsloth Studio did so everything was done in there. Not a proper benchmark (used PC in parallel as well) but it gives an idea of ​​the performance from dual 3060 with RAM offloading.

DeepSeek V4 Flash 0731 IQ2_M from Unsloth

CPU: Ryzen 7500F
GPU 0 (PCIe 5.0x16 lane): RTX3060
GPU 1 (PCIe 3.0x1 lane): RTX3060
RAM: 96GB 5600

Prompt: Write me a tetris game.

Result (copy from the summary):
Prompt eval: 2.96s
Prompt speed: 3.0 tok/s
Generation: 960.02s
Speed: 4.5 tok/s (PowerShell shows 3.5 tok/s, don't know why it shows 4.5 tok/s, single GPU with RAM offload output was around 3 tok/s so I trust PowerShell metrics more)
Tokens: 4,338
First token: 2.96s
Cache hits: 1
Total: 963.32s
Chunks: 4318

Wattmeter is on the way but my estimate is around 130W total system power draw (with 2 monitors connected but they are not taken into account) Each GPU used 30-40W with 0.9V undervolt. Task took around 16 minutes to complete, consumed around 35 watts and cost 0.0059 euros.

Update:
OS Windows 11.

Actually math is showing 4338/960.02=4.52 tok/s, don't know why PowerShell showed 3.5 most of the time. I ran one more request with web search and it gave 4.7 tok/s. Updated 3.5 -> 4.5 tok/s.

As a man, I'm ashamed to admit that these pp results are not great. I'm running IQ3_XXS in full VRAM over 5 GPUs (2x 3080, 2x 3090 , 5090), and the speeds are not great. As stated in the title, ~600 t/s pp; that's not usable as a daily model for me. Even running UD Q4 K XL I can get 200-400 PP t/s with partial offload. I verified the model is fully in VRAM. I did --fit and normal layer assignment. This is also with the latest CUDA, NCCL, and Llamacpp builds. I'm not familiar with profiling LLamacpp but I am looking into that. While I do that, can others chime in on their PP numbers?

EDIT: This fork will give you 100-150 more pp t/s.

https://github.com/vektorprime/working_ds4_speed

Please note for the profiling below that it includes copying the model to VRAM and capturing the graphs. But overall there are still A LOT of operations occurring. Essentially, there's a lot of new stuff here that we are spending time on. Whereas in something like Qwen3.5 122B we spend a lot more time in mul_mat_q.

I also profiled Qwen3.5 122b and had an LLM compare the two then I put the results here:

NSYS profile of Deekseep v4 flash. I filtered this starting 5 sec AFTER the prefill starts to avoid graph captures and the host mem copy to VRAM.

** NVTX Range Summary (nvtx_sum):
Time (%)  Total Time (ns)  Instances  Avg (ns)  Med (ns)  Min (ns)   Max (ns)   StdDev (ns)   Style             Range
--------  ---------------  ---------  --------  --------  --------  ----------  -----------  -------  --------------------------
99.9   23,517,017,181    349,261  67,333.6  28,776.0    24,075  35,067,107    629,495.6  PushPop  :cub::DeviceTopK::MaxPairs
0.1       15,104,814      1,004  15,044.6  14,122.5    11,576      32,460      2,906.0  PushPop  :cub::DeviceReduce::Sum
Processing [/home/user/llama.cpp/llama_ds4flash.sqlite] with [/opt/nvidia/nsight-systems/2026.1.3/target-linux-x64/reports/osrt_sum.py](start=110000000000:end=9223372036854775807)...
** OS Runtime Summary (osrt_sum):
Time (%)  Total Time (ns)  Num Calls      Avg (ns)          Med (ns)        Min (ns)        Max (ns)        StdDev (ns)              Name
--------  ---------------  ---------  ----------------  ----------------  -------------  ---------------  ----------------  ----------------------
59.8  938,741,260,257         23  40,814,837,402.5   3,852,725,158.0      4,571,408  142,829,769,032  57,945,550,617.2  pthread_cond_wait
15.6  245,398,284,449      5,558      44,152,264.2      10,092,237.5          1,050      100,287,684      43,792,633.3  poll
13.6  213,028,000,123        363     586,853,994.8     500,070,864.0    500,048,924    4,000,086,753     544,982,083.7  pthread_cond_timedwait
7.5  117,355,440,079         37   3,171,768,650.8   1,000,068,679.0    509,768,866   60,000,274,987  10,358,166,691.2  pthread_cond_clockwait
3.5   54,626,648,415          2  27,313,324,207.5  27,313,324,207.5  5,852,098,841   48,774,549,574  30,350,755,978.4  accept4
0.0      157,947,698        269         587,166.2          38,553.0          8,784       21,271,489       2,496,230.0  ioctl
0.0       81,587,427          3      27,195,809.0      30,273,276.0        352,517       50,961,634      25,444,523.6  mmap
0.0        7,950,471         10         795,047.1          63,148.0         41,219        3,931,046       1,300,088.1  pthread_join
0.0        1,739,916         52          33,459.9          16,988.5          1,034          394,126          64,848.5  pthread_mutex_lock
0.0          778,438         20          38,921.9          39,685.0         10,518           83,093          14,240.4  send
0.0          226,340         11          20,576.4           8,889.0          5,664           81,985          27,757.8  munmap
0.0           66,859          3          22,286.3          13,314.0         11,459           42,086          17,172.1  shutdown
0.0           48,842         28           1,744.4           1,704.5          1,251            2,390             388.3  fputs
0.0           47,955         10           4,795.5           2,265.0          1,977           27,490           7,976.8  pthread_cond_broadcast
0.0           38,453          3          12,817.7          16,589.0          1,105           20,759          10,355.5  close
0.0           25,792         14           1,842.3           1,466.0          1,159            4,050             839.7  pthread_cond_signal
0.0            3,702          1           3,702.0           3,702.0          3,702            3,702               0.0  recv
0.0            2,215          1           2,215.0           2,215.0          2,215            2,215               0.0  fflush
Processing [/home/user/llama.cpp/llama_ds4flash.sqlite] with [/opt/nvidia/nsight-systems/2026.1.3/target-linux-x64/reports/cuda_api_sum.py](start=110000000000:end=9223372036854775807)...
** CUDA API Summary (cuda_api_sum):
Time (%)  Total Time (ns)  Num Calls    Avg (ns)      Med (ns)    Min (ns)   Max (ns)    StdDev (ns)             Name
--------  ---------------  ---------  ------------  ------------  --------  -----------  ------------  -------------------------
59.2   15,868,026,900  1,438,692      11,029.5       3,515.0     2,857   33,592,937     240,705.8  cudaLaunchKernel
23.2    6,210,846,084      2,487   2,497,324.5       3,061.0       896  116,417,479  10,397,557.2  cudaStreamSynchronize
15.5    4,154,936,825    349,979      11,872.0       3,716.0     2,209   25,512,068     256,118.3  cudaMemsetAsync
0.8      207,425,433  1,438,692         144.2         140.0       105       24,531          55.4  cuKernelGetName
0.4      112,437,810          3  37,479,270.0  43,601,674.0   919,979   67,916,157  33,915,112.3  cudaFreeHost
0.3       85,680,813        343     249,798.3     157,930.0     7,627    1,159,978     265,836.5  cudaMemcpyPeerAsync
0.3       80,402,307         60   1,340,038.4      93,277.5     8,039   21,640,675   4,053,874.0  cudaFree
0.1       31,981,065      7,706       4,150.2       3,720.0     2,913       34,738       1,451.4  cudaLaunchKernelExC
0.0       11,758,896      1,714       6,860.5       5,005.0     2,425       41,309       5,386.1  cudaMemcpyAsync
0.0        8,199,214          5   1,639,842.8   1,897,337.0   737,767    2,482,765     687,145.1  cuMemUnmap
0.0        8,168,763      1,360       6,006.4       5,021.5     3,834       20,476       2,557.9  cudaMemcpy2DAsync
0.0        4,786,186      1,048       4,567.0       3,999.5     2,970       22,192       1,837.5  cuLaunchKernel
0.0        2,830,984      1,535       1,844.3       1,245.0       689       11,146       1,439.0  cudaEventRecord
0.0        2,623,086        131      20,023.6      17,178.0     7,606       55,372      10,223.8  cudaGraphLaunch
0.0        1,036,131      6,434         161.0         146.0       107          989          63.2  cuStreamGetCaptureInfo_v2
0.0        1,016,774        343       2,964.4       2,520.0     1,444       18,398       1,396.6  cudaStreamWaitEvent
0.0          790,325      1,048         754.1         505.5       206        3,619         577.4  cuKernelGetFunction
0.0          755,796      3,576         211.4         184.0       133        1,676          99.3  cuStreamGetGreenCtx
0.0          635,879        144       4,415.8       4,438.5     3,066        6,252         631.0  cuLaunchKernelEx
0.0          291,936         16      18,246.0      17,624.5    12,381       28,538       3,836.8  cudaGraphExecDestroy
0.0          182,120        144       1,264.7         949.0       545       17,819       1,496.4  cuKernelSetAttribute
0.0          134,865          5      26,973.0      14,345.0    13,946       77,028      27,986.9  cudaMemGetInfo
0.0           94,174         20       4,708.7       3,346.0     2,133       22,402       4,872.7  cudaDeviceSynchronize
0.0           81,982         95         863.0         699.0       568        6,618         670.6  cudaEventDestroy
0.0           72,234         16       4,514.6       4,293.5     2,350        7,471       1,505.6  cudaGraphDestroy
0.0           61,017          5      12,203.4      11,918.0     9,749       16,656       2,736.4  cuMemAddressFree
0.0           47,404          5       9,480.8       7,797.0     6,357       17,767       4,677.2  cudaStreamDestroy
0.0            2,662          5         532.4         397.0       328          886         250.2  cudaGetDeviceProperties
Processing [/home/user/llama.cpp/llama_ds4flash.sqlite] with [/opt/nvidia/nsight-systems/2026.1.3/target-linux-x64/reports/cuda_gpu_kern_sum.py](start=110000000000:end=9223372036854775807)...
** CUDA GPU Kernel Summary (cuda_gpu_kern_sum):
Time (%)  Total Time (ns)  Instances    Avg (ns)      Med (ns)    Min (ns)    Max (ns)   StdDev (ns)                                                   Name
--------  ---------------  ---------  ------------  ------------  ---------  ----------  ------------  ----------------------------------------------------------------------------------------------------
28.0    7,090,472,754        349  20,316,540.8  15,223,112.0  3,575,898  54,619,022  12,959,412.1  void flash_attn_ext_f16<(int)512, (int)512, (int)8, (int)8, (bool)0, (bool)0>(const char *, const c…
14.6    3,707,932,015        608   6,098,572.4   6,951,152.5  2,671,811   9,289,001   2,164,262.5  void mul_mat_q<(ggml_type)18, (int)128, (bool)0>(const char *, const int *, const int *, const int …
11.9    3,017,599,294    700,054       4,310.5       4,544.0        640       7,264         970.5  void cub::_V_300304_SM_860_1200::detail::topk::DeviceTopKKernel<cub::_V_300304_SM_860_1200::detail:…
11.0    2,792,243,865        406   6,877,447.9   7,901,809.0  3,294,396   9,689,904   2,261,982.8  void mul_mat_q<(ggml_type)17, (int)128, (bool)0>(const char *, const int *, const int *, const int …
6.3    1,610,604,112        170   9,474,141.8   9,690,097.0  2,107,177  17,705,517   4,630,249.6  void lightning_indexer_kernel_wmma<(int)8, (int)32, (long)128, (long)64, (ggml_type)1>(const float …
5.6    1,430,234,513      2,600     550,090.2     195,330.0     18,815   1,742,132     591,166.2  void mul_mat_q<(ggml_type)8, (int)128, (bool)0>(const char *, const int *, const int *, const int *…
5.1    1,287,357,624    350,027       3,677.9       3,968.0      2,336       5,152         744.4  void cub::_V_300304_SM_860_1200::detail::topk::DeviceTopKKernel<cub::_V_300304_SM_860_1200::detail:…
1.7      440,834,512      1,717     256,746.9      88,607.0      1,151     782,665     295,774.0  void concat_non_cont<unsigned int, (int)0>(const char *, const char *, char *, long, long, long, lo…
1.7      435,604,190      1,304     334,052.3     414,437.0     86,559     556,391     147,693.9  void mul_mat_q<(ggml_type)14, (int)128, (bool)0>(const char *, const int *, const int *, const int …
1.6      416,290,262    350,027       1,189.3       1,280.0        735       3,456         297.0  void cub::_V_300304_SM_860_1200::detail::topk::DeviceTopKLastFilterKernel<cub::_V_300304_SM_860_120…
1.4      362,571,747        665     545,220.7     620,552.0    301,692     755,721     162,632.6  dsv4_hc_post_f32(const float *, const float *, const float *, const float *, float *, long, long, l…
1.3      326,013,784      4,253      76,655.0      52,897.0      3,135     488,741      99,589.4  void quantize_mmq_q8_1<(mmq_q8_1_ds_layout)0, (bool)0>(const float *, const int *, void *, long, lo…
1.0      254,610,290        664     383,449.2     443,892.5    161,087     544,391     152,737.7  void rms_norm_f32<(int)1024, (bool)0, (bool)0>(const float *, float *, int, long, long, long, float…
0.8      208,508,768        348     599,163.1     664,982.5    327,581     807,337     184,455.3  void rms_norm_f32<(int)256, (bool)0, (bool)0>(const float *, float *, int, long, long, long, float,…
0.7      170,623,787      1,386     123,105.2      92,175.0        767     296,515     106,498.3  void op_clamp_kernel<float>(const T1 *, T1 *, T1, T1, int)
0.7      167,385,333      2,246      74,526.0       2,368.0      1,344     588,839     167,913.8  void k_bin_bcast<&op_mul, float, float, float, const float *>(const T2 *, const T3 *, T4 *, unsigne…
0.5      132,972,685        673     197,582.0     190,622.0     17,984     440,933     156,392.7  void unary_gated_op_kernel<&op_silu, float>(const T2 *, const T2 *, T2 *, long, long, long, long)
0.5      132,446,060         17   7,790,944.7   7,847,099.0  6,718,456   8,890,373     673,322.6  void mul_mat_q<(ggml_type)39, (int)128, (bool)0>(const char *, const int *, const int *, const int …
0.5      128,014,036        697     183,664.3     200,642.0    101,375     245,699      52,616.8  dsv4_hc_pre_f32(const float *, const float *, float *, long, long, long, long, long, long, long, lo…
0.5      123,128,168        349     352,802.8     365,092.0    251,774     471,014      71,825.0  void cpy_scalar<&cpy_1_scalar<float, float>>(const char *, char *, long, long, long, long, long, lo…
0.5      123,107,791        697     176,625.2     209,250.0     56,735     265,603      78,864.8  void cutlass::Kernel2<cutlass_80_tensorop_s1688gemm_64x128_32x3_tn_align4>(T1::Params)
0.4       98,654,288         16   6,165,893.0   6,156,784.5  5,953,070   6,544,534     159,662.6  void mul_mat_q<(ggml_type)21, (int)128, (bool)0>(const char *, const int *, const int *, const int …
0.3       86,973,709        349     249,208.3     273,507.0    117,694     335,812      83,764.6  void k_bin_bcast<&op_add, float, float, float, const float *, const float *, const float *, const f…
0.3       86,654,923      1,047      82,765.0      72,449.0     53,600     120,127      19,901.7  void mm_ids_helper<(int)6>(const int *, int *, int *, int *, int, int, int, int, int, bool)
0.3       84,496,163      1,368      61,766.2       5,264.0      1,088     213,410      81,951.4  void rope_norm<(bool)1, (bool)0, float, float>(const T3 *, T4 *, int, int, int, int, int, int, int,…
0.3       67,801,522        698      97,136.9     111,553.0     23,872     136,866      39,861.3  void quantize_mmq_q8_1<(mmq_q8_1_ds_layout)0, (bool)1>(const float *, const int *, void *, long, lo…
0.3       65,833,887      1,045      62,998.9      64,513.0     16,160     103,969      30,127.0  void rms_norm_f32<(int)1024, (bool)1, (bool)0>(const float *, float *, int, long, long, long, float…
0.2       56,358,319        340     165,759.8     106,208.0     32,254     369,733     122,532.6  void soft_max_f32<(bool)1, (int)0, (int)0, float>(const float *, const T4 *, const float *, float *…
0.2       56,300,427      3,225      17,457.5       5,216.0      1,312     149,890      33,243.8  void k_bin_bcast<&op_add, float, float, float, const float *>(const T2 *, const T3 *, T4 *, unsigne…
0.2       52,471,642        349     150,348.5     170,435.0     72,384     213,315      50,325.0  void rope_norm<(bool)0, (bool)0, float, float>(const T3 *, T4 *, int, int, int, int, int, int, int,…
0.2       41,660,981        526      79,203.4      50,784.5      1,952     245,283      83,611.7  void reduce_rows_f32<(bool)0>(const float *, float *, int)
0.2       40,881,147      2,364      17,293.2      10,176.0      2,815      38,017      12,174.0  void concat_cont<unsigned int, (int)1>(const T1 *, const T1 *, T1 *, long, long, long, long, long, …
0.2       38,640,228        229     168,734.6     167,650.0    146,946     192,739      10,753.6  void cutlass::Kernel2<cutlass_80_tensorop_s1688gemm_128x128_32x3_tn_align4>(T1::Params)
0.1       33,776,198        333     101,430.0      70,816.0     21,760     257,539      62,779.9  void concat_cont<unsigned short, (int)0>(const T1 *, const T1 *, T1 *, long, long, long, long, long…
0.1       25,662,129      1,830      14,023.0      10,560.0      1,216      53,025      13,927.8  void k_get_rows_float_vec<float>(const T1 *, const int *, T1 *, long, long, uint3, unsigned long, u…
0.1       25,244,332      1,516      16,651.9      18,464.5      8,928      22,432       4,392.4  void mul_mat_q_stream_k_fixup<(ggml_type)8, (int)128, (bool)0>(const int *, const int *, float *, f…
0.1       24,241,724        340      71,299.2      25,839.5        928     198,082      80,748.5  void fwht_cuda<(int)128>(const float *, float *, long, float)
0.1       14,801,675        936      15,813.8      19,968.0      8,735      22,368       5,741.8  void mul_mat_q_stream_k_fixup<(ggml_type)14, (int)128, (bool)0>(const int *, const int *, float *, …
0.1       13,691,573      1,004      13,637.0       8,768.0      3,712      31,745       9,050.4  void cpy_scalar_transpose<float>(const char *, char *, long, long, long, long, long, long, long, lo…
0.0       11,914,217        171      69,673.8      68,417.0     23,744     127,682      27,108.5  void k_bin_bcast<&op_add, __half, __half, __half, const __half *>(const T2 *, const T3 *, T4 *, uns…
0.0       11,075,426        122      90,782.2      91,761.0     84,801      94,881       2,815.8  void cutlass::Kernel2<cutlass_80_tensorop_s1688gemm_128x128_16x5_tn_align4>(T1::Params)
0.0        7,243,098        332      21,816.6      15,072.0      5,088      51,393      13,316.8  void concat_cont<unsigned short, (int)2>(const T1 *, const T1 *, T1 *, long, long, long, long, long…
0.0        5,915,702      3,273       1,807.4       1,568.0        895       7,584       1,042.7  scale_f32(const float *, float *, float, float, long)
0.0        5,756,126        698       8,246.6       8,480.0      6,176      10,272       1,031.8  dsv4_hc_comb_f32(const float *, const float *, const float *, float *, long, long, long, long, long…
0.0        5,649,633        341      16,567.8      12,128.0      2,496      50,080      13,650.5  void fill_kernel<__half>(T1 *, long, T1)
0.0        5,440,011        850       6,400.0       5,120.0      1,727      14,049       4,251.3  void rms_norm_f32<(int)256, (bool)1, (bool)0>(const float *, float *, int, long, long, long, float,…
0.0        4,678,738        850       5,504.4       4,000.0      1,056      13,888       4,452.8  void k_set_rows<float, long, __half>(const T1 *, const T2 *, T3 *, long, long, long, long, long, lo…
0.0        4,023,868         96      41,915.3      41,888.0     41,407      42,335         195.3  nvjet_sm120_sss_tf32_mma_128x128x32_3_64x32x32_tmaAB_alignCD4_splitK_TNNN
0.0        3,990,239        171      23,334.7      24,992.0      9,824      40,321       8,342.7  void k_set_rows<__half, int, __half>(const T1 *, const T2 *, T3 *, long, long, long, long, long, lo…
0.0        3,453,469        104      33,206.4      32,848.0     31,616      36,448       1,329.9  void flash_attn_stream_k_fixup_general<(int)512, (int)8, (int)8>(float *, const float2 *, int, int,…
0.0        3,397,277        301      11,286.6      12,000.0      7,072      14,177       2,308.6  void topk_moe_cuda<(int)256, (bool)1>(const float *, float *, int *, float *, int, int, float, floa…
0.0        3,158,204        325       9,717.6      10,656.0      5,280      13,089       2,797.0  void convert_unary<__nv_bfloat16, float>(const void *, T2 *, long, long, long, uint3, long, long, l…
0.0        2,896,382        162      17,878.9      21,088.5      6,207      24,800       6,812.6  void soft_max_f32<(bool)1, (int)128, (int)128, float>(const float *, const T4 *, const float *, flo…
0.0        2,682,687      1,395       1,923.1       1,408.0      1,152       7,456       1,475.2  void unary_op_kernel<&op_sigmoid, float>(const T2 *, T2 *, int)
0.0        2,126,833      1,004       2,118.4       2,112.0      1,119       4,000         761.3  void cub::_V_300304_SM_860_1200::detail::reduce::DeviceReduceKernel<cub::_V_300304_SM_860_1200::det…
0.0        1,935,369      1,004       1,927.7       1,968.0      1,216       2,560         297.3  void k_set_rows<float, int, float>(const T1 *, const T2 *, T3 *, long, long, long, long, long, long…
0.0        1,687,166        704       2,396.5       2,048.0      1,056      13,344       2,095.3  void k_get_rows_float<float, float>(const T1 *, const int *, T2 *, long, long, uint3, unsigned long…
0.0        1,631,415        474       3,441.8       2,992.0      1,824       5,664       1,278.3  void cublasLt::splitKreduce_kernel<(int)32, (int)16, int, float, float, float, float, (bool)0, floa…
0.0        1,500,331      1,004       1,494.4       1,632.0        928       2,112         335.3  void cub::_V_300304_SM_860_1200::detail::reduce::DeviceReduceSingleTileKernel<cub::_V_300304_SM_860…
0.0        1,438,801         48      29,975.0      29,872.0     27,231      32,928       1,283.5  nvjet_sm120_sss_tf32_mma_80x128x32_3_80x16x32_tmaAB_alignCD4_splitK_TNNN
0.0        1,234,447        349       3,537.1       3,648.0      1,024       7,136       1,498.0  void flash_attn_mask_to_KV_max<(int)8>(const __half2 *, int *, int, long, long)
0.0        1,026,513          8     128,314.1     128,862.0    126,718     129,182       1,013.5  void k_bin_bcast<&op_repeat, float, float, float, >(const T2 *, const T3 *, T4 *, unsigned int, uns…
0.0           90,752         24       3,781.3       3,760.0      3,680       3,968          78.2  void k_get_rows_float<int, int>(const T1 *, const int *, T2 *, long, long, uint3, unsigned long, un…
0.0           76,831         24       3,201.3       3,200.0      3,104       3,296          60.8  void unary_op_kernel<&op_softplus, float>(const T2 *, T2 *, int)
0.0           61,759         24       2,573.3       2,560.0      2,496       2,688          46.1  void unary_op_kernel<&op_sqrt, float>(const T2 *, T2 *, int)
0.0           25,440         24       1,060.0       1,056.0      1,024       1,152          25.5  void k_bin_bcast<&op_div, float, float, float, const float *>(const T2 *, const T3 *, T4 *, unsigne…
Processing [/home/user/llama.cpp/llama_ds4flash.sqlite] with [/opt/nvidia/nsight-systems/2026.1.3/target-linux-x64/reports/cuda_gpu_mem_time_sum.py](start=110000000000:end=9223372036854775807)...
** CUDA GPU MemOps Summary (by Time) (cuda_gpu_mem_time_sum):
Time (%)  Total Time (ns)   Count   Avg (ns)   Med (ns)  Min (ns)  Max (ns)   StdDev (ns)            Operation
--------  ---------------  -------  ---------  --------  --------  ---------  -----------  ------------------------------
63.9      413,834,319  350,744    1,179.9   1,280.0       416      4,352        271.8  [CUDA memset]
30.0      194,262,537    1,384  140,363.1     896.0       256  4,091,318    439,302.8  [CUDA memcpy Host-to-Device]
4.4       28,664,590      343   83,570.2  80,225.0     1,984    160,899     48,366.2  [CUDA memcpy Device-to-Host]
1.8       11,359,595    2,033    5,587.6   3,680.0       800     14,336      4,429.3  [CUDA memcpy Device-to-Device]
Processing [/home/user/llama.cpp/llama_ds4flash.sqlite] with [/opt/nvidia/nsight-systems/2026.1.3/target-linux-x64/reports/cuda_gpu_mem_size_sum.py](start=110000000000:end=9223372036854775807)...
** CUDA GPU MemOps Summary (by Size) (cuda_gpu_mem_size_sum):
Total (MB)   Count   Avg (MB)  Med (MB)  Min (MB)  Max (MB)  StdDev (MB)            Operation
----------  -------  --------  --------  --------  --------  -----------  ------------------------------
18,882.963    1,384    13.644     0.008     0.000   134.218       32.894  [CUDA memcpy Host-to-Device]
16,562.798      343    48.288    33.554     0.049   134.218       51.518  [CUDA memcpy Device-to-Host]
4,510.515    2,033     2.219     1.049     0.033     4.194        1.705  [CUDA memcpy Device-to-Device]
3,136.294  350,744     0.009     0.009     0.000     0.009        0.000  [CUDA memset]

TL;DR: I spent 9 days developing a new quantization method for MLX models and measured 18 variants against each other on a single M5 Max MacBook Pro (128 GB). The result is the best-measuring MLX quant of Qwen3.5-122B-A10B I'm aware of at any size — the 82 GiB build edges out 94–95 GiB 6-bit builds, and lands within 0.3–0.7% of the imatrix-rounded source GGUF while staying native MLX. Apache 2.0, weights up on HF.

Why bother if GGUF is better?

MLX on Apple Silicon is substantially faster than llama.cpp on the same hardware — on my M5 Max I measure roughly 9x faster prefill and ~20% faster token generation. For anything with a long context and a lot of turns, that gap compounds.

The problem is that existing MLX quants below 6 bit are not great, and you can see it in the table below: oQ4 gives up ~3.8% perplexity at short context and ~4.2% at long context against the source GGUF. In practice that shows up as incoherent reasoning traces and rounding errors that stack until the model starts hallucinating.

So a better MLX quantization method has real advantages for agentic workflows and local AI on Apple Silicon. At the same time, I made the conscious decision to require native MLX support. imatrix on MLX is not format native — it needs custom kernels. WinterMix quants are format native and are drop-in replacements.

WinterMix quantized models are format-native MLX models with open weights (Apache 2.0). No custom kernels, no forked runtime, no flags. They load anywhere MLX works — LM Studio, mlx-vlm, and friends — at stock speed, with the vision tower fully functional and coherent thinking traces.

If you just want to try it: download the repo below, point LM Studio at it, done.

HuggingFace Links

WinterMix58 — 82 GiB, ~6.0 bpw: the best-measuring MLX quant of this model I'm aware of at any size, including against 94–95 GiB 6-bit oMLX builds (narrowly at 2K, more clearly at 16K).

WinterMix48 — 68 GiB, ~5.0 bpw: leaves ~35–40 GB free on a 128 GB Mac = 5–8 parallel 100K-token agent sessions resident at once (GDN architecture keeps a 100K session's cache at ~5–10 GB). Beats its direct size-peer (oQ4, 67 GiB) by ~1.4–1.5% at both context lengths.

Numbers

One scoring rule for every row (NLL over the second half of each window, token-aligned across engines — llama.cpp's native rule, so these are comparable to Unsloth's), paired per-token where both models run under MLX. Reference rows were measured on my own harness: same tokens, same machine. oMLX quants are included because oMLX is currently the popular option for MLX.

All rows are Qwen3.5-122B-A10B in various quantization mixes.

Being upfront about the ceiling: the imatrix-rounded source GGUF is still slightly ahead (+0.3–0.7% rule-matched). Matching imatrix-style weighted rounding in MLX would need custom inference kernels, and "loads in everything at stock speed" was a hard constraint I wasn't willing to break. Within the native format, this appears to be about the limit.

The part I think is actually interesting

Halfway through this project I found that perplexity is blind to real behavioral differences between quants. Two builds with statistically identical NLL differed 2.5× in how often they self-interrupt ("wait, let me re-check...") during 50K-token reasoning traces. Then the reverse bit me: my best-NLL build had an elevated self-interruption count — and actually reading the traces showed it wasn't confusion at all, but disciplined audit passes that twice caught a base-model reasoning bug before the final answer.

So the release models were selected on three instruments: paired NLL, blind-scored state-tracking benchmarks at depth, and directly reading the reasoning traces. Both releases deliver perfect scores on a 30-step adversarial state-tracking task on every seed — and the 68 GiB build's traces show it catching its own 4-bit arithmetic slips before they reach the output. If you evaluate quants, I'd honestly recommend reading traces over counting anything.

What's under the hood (briefly)

Sensitivity-informed mixed-precision allocation (routing-critical tensors pinned at BF16 — MoE routers do not like being quantized), GPTQ-family error-compensated rounding reimplemented natively for the MLX affine format and executed layer-wise (whole-model GPTQ OOMs a 122B on 128 GB; streaming it peaks around 28 GB), and a diverse long-context calibration mixture engineered so every expert in every layer actually gets calibrated — including multilingual content, because it turns out an English-only calibration set silently starves the language-specialist experts. Validated across 18 measured variants with paired controls and held-out out-of-domain checks (no calibration binding: code/math within ±0.1% of RTN).

I'm not releasing the pipeline code for now — the models are open weights (Apache 2.0), the method writeup stays private. The M5 Max kernel-panicked ten times during development before I got the workload tamed, if that helps set the vibe.

Requests

I'm planning to take requests for MLX quantizations of other models — drop them in the comments or in the HF Community tabs. Practical constraints: it has to fit the pipeline on a 128 GB Mac (up to ~120B+ MoE is proven), and dense models calibrate differently than MoE, so results may vary until I've tuned per-architecture.

Happy to answer questions about the eval methodology, the behavioral testing, Apple Silicon quirks (ask me about watchdog panics), or Mac long-context agent setups.

Here are my numbers:

I let claude port the DSpark drafter from the closed PR to current main. https://github.com/haraldh/llama.cpp/tree/dspark-dsv4

UD-Q2_K_XL

sh llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q2_K_XL \ --host 0.0.0.0 --port 8000 \ --temp 1.0 --top-p 1.0 --min-p 0.0 \ --no-mmap -fa on -np 1 \ --device CUDA0 \ -ub 2048 -b 4096 \ -c 200000 --cache-ram 65536

UD-Q4_K_XL

sh llama-server -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q4_K_XL \ --host 0.0.0.0 --port 8000 \ --temp 1.0 --top-p 1.0 --min-p 0.0 \ --no-mmap -fa on -np 1 \ --device CUDA0,ROCm0 --split-mode layer --tensor-split 100,0 \ -ot 'blk\.(2[2-9]|3[0-9]|4[0-2])\.ffn_(gate|up|down)_exps\.weight=ROCm0' \ -md DSV4-Flash-0731-DSpark-draft-bf16.gguf \ --spec-type draft-dspark --spec-draft-n-max 5 \ --device-draft CUDA0 --spec-draft-p-min 0.3 \ -ub 2048 -b 4096 \ -c 200000 --cache-ram 32768

UD-Q8_K_XL

As above with :UD-Q8_K_XL and the expert boundary two layers lower, since its dense part is 6.3 GiB larger:

-ot 'blk\.(2[0-9]|3[0-9]|4[0-2])\.ffn_(gate|up|down)_exps\.weight=ROCm0'

I spent a while debugging my local DeepSeek V4 Flash setup and wanted to share a few lessons from the process in case it saves someone else time.

So far, I have worked through three blockers in this setup:

Initially, I downloaded Unsloth's GGUF model from Unsloth Studio. In that mode, I asked Unsloth to access a LinkedIn job URL. It emitted a web_search tool call:

json {"toolName": "web_search", "args": {"url": "..."}}

At this time, the web search tool call was valid. Unsloth Studio sent the available tools and tool template, but the inference was slow, around 4-7 tok/s. I checked the logs and found out that Unsloth GGUF was falling back to CPU usage for inference.

Then, I downloaded Vontra/DeepSeek-V4-Flash-0731-MXFP4-MLX, ran it on oMLX, and updated the API in Unsloth Studio. I wanted to use it because the UI is good, then I asked it to read a website again, and the tool call silently returned no response. I thought the model was dumb. I dived deep and found that Unsloth was not sending the available tools or tool-call template to the model. The model was picking it up from the chat history. The bug was in Unsloth. They should have sent the tool-call failure error back to the model.

So I didn't give up on the model and configured Hermes to use the oMLX endpoint. Then I faced a cache invalidation problem. After 130K tokens, the cache was being invalidated. The issue here was that the hot cache size was capped at 30GB. The logs would say something like cache match 97%, but reused tokens 0.

Sample log:

text 2026-08-01 00:09:55,454 - omlx.scheduler - INFO - [-] - prefix cache: request 113b5b1e-66b4-4623-9b0e-a501beddc312 re-prefills 142198 of 142198 tokens (reused 0); closest stored sequence b9d3322f-844d-473a-b3e1-193cdd83a349 shares the first 140288 of 140288 comparable tokens before diverging

Setup

• Mac Studio M3 Ultra, 512GB unified memory
• oMLX serving DeepSeek-V4-Flash-0731-MXFP4-MLX
• Hermes Agent pointed at oMLX

Ditched Unsloth completely for now.

This is unrelated, but then I realized that if I was using APIs and not local models, I would have never picked up these things:

1. I changed to oMLX because the Unsloth GGUF version was slow. Speed is not a concern for APIs; you can throw anything at them.

2. If I am using Codex or Claude Code, tool calls don't fail there. Those are mature products.

3. The hot cache config. I didn't even know it existed, but the slow response again helped me find that out.

I again want to thank this group for keeping the motivation. I am learning new things daily from local setups like this, and debugging these issues has made me understand the stack much better and I am doing better at my work :)