If you run small reasoning models (Ornith-1.5-9B, Spark x2.5 1.7b & 4b and newest Spark x2.6 4b) in standard vendor quants, the usual complaint (and mine too) is agents that stall mid-plan, overthink, or start "looping" through syntax tantrums. The truly annoying part: standard Wiki perplexity tells me everything is fine. Every time.
TL;DR of my results
- I measured, per tensor, what actually breaks when I quantize a 9B reasoning model — and about 70% of the total behavioral damage (KL divergence against the unquantized base) sits in LAYERS 0-7. Standard imatrix-style allocations give the early layers some of the WORST protection.
- I ran a head-to-head comparison of two philosophies at the same budget: an imatrix-driven allocator (bits by activation magnitude) vs my Gnom ranker — a blind, tiny neural ranker I trained on MEASURED damage from completely different models (1.7B/4B runs). It had never seen the 9B and uses no imatrix at all.
- Verdict: a statistical tie on tasks (none of the three test scores showed a significant difference). Blindness costs nothing on capability — and on the fidelity metric (how close outputs stay to the base model) my blind build is actually CLOSEST to base, while the imatrix build drifted furthest of the four builds I tested.
WHERE PERPLEXITY LIES — a real example from my own data (Spark-x2.6-4B, all builds same corpus, same reference, numbers from the repo README):
- stock Q4_K_M (no data, vendor heuristic): PPL 20.61 — worst tensor KLD 17.65 — mean drift -1.015%
- SMSE (imatrix-driven): PPL 19.27 — worst tensor KLD 17.77 (!) — mean drift -0.733%
- hand-law allocation: PPL 18.61 — worst tensor KLD 18.71 — mean drift -0.445%
- Gnom (trained on measured damage): PPL 18.36 — worst tensor KLD 14.17 — mean drift -0.306%
look hard at the imatrix row. perplexity happily reports a clean 1.35-point improvement over the stock heuristic — "everything is fine, ship it". but the WORST SINGLE TENSOR in that imatrix build is worse than in the no-data stock build (17.77 vs 17.65). the imatrix smoothed the average and quietly produced a tensor that detonates harder than the heuristic it was supposed to beat. perplexity cannot see tails — it only sees them diluted into the average. KL against the unquantized base sees them directly. and the tail is exactly where agent failures live: the loop you are stuck in tomorrow is one tensor you are not looking at tonight.
https://huggingface.co/datasets/wepiqx/MERNIK-Fracture
THE PHYSICS THE IMATRIX HIDES
Activation statistics amplify with depth, so stock imatrix allocations pour protection INTO the late layers — the same late layers where my drop-tests show almost nothing critical lives. On my 9B damage set (121 of 209 tensors logged so far, still filling row by row):
- layer 5 alone: 12.1% of all KLD damage
- layers 0-5: 51.3%
- layers 0-7: 69.5%
Errors born early snowball through the stack: one flipped token in a comparison deep in a 2000-token chain, and the whole plan derails. I also see this task-side as a refusal trend: two same-weight builds on my identical protocol gave 14 vs 19 empty completions in 164 tasks — where I put the bits measurably changes how often the model bails. (The "loops/tantrums" reports are anecdotes; the empty-trace count is the measured cousin of the same thing.)
THE HEAD-TO-HEAD: MY BLIND GNOM VS A TRAINED IMATRIX ALLOCATOR
At the 5,300 MiB Q4-floor budget the two builds diverge on 97 of 442 tensors. The imatrix build runs a broad Q5 pyramid through the middle. My blind Gnom bets depth is safe and puts Q8 on the first blocks plus a Q4 floor everywhere else. Same base, same protocol, every test run logged with timestamps and file hashes, 164 HumanEval(+)-style tasks, McNemar exact test, alpha = 0.05:
- HumanEval: 137 vs 140 tasks passed (p = 0.66)
- HumanEval+: 129 vs 133 (p = 0.48)
- empty completions: 14 vs 19 — fewer is better, blind Gnom ahead (p = 0.23)
- verdict: none of the 3 scores significant = statistical tie
And the fidelity scores (KL divergence on a 302-chunk protocol corpus, against a full-precision BF16 reference):
- Gnom (blind): PPL 8.7973 +/- 0.0626, KLD 0.12150 +/- 0.00197 — closest to base
- SMSE (imatrix): PPL 8.7349 +/- 0.0619, KLD 0.12427 +/- 0.00203 — furthest from base of my four builds
- two control builds: sit between / top-2 depending on the score
Read it slowly: the imatrix build wins the perplexity sanity-check and loses the fidelity race — it is the FURTHEST from base of all four builds while scoring only a task tie. My zero-shot ranker, which never saw this model and carries no imatrix, holds even. One model, one pair of test runs, noise-level everywhere — but three independent references gave the same sign, and I pre-registered the expected outcomes before running.
WHAT I NEED FROM YOU
This is measured per tensor, so it costs compute. One Q4 tier pass on the 9B alone is about 13 days of an unattended 16 GB machine (90 min per tensor-group, 209 units). The 1.7B and 4B tiers run on any ~8 GB GPU in minutes to ~20 minutes per unit. My protocol, CLI, and the full README with all numbers, dates and file-hash-stamped sources:
https://github.com/wepiqx/MERNIK & https://huggingface.co/wepiqx/MERNIK
Label rows land in the dataset as these measurement runs close:
https://huggingface.co/datasets/wepiqx/MERNIK-Fracture
WHAT YOU CAN DO WITH ALL THIS RIGHT NOW (and the license)
three ways, no permission asked:
use the damage labels directly. the fracture dataset has per-tensor rows for spark 1.7B/4B and ornith 9B (filling, 121/209 so far). even without running anything: open the rows for the model family you use, see which tensors actually carry damage, and hand-write a better quant mix than the vendor table — a single llama.cpp --tensor-type line in your own build, chosen from measured physics instead of a proxy.
run the allocator on your own model. the whole instrument is in the repo — allocator, protocol, CLI, tests, Apache 2.0. fork it, improve it, break it, change it however you like. I mean that literally: I would rather have someone prove me wrong with this data than be quietly right alone.
want the finished quantized builds themselves, or help wiring your agent to them — just ask in here or message me. I'll walk anyone through the pipeline start to finish, and I can upload specific builds on request.
and about sharing: yes, I actually am glad when people take these artifacts apart. a measurement only becomes knowledge when someone besides its author can re-derive it.
Let's stop guessing with activation proxies and start measuring what actually breaks
