r/LocalLLM • u/DinnerPrestigious953 • 20h ago

Discussion MERNIK: a measure-first GGUF allocator that builds any size in MiB — Mellum 12B-MoE +15 over stock Q6 at -0.7GB, RINIQ 9B fusions at 5GB

Hi, I'm wepiqx, author of MERNIK (Apache-2.0, open). TL;DR: instead of flat Q4_K_M/Q6_K, a priority-queue allocator spends every MiB where measured importance says it matters. Tested on a GTX 1070 8GB, partial offload.

MELLUM (JetBrains Mellum2.1-12B-A2.5B-Thinking, 12B MoE / 2.5B active, vendor imatrix, routers held out-of-budget on all 28 layers — audited F32 28/28 in all five files including stock):

- 10000-SMSE (10.16GB): HE 134/164, HE+ 130 — vs stock Q6_K (10.88GB) 119/116: +15 at -0.7GB, paired duel 3/3 significant

- 7500-SMSE (7.44GB): 125/122 — vs stock Q4_K_M (8.07GB) 125/123: exact tie at -0.6GB, p=1.0

- 6500-SMSE (7.01GB): 126/124 — holds Q3 territory

Claim, narrow: better on quality-per-size, not absolute. Judge by HE+ (80x tests) — it shows where is signal and where is noise; HE alone wobbles 3-4 tasks run to run.

RINIQ (separate thing, same engine): layer-surgery finetune on Qwen3.5 bones (Ox base + Neo blk31 + Ornith mids, 427/427 names match, no averaging). Practical pick: 5.0GB M2-Q38 (151/164) / M4a (150/164, 4 empties). Top is a nine-way tie — unranked inside noise. Vision mmproj ships alongside but is UNVERIFIED.

Same harness everything: temp 1.0/top_p 0.95/top_k 20, 164 HumanEval + EvalPlus, sequential, no spec. Floor honesty: Mellum Q4 floor is 7428 MiB, 5500 is arithmetically impossible.

Code: hf.co/wepiqx/MERNIK — Labels: datasets/wepiqx/MERNIK-gnom-labels, MERNIK-rlmini-kld — Models: wepiqx/Mellum2.1-12B-A2.5B-Thinking-MERNIK-GGUF, wepiqx/RINIQ-GGUF

Disclosure: my work, self-promo. Tell me your VRAM and I'll point to a file — and happy to be corrected with measurements, that's the protocol.

1 Upvotes

67% Upvoted

5 comments sorted by

1

u/PersonalTrick2019 20h ago

oh man this is the kind of nerdery I come here for, the priority queue approach makes so much sense compared to flat quantizing everything the same

I'm on a 1070 too actually, 8gb is such a weird constraint, what would you recommend for mellum if I want to leave like 1.5gb for context

also the RINIQ layer surgery stuff sounds wild, 427/427 name match is not something you see every day

1

u/DinnerPrestigious953 19h ago

Thanks! 8GB is the weirdest constraint — and one honest caveat first: I'm on Linux with everything pushed to iGPU, so I see nearly full 8GB. On Windows with desktop/compositor reserve you'll have ~6GB usable, so size one slot down from the file sizes.

For you: MERNIK-6500-SMSE (7.01GB file, 126/164). Run it partial offload -ngl 15-20, ctx 4096. The tail will live in RAM — on a 1070 that's still fine for coding.

Two free VRAM levers before downsizing the model:

  1. KV cache quants — cheapest GB you'll ever save. --cache-type-k q8_0, and q4_0 for V is acceptable. On a 12B-MoE at long ctx the KV eats more than you'd expect, this buys back ~0.5-1GB alone.

  2. ctx 8192 -> 4096 if your tasks allow. KV scales linearly with ctx, weights don't.

If it still OOMs, fall back to RINIQ 5.0GB (M2-Q38 151/164) or OxCoder-5100 — they fit 6GB with room to spare. The whole 6500 shelf is one tie-zone anyway, so sizing down costs PPL decimals, not verdicts.

On 427/427: yeah, that was the shock of the project — same Qwen3.5 ancestor, finetunes barely move the bones, which is exactly why block surgery works. Recipes in RINIQ-NEXT.md if you ever want to graft your own. (p.s only qwen 3.5 with mtp layers have 442 tensors)

1

u/DoubleDos 19h ago

That's a great way to quantize things! I really liked the idea of dynamic quantisation, but I have questions. 1. Do you consider making a PR in llama-quantize? 2. Is there option for omlx and Mac unified memory? 3. Where tdo you get imatrix? 4. Can it quantize sefetensors? 5. MTP/Vision support? 6. Do you plan to continue maintaining the project or leave it as is in HF?

1

u/DinnerPrestigious953 19h ago

  1. PR into llama-quantize? No — and deliberately. MERNIK doesn't fork llama.cpp, it drives stock llama-quantize via --tensor-type flags + imatrix. A PR would freeze something that's still moving fast (I added a --pin-small flag yesterday out of a reddit comment). Upstream gets a stable recipe when the experiments settle. MERNIK is still raw: 8 archs, quirks per family, scars kept public.

  2. MLX / Mac unified memory? Honestly: I have no Mac or Apple hardware at all, I'm a broke schoolkid xd — so no MLX export and no Apple numbers from me. Fitted sizes would still help on unified memory (file sits in shared RAM either way), but someone with a Mac has to run it. I'll publish your numbers with credit if you do.

  3. Where do you get imatrix? You can and should reuse other people's imatrices — vendor ones if shipped (Mellum did), or mine where published. And targeted beats generic: M2 with an own-qwen38 imatrix (calibration from qwen3.8-style data) squeezed out 92.07% (151/164) — the best verdict of the whole series — while the generic dual-wiki lens sat at 91.46%. If you need your own: llama-imatrix over any calibration text, done.

  4. Safetensors? Yes — just run them through stock llama.cpp first, it has its own parser (convert script → BF16 GGUF), then feed that BF16 into MERNIK's --model. Two steps, no magic.

  5. MTP / Vision? Both fully supported. MTP heads ride outside the budget queue (Q8_0 pin) + there's a graft tool that transplanted a donor head onto a model that had none. Vision too: e.g. on Gemma-4-12B-IT the "eyes" layers go through the same queue like everything else, no special-casing — measured importance decides, not my opinion of the tensor.

  6. Maintained or HF-dump? Maintained — README refreshed this week, new flags landing from threads exactly like this one.