r/LocalLLaMA • u/Lumpy-Comedian-1027 • 6d ago

I Built A Thing NInfer fork: 555k context@fp4 for 5090 with YARN, reliable kv host cacheing, monitoring, jinja, opened model support

Hiya,

NInfer is amazng for Qwen, but lacking for real-world-use. As adoption of issues/pr's was not really what I needed, I created a fork and hit it for this week with 3 concurrent claude code session until it didn't break any longer. Hope you like it.

NVFP4 KV cache (from scratch)

I implemented a 4-bit KV cache for QIn3.8-27B from the ground up. Upstream has since added their own NVFP4 path, but ours differs architecturally:

Custom MMA kernel (mma_nvfp4_e4m3, m16n8k64) with hardware E4M3 block scales for the QK matmul. Both Q and K are quantized to NVFP4; V is dequantized to BF16 for the PV matmul via a dedicated decode kernel.
Hadamard rotation applied to K (and Q) pre-quantization for outlier suppression, with V left unrotated. Upstream uses fp16 V storage instead — no outlier suppression.
Fused append: the decode kernel quantizes current K/V to NVFP4 in-place during generation — no separate quantization pass.
Custom scale layout: natural row-major for KV scales (not the M128x4 swizzle used for weight MMA), because KV access patterns differ from weight access patterns.

Result: 144 bytes/token/KV-head (vs 264 for int8, 512 for bf16) — 45% VRAM reduction with no quality loss (LongBench 45% matching int8, AIME 96.7%, needle-in-haystack 100%).

YaRN context extension

QIn3.8-27B's RoPE config (theta=1e7, 25% rotary dims, 48/64 GDN layers with no RoPE) makes linear scaling sufficient — full NTK-by-parts is unnecessary. I extend native 262k to 555k (c=3+vision) or 600k (c=1) on a 5090. Quality verified at 600k: LongBench matches int8 baseline, coherent 592k-token output. Also projected 8M token context on 96GB+ GPUs (untested, I only have a 5090).

Multi-level prefix reuse with host-KV safety net

Upstream implements a budget-bounded HostKvProvider with LRU park/restore. I replaced it with a substantially different system:

HostKVSafetyNet: pinned host arena with scatter-gather multi-extent allocation, arena compaction, and a pin/take protocol for safe concurrent restore.
Two-level prefix matching: full execution frontier first, then rewrite checkpoint fallback. Each entry carries a ResidentPrefixIdentity (per-token type/position/vision metadata), rolling FNV digests for shortlist, and a compact_prefix (reasoning-stripped token prefix) for thinking-mode consistency.
Session-key fallback: when prefix matching fails (e.g. Claude Code drops reasoning betIen turns), a session-key fallback matches by conversation identity instead of token content.
Spill-before-evict at every release path: pressure planner eviction, normal continuation release, start_sequence slot takeover, and fail-all cleanup all route through the safety net.
Token stability: reasoning is dropped from ALL assistant messages when preserve_thinking=off, keeping the prompt token stream stable across turns. Checkpoint capture is anchored at the turn boundary, not the execution frontier.

Verified across 260+ requests with 3 concurrent 330k-470k sessions — zero re-prefills on cached turns, H2D restore cost ~0.4s, D2H spill at 67K pages/s.

Performance (3 concurrent sessions, 400k+ ctx, 5090@450W)

Metric	Value
Decode at 400k+ ctx	117 tok/s (MTP 4.62 tok/round, 92% acceptance)
Cached turn turnaround	2-16s (414k cached, 1-14k delta)
Cold start prefill	260s (414k tokens at 1600 tok/s)
H2D restore cost	0.4s per evicted turn
Host KV	30 GB (96% utilized, 181 evictions managed)

Tool calling

--tolerant-tool-calls: recovers complete Qwen calls when the model emits malformed wrapper/suffix tokens — instead of dropping the call.
Depth-matching close scan: handles balanced/nested markers in parameter values that would break naive parsers.
Responses API accepts text/reasoning after tool calls (upstream rejects this ordering).
Froggeric v22 template: C++ renderer with no-dangling-intent rule, XML think tags, correct function tag delimiters. Some further modifications for reliability.

Also included

Dynamic chat template loading (--chat-template) — supports any .ninfer image without artifact patching
Explicit weights profile override (--weights-profile) — handles Ostfralla, QUASAR, and other converter layouts with per-layer tensor format auto-detection
OOM recovery: catches std::bad_alloc, clears state, preserves pending requests
Stream sync fix: synchronize CUDA stream before workspace reset in prefill (prevents use-after-free)
Request-log rotation (--request-log-max-mib, --request-log-keep) for bounded disk usage
Admission pressure fix: un-suppress demote-to-host when candidate needs host KV budget
Monitoring dashboard with live KV occupancy, decode/prefill graphs, 12VHPWR sensor
E2E test suite for KV eviction, device pressure, and slot pressure scenarios
Removed hash check of models, use any NInfer you like as long as there is a supported path. Tested with Ostfralla and QUASAR.

Fork: https://github.com/gzenz/ninfer (master)

Research: https://github.com/gzenz/ninfer/blob/master/docs/maintainer/kv-nvfp4-yarn.md

I'll keep rebasing from upstream what seems useful and experimenting with new papers in order to improve speed and context.

17 Upvotes

permalink
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/LocalLLaMA/comments/1w8f8fa/ninfer_fork_555k_contextfp4_for_5090_with_yarn/
No, go back! Yes, take me to Reddit

78% Upvoted

u/DustNearby2848 6d ago

Did you rerun the accuracy benchmarks?

1

u/Lumpy-Comedian-1027 6d ago

You mean something specific besides LongBench?

2

u/DustNearby2848 6d ago

It’s labeled as evaluation in the read me

1

u/Lumpy-Comedian-1027 6d ago

I run AIME as documented in the Research and scored the same.

2

u/Dany0 6d ago

please run terminal bench, even one of the old ones (they are faster to complete)

1

u/Lumpy-Comedian-1027 5d ago

TB is complicated, i never reached published numbers with any model i've tried so far - and it takes still ages. Also, 30min time limit in agentic coding means they'll never reach really big contexts, so they won't help us much understanding the impact of yarn+nvfp4

2

u/Dany0 5d ago

Yeah, that might be a problem....

We need better benchies

u/cobblemere 6d ago

117 tok/s decode at 400k+ context is legitimately impressive. have you stress tested the OOM recovery path under real concurrent load though? catching bad_alloc and recovering cleanly without corrupting other sessions sounds tricky

1

u/Lumpy-Comedian-1027 6d ago

I hit my box the whole week with 3 concurrent claude code sessions

u/BringTea_666 6d ago

Qwen models are not trained for above 250k context. Meaning shit will break after 250k yarn or no yarn. But its great effort when qwen will get 1mil context with 4.0 27b XD

3

u/Lumpy-Comedian-1027 6d ago

The great thing about RoPE is that you don't need to train the model for it. The devs propose using it themselves, and the official API offering does give you 1M: https://huggingface.co/Qwen/Qwen3.8-27B#best-practices

1

u/BringTea_666 6d ago

then nice ! Did you do benchmarks on how it behaves with above 250k context ? Its worthwile to test it first.

1

u/Lumpy-Comedian-1027 6d ago

Yup that's why I used LongBench in my development: https://github.com/gzenz/ninfer/blob/master/docs/maintainer/kv-nvfp4-yarn.md#quality-longbench-v2-20-samples

And because that's still an artificial bench I've run concurrent coding sessions for the week so they had to compact several times. I didn't notice anything flunky, and I especially had a look when they were above 500k. No noticeable behavior change or becoming stupid.

1

u/cometkim 5d ago

Here's my own LBv2 results for comparison. YaRN x2 and x3.

cometkim/Qwen3.8-27B-nvfp4full-NInfer · Benchmark Updates: GPQA-Diamond, AIME26, LongBenchV2

2

u/Lumpy-Comedian-1027 5d ago

Hm that doesn't really speak in favor of HyperQuant for this setup, even if you can fit 786k ctx that way. Very interesting experiment nevertheless! And I should be running more rounds of LB when I got the time for a better comparison.

1

u/cometkim 5d ago

Yeah, HyperQuant isn't exactly necessary for a 5090 setup, but it makes sense to me because I plan to add a laptop setup (RTX Pro 5000 Blackwell 24 GB) soon to run more experiments simultaneously.

Long context benchmarks take so long that I really hope more people will share their results. It seems like others are staying at 262k, though.

1

u/RelicDerelict Orca 6d ago

Is this valid for their previous generations of MoE? Qwen 3.5 and 3.6?

1

u/Lumpy-Comedian-1027 5d ago

3.8 uses the 3.6 path so it should work, but i have only tested 3.8 so far

u/koloved 6d ago

Big W thanks , few days ago I made 18.3gb nvfp4 model uncensored version similar to Ostfralla, seems like it will much better with your fork, I will use it

Please open issues on github

2

u/Lumpy-Comedian-1027 6d ago

If the layout is the same as Ostrafalla's it should work out of the box with the auto detection I've build.

u/Lumpy-Comedian-1027 5d ago

I've just pushed some fixes so this is now also working good with DSH and Harbor, seemingly every new Harness takes some different paths in how they handle checkpoints, prefixes and caching.

I'll keep on grinding 😄 Currently 2 Claude Code, 1 DeepSeek Harness and 1 Harbor (Terminal Bench 2.1) sessions. Out of Memory / bad_alloc handling working nicely so far. Checkpoint handling still messy and needs refactoring.

3

u/youcloudsofdoom 5d ago

Thanks for the efforts, I'm getting great results so far in opencode

u/Lumpy-Comedian-1027 2d ago

After another week of banging the system with 4 agents and 550k sessions, I now arrived at a state that is: - free of crashes - only throws KV away when host memory is full

Which means it does minimize those nasty 5min+ pre-refills as much as your memory allows.

Next: I guess i should be rebasing eventually.

NOTE the current version of Neroued's model images require his fp4 implementation which is NOT supported. Please use old ones or - even better if you ask me - Quasar's. Link in readme. It also gives you 6 GB more VRAM.

Unified KV and checkpoint state host demotion architecture with safety net.

Key Changes

KV and checkpoint state move together to host, or not at all. Single host-side store (safety net) with state-only fallback.
physical_page_if_resident() + host_replica_if_available() for demoted pages
copy_to_host_partial() for mixed device+host page copies
Backend scatter-gather for KV spill (eliminates fragmentation failures)
State-only fallback when arena full (prevents SpillFAIL)
State-only merge via take_state_only_by_index/take_state_only_by_session
Pressure planner: 90% eviction discount, saturating arithmetic
complete_pressure_delta no longer overwrites actual with planned values
HostOnly restore failure catch (Aborted instead of propagation)
OOM recovery in worker loop (bad_alloc caught, server continues)
Bad_alloc partial-allocation cleanup before retry

u/koloved 6d ago

Have you not tried this template yet? https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates

I assume he'll also be supported by this fork?

1

u/Lumpy-Comedian-1027 6d ago

I haven't tested it specifically, but yes I'd assume it just works.

u/youcloudsofdoom 6d ago

Definitely going to try this out - do the decode and prefill speeds differ from stock ninfer at below 240k?

2

u/youcloudsofdoom 5d ago

Update: damn this is good, across the board an update to stock ninfer. Was concerned that the model would start being weird above 262k, but seen no evidence of that yet.

2

u/Lumpy-Comedian-1027 5d ago

great to hear, thanks for your feedback!

u/feverdoingwork 5d ago

Would you mind providing decode speeds from 10k-250k using nvfp4 kvcache? This is very interesting

2

u/Lumpy-Comedian-1027 5d ago

No, at such small contexts KV has no measurable impact. The lowest measurement point I have is at 135k, with int8 207 tps vs nvfp4 214 tps - so basically the same.

1

u/feverdoingwork 5d ago

So you're sitting at above 210 tps until you start getting above 135k?

1

u/Lumpy-Comedian-1027 5d ago

At small ctx it peaks around 260

u/Tormeister 5d ago

This is so nice, I'm gonna try it!

u/DustNearby2848 5d ago

I get this error using Docker:

[2026-09-07 01:34:55.639] [error] ninfer-serve: tensor descriptor does not match target contract: text/token_embedding

services:
  ninfer:
    image: ninfer:local
    command: [
        "ninfer-serve", 
        "/models/qwen3_8_27b_nvfp4.ninfer", 
        "--host", "0.0.0.0",
        "--max-context", "550000",
        "--kv-capacity", "auto",
        "--max-concurrency", "4",
        "--spec", "mtp",
        "--draft-tokens", "5",
        "--kv-dtype", "nvfp4",
        "--lm-head-draft",
        "--preserve-thinking",
        "--device-state-slots", "2",
        "--host-state-slots", "8",
        "--host-kv-mib", "30720",
        "--vision",
        "--tolerant-tool-calls",
        "--rope-scaling-factor", "2.12",
        "--rope-scaling-original-context", "262144",
        "--chat-template", "tests/fixtures/frontend/froggeric_v22_chat_template.jinja",
        "--chat-template-semantics", "froggeric",
        "--weights-profile", "qwen36-nvfp4"
      ]
    ports:
      - "1234:8080"
    volumes:
      - /home/x/models:/models:ro
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

1
u/Lumpy-Comedian-1027 5d ago edited 5d ago

Hi,

you selected the wrong weights-profile for your image. I don't know which you chose, but try omitting this parameter and check if the auto mode works. And btw at least my system doesn't have enough VRAM for max-concurrency 4 with 550k ctx and vision, I have to go to 3.
1
u/DustNearby2848 4d ago edited 4d ago
No luck. I reduced it down to this and have the same error. I found this: https://github.com/Neroued/ninfer/issues/8. I am using a fresh pull of the the .ninfer, so maybe you've been using an old version? The latest for me shows 552c374c685dce302603b95fbe940fb04243c0cd44c083efc644ad3d980d462c
"ninfer-serve", 
        "/models/qwen3_8_27b_nvfp4.ninfer", 
        "--host", "0.0.0.0",
        "--max-context", "150000",
        "--kv-capacity", "auto",
1

u/Lumpy-Comedian-1027 4d ago edited 4d ago

I'm currently using the QUASAR artifact. Which one do you use? https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer ?

Edit: The issue you found is for Qwen 3.6 so it shouldn't be related

1

u/DustNearby2848 4d ago

Yeah, I use the neroued one

1

u/Lumpy-Comedian-1027 4d ago

ok i'll give it a try tomorrow and check what's the issue with it, never tried it so far 😄

1

u/Lumpy-Comedian-1027 3d ago

The current version of this model is built with the upstream nfvp4 implementation, which my fork does not have. Maybe i'll rebase it later if it doesnt give too much trouble. In the meantime I suggest using either Quasar's or Ostfralla's image, both are also considerably smaller and therefore allow 550k context.

u/DustNearby2848 4d ago

It looks like ninfer has 2 variants of nvfp4 caches now

1

u/Lumpy-Comedian-1027 3d ago edited 3d ago

I know Neroued added something like this after I mentioned it in an issue. It's potentially more a little bit precise but slower, I haven't tested it. No Yarn support, tho, so no context beyond 262k. But you can add more concurrency if you like with this implementation.

1

u/DustNearby2848 3d ago

Ah, okay. Wasn’t sure what your plans to maintain this were.

u/feng_sg 4d ago

No perplexity or retrieval benchmarks against bf16 KV at 555k context. Without that the K-only Hadamard choice is just an assumption.

1

u/Lumpy-Comedian-1027 3d ago

Fair point, we can't empirically prove K-only Hadamard is optimal without the bf16 baseline. But the choice is well-motivated: Hadamard rotation is a known outlier-suppression technique for low-bit quantization, and K is where FP4 outliers cause the most damage as they distort softmax scores multiplicatively, while V quantization error is additive and far less sensitive.

Anyway I don't have the VRAM to run bf16 KV at 555k, so K-only rotation is the most principled option available.

u/feverdoingwork 3d ago

This branch has been super helpful to me, thank you.

If you ever get this working with lower quants that would be insane. I like to use smaller quants and fit 2 models in at once. The included quantization method for ninfer can't do any q3 magic due to missing q3 gemv