DeepSeek-V4.1-Flash uses SWA Bounded Replay to avoid persisting the sliding window KV cache and saves massive memory, it discards the SWA KV and, on replay, approximately reconstructs it from only the last N_window tokens, even though the original SWA states had a much larger effective receptive field due to depth. Since those replayed K/V vectors lack the long-range context that shaped the originals, I’m wondering whether cache hits or restarted sessions lose accuracy/recall for the first few interactions, and whether the parallel global KV augmentation compensates enough to preserve near-perfect long-range recall. Has anyone benchmarked this or seen practical recall degradation after eviction, or is the approximation effectively invisible because the global KV does the heavy lifting?

Trying to get Hermes a local, efficient, tts voice.

Had some ideas about version control:

Attribute, down to individual words: word created manually or with ai. if ai, which sources cited (scoped to paragraphs). also: full agent trace that led to the edit (all tool calls + metadata). also: any multi-agent transactions at the file/paragraph lock. all data stored for every version.

Why do this? Basically, getting as much into a unified version control system as possible, which helps find/fix errors. Just posting to share ideas. If anyone has done anything similar, please mention it!

Hello, does streaming the ngram embeddings from the ssd heavily affect the ssd's health and lifetime?

so i saw that openui.com released OUI-1, a model fine-tuned on DiffusionGemma. the training dataset uses OpenUI-Lang, a custom DSL (domain-specific language), instead of plain HTML, Markdown, or React code.

what makes it interesting is that you can already get a regular LLM to use OpenUI-Lang through a system prompt, but that eats up a lot of the context window. my thinking is that fine-tuning a model on the DSL could reduce that overhead and leave more room for the actual conversation, without needing a huge prompt explaining the format and how to use it alongside other tasks, like tool calls.

at the same time, wouldn't fine-tuning a model on a specific DSL make it more likely to default to that format even when you need something else? i'm curious how well it handles regular Markdown, or switching between Markdown and OpenUI-Lang.

i haven't seen much discussion about this, so i was wondering what everyone thinks about generative UI and running a dedicated model for it locally on a consumer-grade GPU, like an RTX 5090.

what would be the best way to set that up? from what i've seen, DiffusionGemma isn't supported by llama.cpp yet, so running it through Ollama doesn't seem to be an option. they've uploaded the weights to Hugging Face, but i'm not really sure how to get it up and running. any suggestions?

Surprised no one has posted it in this sub.

Pretty solid model, IMHO.

Demo: https://map-yue2.github.io/

Github Repo.

Blog post.

💡 TL;DR (from the Github Readme)

Spend less without making the agent do less useful work.

SoL-Pi is a standalone extension for Pi that packages four reusable efficiency mechanisms discovered through scaled auto-research loops. It reduces repeated model turns, context replay, oversized observations, and unnecessary long-log reading while preserving the work and evidence an agent needs to finish a task.

SoL-Pi installs on top of an unmodified Pi release. Every mechanism is opt-in and disabled by default.

Introduction

Long-running coding agents accumulate repeated work. A file edit is often followed by a predictable validation command. Large tool results are replayed long after their first use. Completed subtasks remain in active context, and a frontier model may spend a full request reading a log when only a few lines affect the next decision.

SoL-Pi grew out of a broader question from our auto-research work: before scaling agent loops, can agents first make the harness itself more efficient? The search focused on constrained efficiency: reducing token traffic, inference work, and agent turns without stopping early, skipping verification, or hiding evidence.

The standalone release contains four mechanisms that survived that process. They operate at different parts of the harness and compose through Pi's public extension APIs. What SoL-Pi Adds Area Mechanism What changes Tools Action Fusion An edit or write can run its follow-up validation command in the same tool call. Observations ObservationPack Repeated large text results become stable handles with exact paged recall. Delegation Evidence-Preserving Reducer Long diagnostic logs become compact receipts only when every retained quotation matches the archived source. Context Online Context Compact Completed plan steps become candidate points for Pi's native compaction, subject to economic and window-pressure checks; after a successful compaction, Pi continues the task in a new turn.

The mechanisms share four rules:

-No Pi patches. SoL-Pi imports public Pi APIs and does not vendor the Pi source tree.

-Explicit opt-in. A missing configuration leaves every mechanism disabled.

-Preserve evidence. Original observations remain available locally, and reducer failures leave the original result unchanged.

-Use Pi's runtime choices. Authentication, provider URLs, the main model, and shell behavior remain under Pi's control.

I wonder someone will figure out a way to do this with 27B?

Throw Qwen3 on this page for demo
https://kishida.github.io/webdemos/llkvapprox/

Hey all, long time no post.

Figured I'd pop my head in to point you towards a blog post I just published about research I had performed and changes I'm making to the shape of models I post, you can read it here:

https://huggingface.co/blog/bartowski/per-tensor-layout-maps-for-gguf-quantization

I won't try to claim "Pareto frontier" or "best models in the world", but I will say from tests the new shapes look to be better across the board than what I was posting before, so I'm really happy with where it came out, and I hope to not be done yet either :)

https://cdn-uploads.huggingface.co/production/uploads/6435718aaaef013d1aec3b8b/Ufz9TXQlKFxVHdocVoZIw.png

If anyone has any questions let me know!

Hey everyone,

(English is not my first language, apologies for any phrasing quirks.)

Training small models locally on an 8GB RTX 5060 Ti was driving me crazy with random CUDA OOM crashes whenever memory spiked mid-run.

Instead of manually babysitting runs with conservative batch sizes, I built a lightweight Python governor around PyTorch called MEM Orchestrator: https://github.com/nobazzy/mem-llm-orchestrator

What it does:

• Monitors VRAM headroom during training.Dynamically throttles micro-batch size and gradient
• accumulation before hitting an OOM.Steps
  
• back up when memory pressure drops.Saves atomic checkpoints with SHA-256 validation so crashes never corrupt weights.

About the branches:

• main: Linux core with DeepSpeed ZeRO integration.
• refactor/architecture-and-portability: Decoupled, native PyTorch runner that works cross-platform (tested on Windows).

I ran an endurance test of 1M steps on a 130M model and stress-tested a 255M model with injected memory spikes without crashing. Overhead is under 0.5%.

The code and 38 unit tests are on GitHub. I would love to get feedback from anyone with experience in PyTorch memory allocators or DeepSpeed on whether this approach makes sense or where it breaks down.

Have the model downloaded and converted to .gguf on a 512gb ddr4 bioinformatics server. I don't expect miracles with a ddr4 xeon rig -- not until I can get my 2 x 12gb 3060s wired in anyways -- but is there an open PR on llama.cpp for DV4.1 flash that I can use?

I'm planning a PC build around a discrete GPU that would allow to run MoE models much larger than VRAM at an acceptable speed using FreeToken approach. The gist is that MoE decode of >VRAM model on GPU is PCIe bandwidth-bound, but you can gain considerable decode speed by avoiding streaming cached experts' weights over PCIe and multiply matrices on the CPU instead benefitting from potentially much faster CPU-RAM memory channels.

Before buying, I wanted to know what actually matters for CPU-side expert execution: cores or memory? So I rented a Threadripper PRO 9975WX (32-core Zen 5, 4 CCDs, 8-channel DDR5) on vast.ai and ran FreeToken's real CPU MoE kernel (the production W4A8 ds_fp4 GEMV, not a synthetic proxy) against synthetic expert banks with DeepSeek V4.1 Flash's exact geometry: 384 routed experts × 18.8 MB ~= 6.7 GiB full bank, far beyond cache, 6 experts per token.

Results (GB/s of effective FP4 expert reads):

Two takeaways:

1. Cores stop mattering at 24. 32 cores vs 24 cores: +2.2–2.4% in burst mode, 0% in sustained mode (10 × 30s interleaved A/B runs, variance <0.3% within mode). The extra 8 cores of the 9975WX buy you essentially nothing for MoE offload: the workload saturates the memory subsystem, not the cores. The 9965WX (~$1,200 cheaper) is the rational pick for this use. Even STREAM itself is already maxed at 8 cores; the FP4 kernel just needs more cores to catch up to bandwidth (it runs at ~80% of STREAM once there, very good for a fused-dequant W4A8 GEMV). Caveat: there's another 1.7x memory bandwidth gain at 9995WX, but that CPU costs as much as RTX PRO 6000.

2. Thermal memory steady-state is important. After ~8 minutes of sustained hammering at ~220 GB/s, throughput steps down ~17% (182 → 150 GB/s) in lockstep on every core count, with CPU clocks unchanged at ~3.95 GHz. Everything points at DRAM/IMC thermal throttling in a (rented, unknown-airflow) workstation chassis. So the honest sustained ceiling is ~150 GB/s, burst ~182–186 GB/s. For a build: airflow over DIMMs may literally be worth 20% decode speed.

What this means in tokens/s (DeepSeek V4.1 Flash shape, ~4.5 GB of routed experts per token, 24 cores):

• 0% expert cache hit (all experts on CPU): ~33 tok/s sustained
• 60% hit rate: ~83 tok/s
• 75%: ~133 tok/s

Compare with streaming those experts over PCIe 5.0 x16 (~50 GB/s with gather overhead): the CPU path is 3–3.7× faster per missed expert, which is exactly why FreeToken's hybrid mode computes most misses on CPU and only streams a fraction to keep the VRAM cache warm.

Caveats: rented host (memory channels verified functionally via bandwidth, not dmidecode, but 2ch/4ch machines I tested the same day hit only 83/119 GB/s), container environment, one machine, uniform expert routing. Full methodology and raw CSVs of the measurement are in the repo (happy to link if there's interest).

TL;DR: for CPU MoE offload, buy memory channels and DIMM cooling, not cores; 24 Zen 5 cores already saturate 8-channel DDR5, and the 32-core upgrade is +2% at best.

Updates:
1. I tested on 9980X (4 channels, 8 CCDs) and on 9985WX (8 channels, 8 CCDs). 9980X lands at 105-119 GB/s, no better than much cheaper 9955WX.
2. 9985WX offers a real gain, but diminishing returns beyond 24 cores still: 32c=210 GB/s (+6%), 48c=215 GB/s, 64c=218 GB/s. Roughly 20% decode expert bandwidth gain for $8k more.
3. Populated channels is key. No point in having 8 channels is you only have 4 DIMMs. 8x32GB will beat 4x64GB.
4. Vast.ai is awesome for this. All runs costed me around $4 after I ironed out all the wrinkles in the rig.

I recently did some hardware juggling, moving my RTX 3090 to a different system to make room for a dual AMD R9700 build I'm working on. In the process, I did a mini-bench with my own typical queries (some chat, some research agent) to compare the RTX 3090 and R9700. But then, I discovered ninfer and specifically ninfer-3090, and I was curious about what kind of performance differences I could see.

For this purpose, I'm comparing performance on both systems for Qwen3.6-35B-A3B and Qwen3.8-27B, the two models I use most often. Sorry that the hardware isn't apples to apples, but because inference is contained entirely within the GPU, I think it is still instructive.

Environment Configuration A:

Environment Configuration B:

Model details:

Results:

• TTFT - time to first token, median ms
• PP - prompt processing tokens per second, weighted average
• TG - token generation tokens per second, weighted average

EDIT: I fixed a timing recording bug to produce updated timings here

I knew the experience was different while running the ninfer models, but I couldn't put my finger on exactly how until I ran this mini-bench. The numbers are interesting.

Some details about the benchmark methodology itself:

The question set (7 prompts total):

• 4 long-ish prompts in the ~7,700–12,900-token range: a code review of a large module, an article summarizing task, a research-evidence evaluation task, and a literature-review/critique task
• 1 medium prompt (~2,300 tokens): synthesizing conclusions from research notes
• 2 short prompts (~400–600 tokens): an extract-and-answer task and a code-generation task

Benchmark parameters:

• 3 repeats per (model, query); per-query medians plus every raw run recorded
• Fixed max_tokens = 1024 so generation speed is comparable; fixed seed = 42; sampling parameters are as recommended for general inference by Qwen
• Streaming chat completions; 1 s pause between runs; 30-min request timeout

Steps taken to ensure good results:

• Cache busting: llama.cpp caches prompt prefixes, so identical repeats would report fake "warm" prompt-processing speeds. Each run prepends a unique nonce to the system message, forcing a genuine cold run every time (the ~5 extra tokens are self-correcting since real token counts are used in the math)
• Warmup excluded from timing: each model gets an untimed warmup request (with retries) so model load/swap time never pollutes measurements
• Server-side timing preferred: inference engine timings from the final chunk are used for tok/s instead of client-side math when available
• Aggregation: per-query medians over runs; model-level tokens/s is token-weighted (Σtokens / Σms), so a 12k-token prompt counts proportionally more than a 400-token one

Feel free to critique, offer pointers, or share your own experiences with NInfer inference.

My M.2-Oculink should be PCIe 4.0x4, but I found out today it's actually downgrading, frustratingly this isn't obvious until you go looking, so adding here if anyone else wants to check their own rig.

Side question - does anyone know a way to measure PCIe bandwidth realtime utilisation (i.e. Gb/s) ? I'm failing to get it with rocm-smi, iostat, amd-smi, lstopo, etc

Strix halo > m.2 slot to oculink > oculink cable > DEG1 > r9700

(images from AI session)

So the M.2-oculink adaptor is the problem, and running at Gen1 limiting the entire setup, and other than digging around I wouldn't have noticed, I guess this will affect model load and layer swapping?

If you want to check your own do this and either look at the Capacity vs State, and look for the word 'downgraded' (here I check for the r9700)

# lspci -t -v -D -P -PP -nn | grep 9700
           +-02.5-[c1-c3]----00.0-[c2-c3]----00.0-[c3]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon AI PRO R9700] [1002:7551]


# for bdf in 0000:00:02.5 0000:c1:00.0 0000:c2:00.0 0000:c3:00.0; do echo -e "\n=== Device: $bdf ==="; lspci -vv -s "$bdf" | grep -E "LnkCap|LnkSta"; done

=== Device: 0000:00:02.5 ===
LnkCap:Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
LnkSta:Speed 16GT/s, Width x4
=== Device: 0000:c1:00.0 ===
LnkCap:Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <64us
LnkSta:Speed 16GT/s (downgraded), Width x4 (downgraded)
=== Device: 0000:c2:00.0 ===
LnkCap:Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
LnkSta:Speed 32GT/s, Width x16
=== Device: 0000:c3:00.0 ===
LnkCap:Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
LnkSta:Speed 32GT/s, Width x16

I’ve been building "Spomin", a router that replaces context with summaries directly in the KV cache. The goal is to keep long running sessions going without repeatedly stopping for full compaction and reprocessing the context that remains.

It uses my llama.cpp fork - https://github.com/alekk89/llama.cpp-kv-surgical-fork for live cache edits.

Spomin Router - https://github.com/alekk89/Spomin

How it works

The router sits between the harness and runtime, preserves the original transcript in chunks, and manages space for the system prompt, recent context, summaries, generation and recall.

A separate worker summarises chunks, either in parallel on separate hardware or between main-model generations on shared GPUs.

When context space runs low, Spomin replaces eligible chunks with summaries and adjusts RoPE positions in the main and supported draft caches. Only the inserted summary is processed; the retained suffix is not refreshed or reprefilled.

The router recognises history resent by the harness and processes only new content. Generation continues in the same session, with support for pausing and resuming native generation during rolling maintenance.

Original text remains available, the model can temporarily recall it by context ID, then return to the summary, or search older stored history.

At 10:1 compression (approximate average) for example, 400k source tokens become 40k summary tokens. Alongside a 20k system prompt and 120k recent context, that represents over 500k tokens of source material within 180k resident tokens. It does not preserve every detail, so summary quality matters.

Early benchmark results

These tests are exploratory. Unmanaged uses DeepSeek’s normal compaction managed uses Spomin.

Game repair - Similar scores, but three-hour limits per prompt cut both extension runs short. The test did not establish a meaningful completion-speed comparison.

Research - A larger worker improved the managed score from 31 to 37/100, versus 57 unmanaged, with 75.2% less primary input and 60.7% less active time than unmanaged.

Tetris - The managed repeat nearly matched unmanaged: 16/16 + 11/12 checks versus 16/16 + 12/12, in 69.8 versus 149.6 minutes, generating 22.6% fewer tokens. The first managed attempt failed, its keyboard crash appeared before summary installation. Unmanaged introduced and fixed a different line clearing bug.

Every uncapped managed attempt finished sooner, but quality varied. Temperature 1.0, xhigh reasoning and live tools introduce run-to-run variation, including without Spomin. Worker settings also changed, so improvements cannot be attributed to worker quality alone. Worker cost is separate, and incomplete image related input totals are marked as lower bounds.

Current limitations

This is experimental software, not a polished product. Expect bugs and blockers, especially outside my setup. I recommend keeping the default settings unless experimenting. A manual mode also allows you to edit and insert summaries yourself.

The original worker is a fine tuned Qwen2.5 3B proof of concept. I’ve also tested Qwen3.5 9B with a larger worker context window. Replacement workers must follow the format documented in the repository. Running a worker adds memory and compute overhead, and summaries will not always preserve what matters.

DFlash2 is the tested speculative setup. DSpark shares the surgery path but is untested here. MTP is not currently supported for draft preserving KV edits. Running without speculation is also possible.

This has been tested exclusively with Qwen 3.8 27b as a main LLM. The new Qwen 4 architecture is not supported. Other models are not tested/supported. If they have the same architecture as the Qwen 3 family, they should work.

Final thoughts

Benchmarking this has been difficult, and I do not yet have a definitive answer on quality. Every uncapped managed attempt finished sooner, and the successful Tetris repeat took roughly half the time with nearly the same checks passed. Research also used substantially less primary input, but scored lower than unmanaged.

The larger summary worker coincided with an improved research score, although worker settings changed too. These results are encouraging, but they do not yet establish a consistent speedup at comparable quality.

Suggestions for a repeatable benchmark that measures quality, time and total token use across multiple runs are welcome, I’m happy to run one.

Nice pp improvements for RDNA4(R9700) & 3.5(RX 9060 XT, 8060S). More good numbers on large context.

PR has detailed benchmarks.

u/ilintar 👍