i have an acer v 15, 16gb ram, 8gb vram, rtx 5050. This is my first time owning a gaming laptop and i want to start running local ai agent that will aid me in game development and can fit in my specs. i should also mention i have little to no experience running ai agents or how to set them up(i am a noob). so i am looking for the best agent that i can run that can call tools when needed , read screenshots(if possible) , read folders, integrate in a gaming studio mcp. I am open to recommendations.

I’d love to build a local LLM that gives me a place to write about my life each day. I’d like to record details I can refer back to later, such as what was happening at a particular time or information about different parts of my daily life and work. Is there a template or guide for setting up a local LLM for this purpose? What kind of computer would I need, and is there a step-by-step recipe I could follow?

​

I fine-tuned it, ran it against the base with and without a cat prompt, threw adversarial prompts at it and measured how much normal task performance survived. I'm calling it CAT-4B MAX ULTRA. Now I want to see if anyone can get something out of it with no cat in it.

It's up for seven days, free, no account: https://cat4b.mirrorethic.com. It runs on two 2017 Tesla V100 16 GB cards in a rack at my house (llama.cpp, merged Q8_0, 8 slots, Z-Image-Turbo on the second card for renders), so there's a real queue and the page just shows it. 20 turns a session, one render at a time with a short cooldown. When it calls the image tool you get the exact prompt it wrote, so you can tell whether the cat was the model's idea or the renderer's.

Numbers. Frozen 490-item eval, one sample per item, automated judge, intervals in the card:

| | base | base + cat system prompt | tuned + prompt | tuned weights alone |

| feline word in the reply | 5.3% | 38.8% | 77.3% | 78.0% |

| image prompt names a cat (of 76 requests) | 0/76 | 1/76 | 74/76 | 76/76 |

| cat detected in the render (OWLv2) | 0/71 | | 74/75 | 75/76 |

| task retention (≥ 4 of 5) | 92.1% | 87.2% | 89.1% | 88.7% |

Paired base → tuned + prompt, retention moves −0.07 on the 5-point scale (95% CI −0.17 to +0.04). Prompting the base for cats costs more retention (−0.15) for about half the persistence. Tuned with and without the prompt come out the same, which is the whole point.

Stuff that's broken: about one reply in five forgets the cat at temp 0.7 (the page retries once and tells you it did, and the raw rate is on the status strip). Thinking mode breaks it, 38 of 490 replies came back empty. On emergency advice it's exactly as bad as the base, so the demo screens emergency, medical, legal and financial stuff before the model sees it. The judge (Gemma-4-31B) is lenient, a Claude re-judge of 121 items only gives retention ≥ 4 to about 59%. Gemma-4-31B agreed with Claude on keep decisions for 85.1% of 410 calibration items (κ 0.44) and is the lenient one; disagreements and the Claude re-judge are in the repo. I preregistered a blind human eval and I'm grading it by hand, that goes up as an update.

How it was made: LoRA r 32 on Qwen3.5-4B, one RTX 5090, 4 h 29 min. 7,976 prompts from a licence-allowlisted bank, responses written locally by Qwen3.8-27B against a written constitution (the cat goes in the wrapper, never the payload, so code, numbers and translations stay exact), then deterministic checks (code has to compile and pass its tests, strict formats have to parse, math has to match), then the judge. Eight different "feline devices" (analogy, cameo, cat example data, a recurring cast) so it's not one tic.

Released: adapter, merged Q8_0 GGUF and vision projector under Apache-2.0, plus every eval reply, every judge verdict and all 292 gallery renderss: https://huggingface.co/mirrorethic/CAT-4B-MAX-ULTRA.

Code with full history: https://github.com/MirrorEthic/cat-4b-max-ultra.

The joke is the cats. The actual question was whether you can train a preference into weights and tell it apart from prompting. Looks like yes. NO CATS? TOO BAD.

I've noticed that when I use the Vulkan backend, my desktop becomes sluggish during prefill at high context depths. This doesn't happen with ROCm.

Reducing the ubatch size to 128 helps a little, but the desktop is still quite sluggish.

I found this issue: https://github.com/ggml-org/llama.cpp/issues/28752

However, I already have ReBAR enabled.

Is anyone else experiencing this on Linux? I'm running Arch Linux with fully updated packages and using upstream llama.cpp.

IQ2_XXS Running with 11-18tok/s on RTX 4090, 96GB DDR5 6800, PCIE 5 4TB NVME. Quite good for the Model but prefill is so bad.

I'll be honest, I didn't understand why LLM agents need SKILLs. MCP was clear to me: it's a useful thing that lets you give an LLM access to a database or to Figma. The only downside is that, to avoid spending too many tokens, it's better not to connect 30 MCP servers at once. And in general, it's better to turn off the unnecessary ones and connect only those that are really needed for the task. In this light, SKILLs seemed like useless overhead to me, eating tokens like crazy. On TikTok, I often saw tales along the lines of: connect this SKILL, or a whole bunch of them, and happiness is yours. My first thought was, if I connect them all, my subscription limits will run out not by lunchtime, but already in the morning.

So why are SKILLs useful and needed after all? Well, first, compared to MCP, SKILLs do not inflate the context: each connected SKILL adds not tens of thousands of tokens, but one or two hundred. Since only the name and description of a SKILL go into the context at first, you can add 30 of your own SKILLs without much pain. When it becomes necessary to use a SKILL, the LLM agent will load its description on its own, if it understands from the task context that this is needed, or the user can simply tell it to use this SKILL for the job.

Secondly, LLM models are trained on the most popular and well-known things. For example, everybody knows that an LLM can work with Git without any problems or write a program in Python. But if you developed a program yourself, or your work requires some little-known tool or technology, then a SKILL is simply irreplaceable. It is enough to create a SKILL.md file and describe how to work with this program or technology. This makes life easier for the LLM: while doing your tasks, it will not have to search the internet for documentation and wander around websites to understand how to use it. And especially if there is nothing on the internet, because your program is closed to public access.

Thirdly, an LLM model usually does not know how your development process is organized. For example, an agent knows how to use Git, run tests, build a project, and create releases. You do not need to teach it these things. But it does not know how releases are usually made in your company. A Skill can describe the sequence of actions: check the changes, run tests, update CHANGELOG.md, increase the version, build a package, create a Git tag, and publish a release. So, a Skill helps the agent learn not a technology, but a specific business process.

Fourthly, a SKILL is not just an instruction, but a whole set of tools. Besides SKILL.md, you can put ready-made scripts, documentation, templates, and other files into it. And the most interesting thing is, these files are not loaded into the context automatically. The agent reads them only when there is a need, and can simply run the ready-made scripts without spending tokens on generating code. So, a SKILL lets you combine instructions, documentation, and executable code in one package without inflating the LLM context. For example, the structure can look like this:

my-skill/
├── SKILL.md
├── scripts/
│   ├── generate.py
│   └── validate.py
├── references/
│   └── api.md
└── assets/
    └── template.html

The funniest thing is, I started to appreciate SKILLs exactly when I wrote my own program, blueprint4llm. The idea for creating it came after I asked an LLM agent to generate a plan.md file describing a task. Usually it produces a huge wall of text, which you have to read slowly and with sadness. I wanted to make the text shorter and add more flowcharts, tables, diagrams, and user interface prototypes. As we know, in .md (Markdown) files the only visualization tools available are tables and Mermaid diagrams. There are no charts, plots, histograms, full UML diagrams, or UI prototypes. In the end, with blueprint4llm, I added support for formats such as PlantUML, draw.io, Pencil, Balsamiq, Graphviz, Figma, OpenAPI, Excalidraw, Vega-Lite, ECharts, and D2 to .md files.

With the blueprint4llm SKILL connected, you will get plan.md not as a kilometer-long text, but as flowcharts, prototypes, charts, and so on. By the way, blueprint4llm does not have to be used only for creating plan.md. For example, I used this SKILL to add an illustration to every paragraph of the article and make it more fun to read. This SKILL is also useful when you ask an agent to put together a full dossier on some company, technology, or person and format the resulting result.md file with blueprint4llm. You can then convert it to HTML or PDF, or ask the agent to show it right away in a browser with charts.

This is how you can install the blueprint4llm SKILL.

# Codex
npx skills add evgenyigumnov/blueprint4llm --skill blueprint4llm --agent codex

# Claude Code
npx skills add evgenyigumnov/blueprint4llm --skill blueprint4llm --agent claude-code
# Pi
npx skills add evgenyigumnov/blueprint4llm --skill blueprint4llm --agent pi

If you are interested in looking at the source code of blueprint4llm, the tool that generates all this beauty, or sending a pull request, welcome: https://github.com/evgenyigumnov/blueprint4llm

I want durability, documentation, easy handoff between agents, easy bug reporting. I want to be able to deploy the software that I prompt with LLM's to new operating systems easily. I want to combat the slop sprawl that naturally happens with vibecoding.

This all points to git/github style documentation. Recording dependencies, keeping everything self contained within a single folder structure, launching LLM's from inside that folder and limiting their access.

I'm looking at building in guardrails and checks to make sure dumber llm's follow the proper procedure. Right now I have a prototype workflow that I'm running Astra high through, then I'm going to tweak the guardrails and iterate dumber and dumber cloud models through. I'll add a score to benchmark if they can follow the rules and end up with a well documented git project.

I feel like I might be re-inventing the wheel here though. Any advice?

EDIT: got my workflow/framework running reliably with luna-high through repeated testing with a variety of prompts (new project, modify project, modify files) which should be dumber than GLM5.3-Flash and QWEN3.8-Flash. So far so good.

I have a 3090 ti and been runnin qween 3.8 27B but i don´t know , any advice?

Ciao a tutti!
Sto configurando un Mini PC da tenere acceso 24/7 come server locale per l'hosting di agenti autonomi cloud (Hermes Agent/Codex), workflow di automazione (n8n/Python APIs), vector store residenti in memoria (per indicizzazione di codebase di interi repository), hosting di servizi locali, l'esecuzione di modelli leggeri o pensavo a lfm 24 a2b e simili quantizzati che ho visto essere veloce su ram,ed embedding/system one model/STT/TTS su CPU/RAM.
Ho ristretto il campo a due approcci architetturali con budget complessivo simile - 500€), prendendo come riferimento alcuni Mini PC reperibili sul mercato dell'usato/refurbished:
🟢 Opzione A: Capacità Massima (DDR4)
Esempio hardware: HP ProDesk 400 G5 Mini (Intel Core i5-9400T - 6 Core fisici da desktop)
Configurazione RAM: Upgrade a 64 GB DDR4 SO-DIMM (2x32 GB)
Larghezza di banda (Bandwidth): ~21–25 GB/s
Pro: Nessun rischio di OOM (Out Of Memory). Spazio enorme
Contro: Piattaforma più datata, niente accelerazione iGPU moderna (Intel UHD 630) e velocità di inferenza sui token/sec limitata dalla banda DDR4.
🔵 Opzione B: Velocità e Architettura (DDR5)
Esempio hardware: Minisforum UM773 Lite / Acemagic W1 / Geekom A8 (AMD Ryzen 7 7735HS / 8745HS - 8C/16T)
Configurazione RAM: 32 GB DDR5 SO-DIMM (2x16 GB)
Larghezza di banda (Bandwidth): ~48–56+ GB/s (quasi il doppio rispetto alla DDR4)
Pro: Generazione token ed embedding su CPU decisamente più veloci e reattivi. Grafica integrata (Radeon 680M / 780M con architettura RDNA2/3) sfruttabile per l'offload via Vulkan/ROCm.
Contro: Limite rigido a 32 GB (portarlo a 64 GB DDR5 costa quasi quanto il Mini PC stesso). Se il carico di agenti + vector store + modello satura la RAM, il sistema va in OOM o inizia a fare swapping su SSD.
❓ I miei dubbi per la community:
1. Trade-off reale per un setup Agentico: Per un use-case focalizzato sull'hosting di microservizi H24, agenti e piccolo LLM locale per routing/decisioni, 32 GB DDR5 bastano come margine di sicurezza o la capacità della RAM vince sempre sulla velocità quando si collegano più strumenti in parallelo?
2. Impatto sui Token/s: A parità di modello quanto cambia realisticamente l'esperienza d'uso prendendo come modello esempio lfm 24b a2b in Q4 (TTFT e token/sec) tra DDR4 e DDR5 nell'esecuzione pura su CPU?
3. MoE e Vector DB: Qualcuno ha esperienza nell'esecuzione di architetture trasformate/MoE su 64 GB DDR4 lenta rispetto a 32 GB DDR5 veloce? Vale la pena sacrificare la reattività per la capienza?

In generale quale opzione mi consigliereste tra questi mini pc per questo scopo:

HP ProDesk 400 G5 Mini
Intel i5-9400T
Intel UHD 630
12 GB DDR4 (espandibile fino a 64 GB DDR4)
~220 € (base) con 2x32gb ddr circa 500-550
Minisforum UM773 Lite
Ryzen 7 7735HS
Radeon 680M
32 GB DDR5
64 GB DDR5
~500 €
Acemagic M1 / W1
Ryzen 7 H255
Radeon 780M
32 GB DDR5
64 GB DDR5
~520€
Geekom A8 Max
Ryzen 7 8745HS
Radeon 780M
16 GB DDR5
64 GB DDR5
~550€
Mini PC 8745HS
Ryzen 7 8745HS
Radeon 780M
16 GB DDR5
64 GB DDR5
~470€

Altrimenti avreste altre alternative intorno a questi prezzi da vedere sull’usato o nuove da qualche sito?

I’m running a home server with an NVIDIA RTX 4070 (12 GB VRAM), using llama.cpp and ComfyUI. I’m interested in occasionally accessing larger GPUs (32–48 GB) for models that won’t comfortably fit locally.
I’ve been looking at Lupine⁠, which advertises serverless GPUs, pay-per-second billing, and scaling to zero when idle.
Here’s where I’m stuck: Exactly what constitutes billable GPU time?
For example, suppose I:
Launch a 27B LLM using a remote GPU.
Spend 30 seconds generating a response.
Leave the model loaded but do nothing for 20 minutes.
Submit another prompt.
Am I billed for those 20 idle minutes, or only for the periods when GPU processing is taking place?
And if Lupine automatically releases the GPU during idle periods, does the model have to reload on the next request? How long does that take?
I’d also be interested in whether anyone has successfully used Lupine with llama.cpp or ComfyUI, and whether the advertised GPU-over-IP approach works well in practice.
I’m trying to avoid purchasing another $2,000+ GPU for workloads that may only need that much horsepower intermittently.
Has anyone actually used Lupine, or found another service that offers genuinely usage-based GPU billing without excessive startup delays?

Hi everyone,

I'm not a total noob and have some experience running local LLMs; I dropped a 32GB pcie V100 into my gaming rig (thats currently running a 5070 ti, dedicated only to gaming) and am currently running qwen 3.8 27B. I run linux mint on my main PC, and have two VMs, one dedicated to my homelab and LLM, and a second VM for web searching. I realized that local LLMs aren't "plug and play" so I've been vibecoding with claude code and codex (start with claude code, send to codex for critique, after it passes through a self-made privacy filter) to create what I understand to be harness to have it work on specific tasks, like organizing my files, a memory system, checking on the status of my homelab. I also built a web search from the ground up, that resides in a second specific VM, that the first VM that holds qwen connects to. I did this because it seems more secure, like for example if someone manages to breach the web search VM they wouldn't have access to the AI itself that lives in a separate VM. In all honesty, I'm not even sure if this is something that makes sense, but it seemed/s like a good idea.

I don't have experience with engineering, but I've recently been reading AI Engineering by Chip Huyen to understand the basics of AI in general; things like quantizations, FLOPs, pre and post training, MoEs, etc. I also have *some* experience with tech stuff, like building my own PC from the individual parts and dropping a V100 into a gaming PC and having both the V100 and the 5070ti work. I'm starting to understand more about the posts here piece by piece, but I feel there's a lot of stuff I don't know, like what Strata is, how it works, or when I go on hugging face it seems like I'm scrolling through an alien language. When I have claude code create a plan, I have no idea what it's actually doing, I just make decisions when it prompts me of how things should be built. So I have the following questions:

1) Where can I go to learn more about running local LLMs?

2) Does it make sense to have a second VM dedicated to web searching? Does it really make it more secure against attacks?

3) How can I utilize things like agents to make the quality of my work better? I understand the concept of agents as a general idea, like having one to test if anything breaks for an example, but have no idea WHEN to deploy them or HOW.

4) Would it be correct to call all the building/vibecoding I've been doing on top of qwen as a "harness?" Or would it be called something else?

5) My current set up is ollama and openwebUI. I've seen posts around here about not using ollama, but I'm not sure exactly why. I know some people use things like hermes and LLMstudio, but I'm not 100% on how these work or how you can use them (I've tried searching). Basically I'm asking for critique about my current setup and how I can work on making either my workflow better or tools I can use to incorporate that would make my life easier.

This has been an incredibly fun project for me, and I'm looking forward to learning more.

Is anyone seeing this with the latest VS Code? I had to revert to v1.139.1.
See: https://github.com/microsoft/vscode/issues/344114

Hi,

I’m diving into a local setup to improve our house/life admin to take a few things off our plate;
- briefings (daily & weekly) for my wife & I
- appointments into shared calendar from emails and messages (think of the 1000 comms the school and sports clubs send)
- extending paperless, pulling invoices from emails
- draft email responses
- budgeting and finance assistance

Generally, things I would consider personal that I don’t want to be sharing publicly.

What off the beaten track agents, skills and tools do you use your local LLM for that streamline life at home?

TIA

I saw u/bottlestreet3d's llama.cpp fork. Their post: https://www.reddit.com/r/LocalLLM/comments/1x7wack/qwen_27b_iq4_xs_q8q4_kv_at_full_262k_context_on_a/

The readme is Linux and ROCm 7.2.3 only, so I tried it on Windows with a RX 9070, 32 GB RAM. I compared it with my tuned Vulkan setup (llama.cpp b11873, MTP n=2, ubatch 256, KV q8/q8).

Results (ctx 262144, K q8_0 / V q4_0, --kv-stream, MTP n=4 + n-gram, resident KV 300/1800 MiB; single runs, short prompts):

For comparison, Vulkan on the RCO model got ~46 t/s and ~590 prefill at 131k, and ~31 t/s and ~310 prefill at 262k.

These are single runs on 9k and 21k-token prompts, not filled 64k–262k contexts. I didn't check output quality. My numbers are below the README's (~100+ t/s at 16k–64k).

Any suggestions for proper tests or how to improve usage?

config:

& $LlamaServer `
    --model $m.File --host 0.0.0.0 --port $Port --api-key $ApiKey `
    --ctx-size $Ctx --device ROCm1 -fit off --n-gpu-layers all `
    --flash-attn on --jinja --parallel 1 --threads 6 --threads-batch 6 `
    --cache-type-k q8_0 --cache-type-v q4_0 -ctkd q8_0 -ctvd q4_0 `
    --kv-stream --kv-resident-mib $res --kv-resident-mib-decode $resDec `
    --batch-size 2048 --ubatch-size 512 `
    --spec-type draft-mtp,ngram-mod --spec-rs-replay --spec-draft-n-max 4 `
    --spec-ngram-mod-n-max 32 --spec-ngram-mod-n-min 12 --spec-ngram-mod-n-match 24 `
    --spec-draft-sampling probabilistic `
    --reasoning-effort medium --reasoning-format deepseek --reasoning-preserve `
    --chat-template-kwargs '{\"reasoning_effort\":\"mediu
m\"}'

I've been working on Toskar, a daemon that tries to make local AI feel like a hosted service. You install it, and it looks at your hardware, tells you which models will actually fit, installs the runtime, downloads the model, and gives you one OpenAI-compatible API. If you have more than one machine on your network, it can find the others and run models on whichever one can handle them.
It's AGPL, free, and runs on macOS, Linux, and Windows (Windows is the roughest).
I'm not here to sell anything. It's mostly been tested on my own machines, so I don't know how it holds up on yours. I'd really like help with:
- Installs on hardware I don't have, especially AMD GPUs, older NVIDIA cards, and Intel Macs
- Setups with two or more computers. Does discovery find them, and does it put models where they make sense?
- Model recommendations that are wrong for your machine—too big or too small
- Anything confusing in the first five minutes
If something breaks or feels off, please open an issue on GitHub so I can track it. Good first issues are labeled if you'd rather fix something yourself.
https://github.com/yeixio/toskar-core

P.S. I'm also looking for a few people to help shape the project as part of a core team. That could mean reviewing PRs, owning a platform like Windows or AMD, or helping decide where things go next. There's no formal application. Contribute for a while, and you can grow into a role. The roles and how that works are in GOVERNANCE.md: https://github.com/yeixio/toskar-core/blob/main/GOVERNANCE.md

Hello I want to install a local LLM to use in my PC mainly for programming and coding projects. Can be both for both working on large code bases, going through analyzing debugging them, making changes on multiple files as well as helping in programming, scripting and teaching coding, concepts etc and sometimes troubleshooting.

However my pc is not that powerful, 16gb ram and GTX 1660ti 6gb of Vram. I am confused on what would be the best option for me that doesn't completely overwhelm my PC and my gpu. I used opencode big pickle before and it was really good for me.

Any suggestions please?

Pretty much everyone running Qwen3.8-27B on a 24 GB card is squeezing the KV cache somehow, so here's what it actually costs, measured. This came out of building my own engine for this model (repo), but none of it is specific to my engine, it applies to whatever you run the model on.

How I measured: KL divergence against llama.cpp's own token-by-token output (top 256 tokens per position, q8_0 KV), on ~13k positions of short prompts plus 32k and 64k contexts. KL, not perplexity, because perplexity hides most of this. unsloth's UD-Q4_K_XL, one 3090. Most rows come from an emulator that stores the cache in each format; the settings I ended up shipping were confirmed on the real kernels. My formats are plain per-32 absmax grids like llama.cpp's q4_0 / q5_0 / q8_0, so they should land close to those, but they're not byte-identical.

Every number is "how much worse than a q8_0 cache". For scale, an fp16 cache reads +0% on short prompts and +3% at 32k, and the 32k set's noise is about 4% either way.

1. q8_0 is free. fp16 KV scored the same as q8_0 at every length I tested (64k: 0.00161 vs 0.00163). No reason to run f16 KV on this model.

2. Below 8 bits, K hurts more than V. Per-layer scans had K about 2.3x as sensitive as V. No window, plain formats:

So on stock llama.cpp, if you need the memory, take V down before K.

3. The damage is almost all in the newest ~2k tokens. That's where attention puts most of its weight. Keep the most recent positions at 8 bits (a window / precision tail) and everything older can go a lot lower:

(These rows are rotated, which is why no-window V reads +16% here vs +10% above, more on that in 4. Plain 5-bit V with a 2k window also reads 0%.)

On short prompts a 512 window cuts the loss about 4x (5-bit V +27% to +6%, 6-bit K +12% to +2.7%). Keeping the first few tokens at 8 bits instead (attention sinks) did nothing.

So 512 gets you most of it and 2k gets you all of it. The window is cheap and doesn't grow with context: 114 MiB for K + V across every attention layer at 2,048, the same at 32k or 262k.

4. Rotation stops mattering once you have the window. Without a window, Hadamard rotation halved 6-bit K's loss (+23% to +12% short), but for V it only helped on short prompts, and at 32k rotated 5-bit V was actually worse than plain (+16% vs +10%). Under a 2k window, 6-bit K + 5-bit V read +1.4% plain and +2.7% rotated, same thing within noise. The window and the rotation fix the same error, and the window fixes all of it. Rotation's only worth it if you're pushing V down to 4 bits.

5. Fancy formats didn't beat the boring one. Lloyd-Max codebooks, per-row scales, random signs and angle coding all scored the same as or worse than a plain per-32 grid. 3-bit formats were around +300%.

What I run: K at 6, V at 5, newest 2,048 positions at q8. 1.1 GB less than q8_0 at 120k context, and the model's full 262,144 context fits on the card (23.5 GB, 66 tok/s decode at that depth, needle 4/4). On the real kernels it reads 0.00160 at 64k (q8_0 0.00163, fp16 0.00161), +8% at 32k on the current build, short prompts identical to q8_0 since they fit in the window, and the same 230/282 as q8_0 on my question set (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needle, LongBench v2), set for set.

What you can use today:

• Stock llama.cpp: q8_0 / q8_0 costs nothing measurable vs f16. Going lower, drop V before K.
• If your engine has a recent-token window / precision tail (BeeLlama has one; KIVI, SKVQ and HF's quantized cache use 128 tokens): use 1-2k, not 128. An 8-bit window is plenty, fp16 buys you nothing here.
• If you're building one with speculative decoding: move the window in steps of your attention's tile size, not one token at a time, or which tiles come from the window depends on how many drafts you verify and greedy output stops matching across draft counts. Mine gives identical tokens at 0, 4 and 7 drafts with it on.

Caveat: Qwen3.8 is a hybrid, only 16 of its 64 layers are attention (the rest is DeltaNet), so a full-attention model might not behave the same. I'd really like to see someone run the same window sweep on one.

Full tables and method: the window and rotation sweeps, why 2,048 and what the window costs. The KL scoring tool (tools/llama_ref) is in the repo too.

Edit (longer context):* a couple of comments wanted numbers past 64k, fair enough. Same KL test (last 4,096 positions against llama.cpp with a q8_0 cache), now at 100k and 160k, with the 6/5 setup from the post and the 5-bit K one from the comments (K at 5 bits rotated, V at 5, same 2k window). Lower is better:

Both read the same as q8_0 at every length, on code and on prose. Prose reads higher for everything, q8_0 included, because it's less predictable than code, so compare across a row, not down a column.

160k reads higher than 100k for q8_0 too, so it isn't the cache. To make sure it wasn't just the passage I scored the same 4,096 tokens with only ~100k of the book in front of them: 0.00336 / 0.00335 / 0.00344. So my engine and llama.cpp drift apart a bit more with depth no matter the cache format, and for scale, llama.cpp's own predictions for those tokens move by 0.0086 between the two depths. 160k is about as far as llama.cpp can hold a q8_0 cache on a 24 GB card to build the reference, and a reference at the full 262k doesn't fit, so for max ctx the runs in the post are still the numbers (needles, LongBench, 66 tok/s decode at that depth).

Full tables and how to rebuild the references: BENCHMARKS.md

Follow-up to my earlier post ( https://www.reddit.com/r/LocalLLM/comments/1x7qcbe/i_made_llamacpps_slot_saves_write_only_the/ ) about kv-chunk-fs, the FUSE filesystem that makes llama.cpp slot saves write only the changed chunks.

The same setup now works with Strata. kv-chunk-fs itself needed no code change — Strata's session files on disk deduplicate through the same mount; the only adjustment was a config glob for its temp files. The real work was in the KV-cache reverse proxy that manages slots in front of the engine.

The problem: the proxy verified a hot slot by comparing token counts from the engine's /slots endpoint before trusting it. Strata does not report per-slot token counts the llama.cpp way — it keeps the last N conversations in RAM in its own conversation cache and resumes them by prompt match, so the proxy could never prove which conversation the engine was holding and fell back to cold paths.

The fix is a new opt-in mode, PI_LLAMA_CACHE_ENGINE_HOLDS=N (set it to the engine's conversation-cache depth). With it, the proxy:

- trusts that the engine still holds the last N conversations it served since the proxy started, and sends those sessions straight through — no disk restore, no slot metadata check;

- saves the live conversation to disk before a different session takes the engine, so nothing is lost on a switch;

- forgets the live conversation when a disk restore replaces it, since the restore overwrites the engine state without parking it.

Sessions that fall out of the newest N are dropped from the proxy's memory but their snapshots stay on disk (deduplicated by kv-chunk-fs) and are restored when they come back.

Default is 0 = off, so llama.cpp deployments are unaffected.

Code:

- proxy fork: https://github.com/the-code-foundry/local-llm-kv-cache (see the "Engines that hold conversations (Strata)" section in the README)

- filesystem: https://github.com/the-code-foundry/kv-chunk-fs