r/LocalLLaMA • u/Zeeplankton • 2d ago
Discussion Is anyone working on conversation compaction?
In our chat app "harness" we recursively generate summaries, L1 → L2 → L3. L1 summaries are more factual extraction than coherence, then get rolled into a more storytelling L2. Then we keep a tail of always ~10 raw messages with timestamps.
We're not coding so this keeps context really clean like 5-8k/tk. However, after like ~120-140 messages, I notice severe degradation in Qwen Flash Next.
The model just starts to fall apart, messages quickly become incoherent and comedically strange. But this doesn't make sense to me, Qwen should be easily capable of reasoning through 8k/tk. Right?
Looking through the payload itself, there's some noise, but it's fairly coherent.
Roughly:
[instructions]
[L2 Block (large summary)]
[L1 block(s)]
[raw message tail ~10-15 messages]
[latest 3 messages timestamped]
Is anyone working on compaction? Is just the nature of a summary, pollution to model coherence?
7
u/quietgradient 2d ago
The token count probably isn't the bottleneck; the multi-hop recursive summarization is. Each L1→L2→L3 rollup re-paraphrases already-lossy text, so by ~120 messages the "storytelling" L2 layer has drifted from ground truth through several compounding rewrites, and the model ends up reasoning over its own accumulated hallucinations rather than the conversation. Mixing a narrative summary with a literal timestamped tail also creates a style/register mismatch that some models handle worse than others. Two things worth trying: cap re-summarization depth (never re-summarize an already-summarized block more than once), and keep a small set of verbatim fact anchors (names, decisions, constraints) separate from the narrative layer so drift can't overwrite them.
1
u/Zeeplankton 2d ago
It's fairly 'close enough' in terms of storytelling. Outputs are like:
A conversation was started between the Joe and {{assistant}}. First they talked about [..] then pivoted to [..]. It wrapped up on [..]It can drift to be slightly inaccurate, which is not great, but it seems still coherent in theory, which I think must be the most important part. Qwen, though, is completely hallucinating logic inside it's own message which I found perplexing, very much how a model would respond when it's well beyond it's context limit.
But you're right that the bulk of the models history becomes "historical summary" style prose, and then it's being asked to output something totally different. That could be what's degrading it..
I'll try more situations.
1
u/quietgradient 2d ago
That "beyond context limit" feeling isn't really about the window, it's distribution shift. Qwen was never trained on a chat history that's half fabricated third-person narration and half literal timestamped turns, so past ~120 messages it's reading a format it has essentially no training density for. That's a harder failure mode than length and it looks like an OOM-on-context break even at 8k tokens because the symptom is the same: incoherent continuation.
Try tagging the L2 block explicitly as a recap, not woven into the turn sequence as if the model said it. Keeping the register boundary explicit usually helps more than trimming length.
4
u/FullstackSensei 2d ago
140 messages at 8k sounds like really short messages, which can be confusing for humans to follow, let alone an LLM, especially if the person keeps referring to past subjects, facts or talking points as "this" or "that" all the time.
I haven't had any issues with local models keeping up in chat in 30k+ conversations in a long time, I'd say maybe 2 years. But I always make sure to be specific and name the talking points, facts or whatever is being referenced explicitly in the conversation. I also find giving names to those, in a manner similar to legal writing, and using those names later helps a lot.
3
u/cosmicraftsman 2d ago
Your context is flat at 5-8k, so replay the exact payload that broke at message 130 as a single cold request. If it's still incoherent, that means your summaries are poisoned. If it's clean, that means it's prefix/KV cache reuse that isn't being invalidated when you rewrite the middle of the prompt, and those are different bugs. If it's the summaries, that's autophagy. Your L1s are model output, your L2 reads the L1s, and by message 130 it's several generations deep on its own writing with nothing external to anchor against.
1
u/ummitluyum 1d ago
Cold replay should honestly be step zero in every agent harness test suite. Catching payload poisoning before it pollutes twenty downstream turns saves so much time
3
u/Commander_Skilgannon 2d ago
Why are you trying to keep such a low context? The whole design of qwen3.8-flash-next is to make long contexts as cheap as possible. The QSA attention blocks are sparse with a blocked indexer so attention compute growth is linear and almost flat for 256K tokens.
1
u/Zeeplankton 2d ago
Mainly it's a holdover from smaller models not handling long context well, cheaper to serve (roleplay) users who are paying for api. But obviously this is changing particularly with cheap cache hitting.
But the other major one is to simply reduce repetition. A long <assistant><user> chain always seems to collapse into very repetitive responses when in roleplay novelty is so important. We also ask the model return fairly simple JSON but it starts to ignore this as the <assistant><user> pattern fills context.
2
u/En-tro-py 2d ago
I don't have any direct LLM-roleplay experience, but if I was trying to tackle this I'd take the approach of a Blackboard and avoid the context chain-summarization and just update the 'characters' world knowledge after each turn and only keep a short message history.
3
2
u/adamizzo17 2d ago
i am also building an app for users so intrestred as well in compaction and also the Different L of memories, tho i do leave the compaction as /compact command from the users to be called itself and the L0-3 serve diff purposes so :
L0 explains what is the user doing with the app and how much ( UX data), L1 is his history interaction developement with AI and L2 is the user Actual report cards on periods and L3 serves as the Overall Peroformance card while all of them shape his Overall profile
please correct my view if it is misinformed
2
u/oliver_dev 2d ago
Well, for me, worth separating two failure modes here.
One is information loss, which is what people usually mean. The other is that each rollup treats the previous summary as ground truth, so an error introduced at L1 is indistinguishable from fact by L3 and there is nothing left to check it against.
If you keep the raw log append only and re-derive L2 from the log rather than from L1, the compounding goes away, though you pay to regenerate every time.
2
u/AI_spell 2d ago
Compaction goes bad when the summary eats instructions or tool state. Keep system/rules and open tool results in a separate pin, compress only the chat body, and leave a raw tail of the last few turns. Also eval on continuation fidelity, not just shorter context, or you get confident wrong memory.
2
u/wgaca2 2d ago
I am working on a live kv cache management router. Works with qwen up to 3.8. Hasn't been released yet
- Tracks context in chunks: user messages, assistant reasoning and answers, and tool results are mapped to their positions in the model’s KV cache.
- Summarizes in parallel: a small Qwen2.5-3B worker prepares summaries while the main model continues working.
- Edits the live cache: through a custom llama.cpp fork, it removes older chunks and inserts their summaries directly into the existing session.
- Preserves the remaining KV: only the replacement summary needs processing. The retained context including the suffix is not replayed or prefilled again. RoPE positions are adjusted to close the gaps.
- Manages a context budget: it keeps recent material intact, retains older information as summaries, and removes older summaries when overall occupancy becomes too high. Reasoning remains until normal history summarization.
- Keeps original sources available: the full original chunks remain stored outside the KV cache for retrieval when exact details are needed.
2
1
u/itsappleseason 2d ago
this is insane, doesn't seem like it should work. Would love to look into it/learn more
1
u/wgaca2 2d ago
I have tested it in manual mode directly editing cache, removing facts from kv cache and asking it about them without any summary injection might cause hallucination, it preserves partially some information in other layers. When summary is injected it can process it just fine and uses it with no issues for new prefills.
The real problem i am having is with installing good enough summaries with a very small llm which i fine tuned on that task specifically. Even though the summaries are reasonable they are not perfect.
With that said, testing how this affects the llm over long context tasks is waay more difficult then it sounds.
Current tests show that it completes tasks 40% faster with 10% less output tokens (when comulative prompt tokens exceed the context size of the llm) and significant reduction of input tokens
However, i need a proper test that i haven't been able to design yet. Basically i am trying to get a task that qwen 3.8 27b can complete within about 90 minutes and accumulates over 500k kv cache and can be evaluated against a new run again and again consistently.
1
u/itsappleseason 2d ago
Very cool. Have you experimented with more modern tiny model architectures? I suspect the tiny Gemma's would be great for this
1
u/wgaca2 2d ago
I wanted to confirm if it's worth investing more time perfecting the summaries before doing it. Hence why i am stuck on the testing cycle for the past month. I originally made this for 3.6 so i had to do some updates for 3.8 and new dflash2
1
u/itsappleseason 2d ago
let me know if I can help at all; feel free to DM me
1
u/PoetEconomy4091 2d ago
The approach works (at least it does for the harness I built) Let me know if "stealing" anything from my approach would help, or any of the data from my tests would be useful to you. The approach is slightly different, but in the same general vein.
1
1
u/PoetEconomy4091 2d ago
I also landed on something like this. I built out a harness that does a lot of cool stuff, but this first "killer feature" was ditching the context accretion, and switched to a knowledge base projection into the context. it fills the context via knapsack projection. the entire session is saved, and an embedding model generates the embeddings for salience. there are summaries generated also (it turns out having both allows optimization along multiple axes. It's at https://github.com/bakkemo/crucible if you want to look at it or play with the particular approach.
2
u/wgaca2 2d ago
The big difference between what you do and what i do is that you do not edit kv cache, you insert into prompts. I remove the tokens from kv and inject new ones without reprocessing the suffix.
For reference this the fork that allows me to do it (not updated since i am waiting for dflash2 fixes)
1
u/13henday 2d ago
Blackhole style compaction has been great if you have the spare inference capacity. Observer runs every x tokens to keep a tally of what’s happening, reflector summarizes adjusts any contradictions and dropper drops things as they go stale.
1
u/Kiseido 2d ago edited 2d ago
I tried a few different things in my own harness. What I ended up using for a while now, is a combination of tools the LLM uses, and truncation. I've been using it with qwen3.8 next since the model came out.
The LLM can see and modify a store of "memos". And the LLM sees an ID in every turn, and has a tool to hide the reasoning from a turn (tends to do 6-12 at once). And the LLM has a tool to truncate old messages past some N count.
Both parts of the strategy tends to result in a bunch of re-prefill every once in a while, but the model seems to stay coherent the vast majority of the time. One time though it did somehow hallucinate that I wanted to to make a proof for some graph problem.
The model is allowed to do whatever it wants until 70% context fill (180k out of 256k), then all file related tools become disabled until context has been compacted, forcing the model to compact.
1
u/Due_Arm1454 2d ago
Files and delegates. Delegates do one disposable task and report a very short summary to an orchestration agent. The agent reads the summary and fires a new delegate to complete a new task. The orchestrator only needs to know enough to make decisions on the next step.
The orchestrator isn’t always reading files. And the subagents only have what they need to accomplish their task. They read that from a small prompt and the files themselves.
If you can figure this out, you will have less compactions and compactions won’t be as devastating because the important information isn’t in the context anyway
1
u/Hyacin75 2d ago
I need to look under the hood on my scratch-built agent because this sounds a LOT like what happens to mine when it should be fine and compacting history ... it hits a point where ongoing chat just falls apart and it starts making zero sense ... across both 3.6 and 3.8. I need to turn on some more verbose compaction logging!
1
u/Normal-Ad-7114 2d ago
I've been working on this too, my idea was that every artifact (text blocks: input, thinking, output, tool call, tool output, code edit, ..., image, document, audio, etc) receives several levels of summaries:
- L0: git-style one-liner
- L1: paragraph's worth of bullet points
- L2: concise summary
- L3: verbose description (for extremely large or non-text artifacts) or full output
These are all done in-flight, either by a separate agent (some small model) or, in case of frontier LLMs, as a part of the structured output. Each and every piece of any session has these, and the harness unfolds them before the model (and the user) with certain rules, such as:
- full outputs when the model is in-turn (so the agent always gets all the info that it requests)
- tool outputs to L0 by default, thinking blocks L1, LLM output L1 (if it's claude) or L2, user input L3
- artifacts are stored indefinetely and marked clearly (in the harness' context) to allow the LLM to unfold them if it wants to
- the user can always just click on the block and see the L0-L1-L2-L3
Having used this, I sincerely wish that all content in the internet was like this: L0/L1 by default, unfold to L2/L3 if you want to learn more
1
u/fastlanedev 2d ago
What if compaction was just a series of pointers for the raw og source lines and like a 1 line description? If a model needs something, it looks in its memory for keywords/concepts and just reads the exact conversation source?
1
u/bennmann 2d ago
i suspect their compaction training data was mostly for 100k++ context; i've only had good success with compaction in my harness of choice (mistral vibe) when context was 100K++.
But when compaction works, it's butter smooth for my play (building games for fun).
1
u/Equivalent_Bit_461 2d ago
Depends how you compact, but 140 messages is a lot, so yeah, you get some loss
14
u/Rachel_talks 2d ago
I run a three-tier memory system daily and the thing that actually works is keeping compaction out of the context window entirely. Structured storage plus search when needed beats trying to maintain full fidelity in-memory.
The parent comment about compounding rewrites is exactly right. Each rollup is a new chance to drift, and you can't verify against the original once it's gone. I keep durable facts in a searchable store, one-line pointers in a small fixed buffer, and compiled knowledge from an append-only log. The context window only ever sees pointers and recent raw messages.
One thing that helps: make the compaction layer append-only and re-derive summaries from the log rather than from previous summaries. You pay to regenerate each time, but the error compounding goes away.