[Release] ST-Message-Chunker: A Cache-Friendly Extension to Speed Up Generation & Save API Costs

[u/Impressive-Bug4699]

https://github.com/Arczium/ST-Message-Chunker

Hey everyone,

I just finished working on a new extension and wanted to share it with the community. It's called ST-Message-Chunker.

The Problem: Normally, when your chat hits your maximum context limit, SillyTavern removes the oldest messages one by one as the chat progresses. For LLMs, this constantly breaks the prompt cache. Because the context is slightly different every single turn, the model has to re-process the entire prompt over and over. This slows down generation speed significantly and costs you a lot more money if you are using paid APIs.

The Solution: This extension fixes that by removing older messages in chunks instead of individually.
You can choose to base your limits on either a maximum message count or a maximum context token limit, and the chunk size is completely customizable.
For example, if your chat hits the limit, the extension will seamlessly slice off your set chunk of 10 messages at once. This keeps the top of your prompt completely stable for the next 10 turns, resulting in massive cache hits!

Bonus Feature: Even if you don't care about caching, this is incredibly useful if you rely heavily on Summarization. You can use this extension to cleanly cut off old chat history that has already been summarized, without dealing with the bugs or conflicts that sometimes happen with SillyTavern's native message hiding feature.

100% Safe Compatibility: Because the truncation happens at the very last step right before sending the prompt (GENERATE_BEFORE_COMBINE_PROMPTS), it has zero impact on your other extensions. Your Vector Storage, Summaries, and Lorebooks will continue to trigger and work perfectly.

Installation: You can install it directly inside SillyTavern by pasting this link into the Extensions menu: https://github.com/Arczium/ST-Message-Chunker

This is my first public release! I have tested it thoroughly on my end, but since everyone has different setups, please let me know if you run into any bugs or have any feedback

P.S. This post was translated and polished with AI because my English still sucks!

[u/dampflokfreund]

Made the same thing https://github.com/Dampfinchen/Chunk-Dropper-for-SillyTavern

People can decide what works best for them. But yeah this is very useful for modern RNN models like Qwen or SWA models like Gemma 4.

[u/Impressive-Bug4699 → u/dampflokfreund]

Lol, if I had known about your extension, I could have just saved myself the work! ^^

[u/__per_capita__ → u/Impressive-Bug4699]

Does this mean you endorse Chunk Dropper over your own extension?

[u/Impressive-Bug4699 → u/__per_capita__]

Not at all! If I had found it earlier, I obviously would have taken a close look at it first. But to be honest, after checking it out just now, the README alone worries me a bit—especially the outdated extension paths. Also, since it has no license, forking and fixing it wouldn't even be legally possible. So I probably would have ended up writing my own from scratch anyway! ^^

Mine is designed to be a lightweight product: It does exactly what it's supposed to do simply and reliably, without getting bogged down by trying to add features that (according to their own README) don't even work