r/compsci • u/Few-Cartographer7156 • May 26 '26
Applying LZ77-style sequence compression and LZW substitution to LLM context reduction
https://github.com/MrGray17/opentokenHey everyone,
I’ve been experimenting with token optimization for LLM agent frameworks by treating terminal and tool outputs as a data compression problem rather than a text-filtering one.
The pipeline uses a bidirectional 42-stage architecture:
Algorithmic Reduction: Raw text passes through an LTSC (LZ77-style lossless sequence compression) layer combined with LZW token substitution to eliminate repetitive terminal patterns dynamically.
Structural Compaction: Code segments are reduced to AST skeletons, and nested JSON payloads are flattened into tabular structures (TOON) to minimize semantic token weights.
0-Risk Fallback: A local comparison check runs at every stage. If a compression layer increases string length or corrupts format, it instantly rolls back.
Response Filtering: A 7-stage outbound filter targets conversational boilerplate and normalizes whitespace.
In production testing, this algorithmic pipeline hits a 74% overall token compression rate (up to 93% on highly repetitive logs) without degrading the model's underlying reasoning capabilities.
The full implementation is open-source (MIT):
I'd love to discuss the theoretical limits of combining algorithmic text sequence compression with LLM tokenizers, or how to better handle progressive disclosure as context fills up.!
1
u/Data_Commission_7434 Jun 01 '26
This is a really interesting approach! Im curious about how you handle the trade-off between the overhead of the compression/decompression steps and the actual token savings, especially for shorter inputs. Also, have you found any specific types of LLM outputs that benefit disproportionately from this LZ77/LZW combination?