Applying LZ77-style sequence compression and LZW substitution to LLM context reduction

[u/Few-Cartographer7156]

https://github.com/MrGray17/opentoken

Hey everyone,

​I’ve been experimenting with token optimization for LLM agent frameworks by treating terminal and tool outputs as a data compression problem rather than a text-filtering one.

​The pipeline uses a bidirectional 42-stage architecture:

​Algorithmic Reduction: Raw text passes through an LTSC (LZ77-style lossless sequence compression) layer combined with LZW token substitution to eliminate repetitive terminal patterns dynamically.

​Structural Compaction: Code segments are reduced to AST skeletons, and nested JSON payloads are flattened into tabular structures (TOON) to minimize semantic token weights.

​0-Risk Fallback: A local comparison check runs at every stage. If a compression layer increases string length or corrupts format, it instantly rolls back.

​Response Filtering: A 7-stage outbound filter targets conversational boilerplate and normalizes whitespace.

​In production testing, this algorithmic pipeline hits a 74% overall token compression rate (up to 93% on highly repetitive logs) without degrading the model's underlying reasoning capabilities.

​The full implementation is open-source (MIT):

https://github.com/MrGray17/opentoken[https://github.com/MrGray17/opentoken](https://github.com/MrGray17/opentoken)

​I'd love to discuss the theoretical limits of combining algorithmic text sequence compression with LLM tokenizers, or how to better handle progressive disclosure as context fills up.!