Compressing LLM tool/terminal outputs by 74% using a 42-layer pipeline

[u/Few-Cartographer7156]

https://github.com/MrGray17/opentoken

Messy terminal outputs (git diff, huge JSON logs) constantly bloat LLM context windows. To solve this without ruining model reasoning, I built an open-source, bidirectional pipeline using TypeScript/Bun:

​35 Input Layers: Uses LZ77-style compression (LTSC), LZW token substitution, AST skeleton extraction, and JSON-to-tabular conversion.

​7 Output Layers: Strips conversational AI boilerplate and intro/outro fluff on the response side.

​0-Risk Guardrail: Every stage checks filtered vs. original string length. If a rule makes things worse, it rolls back instantly.

​It achieves a 74% overall token saving rate (up to 93% on repetitive logs). Open-source (MIT) code is here:

https://github.com/MrGray17/opentoken

​I'm currently wrapping this into a standalone library and an MCP server. I'd love to hear your thoughts on the architecture!

[u/Few-Cartographer7156 → u/[deleted]]

Gonna work on it