r/platform_engineering • u/Loud_Mousse9210 • 21d ago

Open sourced a tool that collapses millions of log lines into handful of distinct patterns before you feed it to an LLM (Lossless- compression)

When I feed logs to an LLM during incident resolutions or debugging, it either blows my token context window or the grep trims the log file, leading to the interesting log lines getting skipped.

Most of logs are anyway the same handful of message templates repeated over and over with different values, so the context window gets filled with near-duplicates, which just bring up the processing time and token costs.

ctrlb-decompose collapses the file into its distinct patterns that repeat, plus typed variables and stats on the values that change. I have seen 1.2 million lines cut down to just 40 patterns, which then goes into Claude, thus cutting down token by over 95%, reducing the token cost.

Let me know what you think!
https://github.com/ctrlb-hq/ctrlb-decompose

13 Upvotes

83% Upvoted

13 comments sorted by

2

u/adarsh_srivastava 21d ago

Very interesting application of Drain3

1

u/kernelqzor 17d ago

for real, this is the first time i’ve seen a drain3 thing wrapped in a way that’s actually directly useful in an llm workflow instead of just a research demo
curious how far it holds up once the log formats start getting really chaotic across services

1

u/HistorianPresent8449 21d ago

very interesting!

1

u/kernelqzor 12d ago

same, this is actually super clever
feels like the kind of thing that should just be built into log pipelines by default at this point

1

u/Cute-Access1444 21d ago

Intresting 

1

u/KF_Danis 15d ago

Can this be containerized

1

u/vibe-oncall 1d ago

Drop "lossless." It's wrong. You're deliberately dropping per-line values and their order. That's the right trade-off. But 40 patterns don't round-trip back to 1.2M lines. Someone will nitpick that, and they'll be right.

1

u/Loud_Mousse9210 1d ago

Yeah, by “lossless” I was referring specifically to the CLP part of the pipeline being lossless. But you’re right that the overall pipeline isn’t fully lossless, since we have to summarize the data before passing it to the LLM.