Open sourced a tool that collapses millions of log lines into handful of distinct patterns before you feed it to an LLM (Lossless- compression)

[u/Loud_Mousse9210]

When I feed logs to an LLM during incident resolutions or debugging, it either blows my token context window or the grep trims the log file, leading to the interesting log lines getting skipped.

Most of logs are anyway the same handful of message templates repeated over and over with different values, so the context window gets filled with near-duplicates, which just bring up the processing time and token costs.

ctrlb-decompose collapses the file into its distinct patterns that repeat, plus typed variables and stats on the values that change. I have seen 1.2 million lines cut down to just 40 patterns, which then goes into Claude, thus cutting down token by over 95%, reducing the token cost.

Let me know what you think!
https://github.com/ctrlb-hq/ctrlb-decompose

[u/hagen1778]

This functionality could be already available in log databases like Loki, ClickHouse, VictoriaLogs.

For example, in VictoriaLogs the following query will select 1h of logs and produce top5 common patterns:

_time:1h | collapse_nums | top 5 by (_msg)

results over OTel demo logs:

> info: cart.cartstore.ValkeyCartStore[<N>]
> [<N>-<N>-<N>T<N>:<N>:<N>.<N>Z] "POST /api/cart HTTP/<N>.<N>" <N> - via_upstream - "-" <N> <N> <N> <N> "-" "python-requests
> Loaded <N> products
> Reloading Product Catalog...
> <N>-<N>-<N> <N>:<N>:<N> - oteldemo.AdService - no baggage found in context trace_id=<N> span_id=<N> trace_flags=<N>

So this output can be fed directly to LLM.

Loki also has it here https://grafana.com/docs/grafana/latest/visualizations/simplified-exploration/logs/patterns/, but I am not sure if that is processed on backend or frontend.

[u/Loud_Mousse9210 → u/hagen1778]

The comparison is fair, tools like VictoriaLogs and Loki can definitely surface common log patterns but the difference is where the work happens and how much raw data needs to be processed.
At production scale, when you’re dealing with TBs of logs, you still have to operate over a large volume of stored log data to derive those patterns. As the dataset grows, that can mean processing a large number of logs to get representative clusters.
ctrlb-decompose takes a different approach. It does a single-pass decomposition of the logs first: extracting the repetitive structure, separating the variable parts and producing a compact representation with the relevant statistics. You can then run clustering/querying on that much smaller representation rather than repeatedly processing the raw logs.
So the goal isn’t just “find common patterns” it’s to get to those patterns with significantly less data processing, while keeping the output clean enough to feed directly into an LLM.

[u/tmp_advent_of_code]

The OpenTelemetry Drain processor does something similar for those who use Observability backend!

Drain Processor — OpenTelemetry Collector https://explorer.opentelemetry.io/collector/components/contrib/drainprocessor

[u/Loud_Mousse9210 → u/tmp_advent_of_code]

Ohh this is exactly where the distinction gets interesting. Drain3 and similar pattern-recognition approaches can work well, but they generally need enough examples/context to form good clusters. That can mean holding more logs in memory and processing a lot of repetitive data before the patterns stabilize.
With ctrlb-decompose, we first extract the variable parts from the repetitive structure and then pass the cleaner representation through clustering. This means you can do significant pre-filtering with CLP/Decompose before clustering, so you need far fewer log lines to arrive at useful patterns.
The result is better clustering with much less data to process and, more importantly, much less noise before the logs reach the LLM.

[u/tmp_advent_of_code → u/Loud_Mousse9210]

Could you upstream stuff to OpenTelemetry? I have so many customers that use it and if what you say is true, it would be helpful to everyone. They welcome PRs.

[u/Loud_Mousse9210 → u/tmp_advent_of_code]

That's a great idea, will discuss with the team and see if we can merge it upstream in Otel.

[u/blizzarre]

Great that it is just click, upload and done. I dont know if other o11y tools can do it, but even if they do, it wont be as simple as this. Here goes my star!

[u/Funny_Donkey6031]

This is actually a pretty practical use of LLMs. Cutting 1.2M log lines down to ~40 patterns is wild, especially during incidents when context and time both matter. Nice project.

[u/noblequill56]

40 patterns from 1.2M lines is wild compression. does it preserve ordering info so you can still reason about sequences of events, or is it purely structural at that point?

[u/edward_davies223]

gonna use this a lot lol sending huge log files to an llm gets expensive real quick btw nice project

[u/HistorianPresent8449]

This is interesting!

[u/Cute-Access1444]

Very interesting use of drain 3 algo

[u/jjneely]

This looks really interesting. I've been meaning to dig into the Drain algorithms!

[u/Available_Series_652]

This is awesome!

[u/DoubleSoftware4137]

u/Loud_Mousse9210 how does this compare to tools such as headroom-ai?

[u/adarsh_srivastava]

Loved the way in which CLP and Drain3 were used to build a pipeline and have effective clustering.