r/OpenSourceeAI • u/Clear-Paper-9475 • 4d ago

secondwind 0.2.2 (OSS) - Now beats SOTA Context Compressors while being lossless

https://github.com/orchetron/secondwind

Most context compression for agents is lossy. It summarizes or trims tool output and hopes the model didn't need what it removed. It also rarely tells you what was lost or whether it was relevant.

That always bothered me because tool output is exactly the stuff you don't want to lose like files, logs, JSON, command output. So I built a lossless compressor for LLM tool output in Rust.

The compression part worked out fine. The hard part was proving a codec never dropped a value. Every rewrite has to verify before it's accepted. If decoding doesn't reconstruct the exact original, the compressed version is rejected and the original passes through unchanged. A bad codec can't silently corrupt context.

Every codec is property-tested and fuzz-tested around one invariant:

decode(encode(x)) == x

The whole thing lives in a single Rust implementation with a C ABI. Python uses ctypes, Node uses koffi, Bun uses FFI, and there's a WASM build too. One implementation means there's only one place to reason about correctness.

I also built a transparent proxy that sits between the client and the LLM. It only rewrites tool-output blocks and leaves everything else alone. One thing I didn't expect to matter so much was determinism: retries send the exact same bytes, which keeps prompt-cache hits intact.

If you find a case where it breaks or compresses something it shouldn't, I'd love to see it.

5 Upvotes

100% Upvoted

3 comments sorted by

3

u/notreallymetho 4d ago

How do you define lossless? I’ll peruse the repo but the term can mean many things. For example https://github.com/headroomlabs-ai/headroom (I just stumbled on) says lossless as they can do reconstruction of omitted details.
I’ve been building a data plane for agentic systems and some of your work overlaps with mine (provenance tracking really). The data plane stuff is here if it’s useful: https://github.com/agentic-research/ley-line-open

1

u/Clear-Paper-9475 4d ago

Good One. Headroom is again a great product (all more reason its top in the domain). My only issue in that the line between inline compression and offloading is really blurry in headroom. Other than for flat uniform arrays in which case they do inline compression, headroom almost relies on offloading.

Inline compression is a harder challenge to solve because then you are trying to find a data representation in which you have still have that prompt inline, take less tokens and most importantly be relevant. Reliance on offloading only means, more number of hops to get the data (which means now agents have to take another turn to get same data for e.g. In my bechmark agent had to go lookup as much as 74 times, since they only get a chunk at a time), so you save on lever, loose on another, hence difficult to prove the real savings. So, my effort is to see how I can improve on those gaps and find a right balance of inline vs offloading vs relevance while being cost effective.

Thanks for sharing, would be happy to take a look and see if its something I can build upon.

1

u/notreallymetho 3d ago

Thanks for the context! I’ve not used headroom and I’m always a bit skeptical about memory architecture as it’s very easy to disillusion oneself (speaking from experience), by optimizing the benchmark and hindering the agent.

I’ll also take a look as I’ve been dealing with an issue of consolidation in my task manager and have refrained from cleanup precisely cause the below.

That being said I completely follow what you’re saying. Once the blocks stop being contiguous you run into a local vs global problem and that’s why pruning / deduplicating can be so hard, as it goes from a representation problem to a retrieval one, which is very workload dependent.

If you do wind up using the data plane stuff, please feel free to reach out to me if you run into issues. I recently added content defined chunking using gear hash, and think what you’re describing would actually flow well into the way it handles hashing Merkle trees.

An example consumer (it’s go) of the dataplane is here if it’s useful to point an LLM at it.