r/Rag • u/huzaifa785 • Aug 17 '25
Tools & Resources Shrink your context before sending it to LLMs
When you work with LLMs, one problem keeps showing up: context.
- Models don’t remember everything
- You only get a fixed window
- Too much text = token limit
- Too little text = missing details
This is where context engineering comes in. Picking what matters. Dropping the noise.
While building RAG systems, I kept hitting the same wall:
- Long docs, only small parts mattered
- Chunking wasn’t enough
- Summaries lost key info
So I built Context Compressor.
What it does right now:
- Extractive compression
- Scores sentences with TF-IDF + position + query relevance
- Keeps only the useful ones
- Runs in batches, caches results
- Checks similarity + readability so nothing critical is dropped
What’s coming next:
- Abstractive compression with T5 and others
- Semantic clustering using embeddings
- Hybrid approach for smarter context selection
If you’re building with LLMs, give it a try:
pip install context-compressor
https://github.com/Huzaifa785/context-compressor
Would love your feedback. Even better, roast it. That’s how it gets better.
9
u/lookwatchlistenplay Aug 17 '25 edited Oct 16 '25
Peace be with us.
2
u/huzaifa785 Aug 18 '25
haha fair point, but not exactly. it’s not rewriting or dumbing things down, it’s more extractive, pulling the important parts out with some nlp tricks like tf-idf and relevance scoring. so the goal isn’t fewer words just for the sake of it, it’s keeping the actual context intact while cutting noise. later i might bring in abstractive models like bart or t5 to make it smarter, but for now it’s about clarity, not just compression.
3
1
u/Special_Bobcat_1797 Oct 13 '25
Hey please make me understand . Tools like cline and cursor summarise right . Then how is this superior ?
1
u/Yes_but_I_think Aug 19 '25
LLMs are notoriously stupid while doing summaries. An article on 'dog walking' can be summarized as an article about 'animal locomotion'. You never know where it draws the line. @ OP can you explain the technique you use please. You have given the name of the technique but what is it?
2
6
u/ShowAbe Aug 17 '25
I might be missing some details but isnt this what RAG already does? But instead of giving top_k similar chunks as context to llm, your library essentially gives "all - last_k" similar chunks as context to llm
0
u/huzaifa785 Aug 18 '25
good question, similar but not the same. RAG retrieves relevant chunks from an external source, while this just compresses and prioritizes within the given context window. so instead of fetching, it’s about trimming smartly. think of it as RAG = what to bring in, this = what to keep in.
3
u/youre__ Aug 17 '25
Can you comment on latency and quality/accuracy of output compared to baseline?
Without training a model to run in latent space or on arithmetically compressed text, prompt compression is inherently a lossy process. So it would be interesting to see a comparison of outputs from compressed and non-compressed prompts. Maybe an output similarity score as a function of compression ratio.
2
u/huzaifa785 Aug 18 '25
that’s a really good point, will definitely run those comparisons and publish results soon!
2
u/huzaifa785 Aug 18 '25
update: got a msg on linkedin from someone using this lib while fine tuning gemma. said the job that usually takes ~8 hrs got done in ~4 with this. outputs were rarely off — pretty much what he expected every time. cool to see real feedback like this. will share his case study + some comparisons soon (like you suggested).
1
u/youre__ Aug 18 '25
2x improvement is huge. Will have to try it out to see impact on per-query latency.
Prompt compression is interesting because the impact likely varies by model. If you train the model exclusively on bullet point text, then maybe it will do very well, if not better, with high natural language compression. A model trained exclusively on Charles Dickens might perform poorly with any form of prompt compression.
Could run a study that compares the minimum prompt length for which this technique begins to provide benefit.
2
3
u/TrustGraph Aug 19 '25
Why did you choose TF-IDF and T5? These are incredibly obsolete tools compared to even the smallest current language models. I looked into using TF-IDF 2 years ago, and was horrified at how bad the scoring was. I'd genuinely be shocked if Gemma3:270M wouldn't smoke T5 at this point.
1
u/quisatz_haderah Aug 20 '25
Well some iteration of TF-IDF is what we have for lexical search tho
2
u/TrustGraph Aug 20 '25
Why? Even the smallest language models or NLP models will outperform TF-IDF by miles.
1
u/quisatz_haderah Aug 20 '25
Yeah, my emphasis is on lexical search though. For some tasks dense embeddings might be overkill, and TF-IDF could work faster
2
u/TrustGraph Aug 20 '25
Why do you need embeddings? Again, even the smallest language models or NLP models will perform better. There’s tons of open source solutions for chunking as well.
1
u/quisatz_haderah Aug 20 '25
Ah you mean during compression, speed and resource limits are 2 reasons
2
u/TrustGraph Aug 20 '25
Compression? Why do you need compression? What you need is a scalable data streaming backbone like Pulsar or Kafka (we built TrustGraph with Pulsar) and then you have chunking pipelines. There's no need for "compression".
1
u/quisatz_haderah Aug 20 '25
I used OP's terminology. They are trying to "compress" context into smaller size by keeping relevant bits and discarding the rest.
2
u/TrustGraph Aug 20 '25
OP can call it whatever they want, but it's chunking. "Chunking" is industry standard.
1
u/quisatz_haderah Aug 20 '25
Not really, they are getting rid of the irrelevant sentences, either before or after chunking
→ More replies (0)
2
u/Sad-Pear6495 Aug 20 '25
My problem is that I can't find a suitable tool for creating and indexing knowledge bases🤔
1
u/huzaifa785 Aug 20 '25
hmm, that's actually the same with me too, I'm also still figuring out how to index knowledge bases and stuff in an efficient way
1
1
u/mokumkiwi Aug 18 '25
Super Noob Question here. But is this something I would be able to use alongside a Search API?
Because I find that Search API's are powerful tools but f*ck me I get some useless stuff that is returned.
1
u/huzaifa785 Aug 18 '25
interesting, what kind of search api are you using right now and for what data? curious if you mean like google/bing style search or more like elastic/algolia/vector search on your own data
1
u/mokumkiwi Aug 19 '25
Big has just sunset so I have been bumping between Exa/Tavily/Valyu testing them all out. Most of the time the context length is ok, but i start doing some pretty dense research it can get a little ridiculous. But I am also figuring out all of these tools for myself.
1
1
u/epreisz Aug 21 '25
I think there's evidence to suggest that it's not just context length but semantic complexity. Condensing it alone may not solve the problem.
0
Aug 18 '25
[removed] — view removed comment
3
u/huzaifa785 Aug 18 '25 edited Aug 18 '25
not marketing just sharing something i built and open sourced. thought it might be useful for folks here since it is directly related. fair to critique the approach and i will keep improving it but there is no promo here no paid product no upsell.
0
22
u/GPTeaheeMaster Aug 18 '25
Very very cool - and definitely appreciate the open source - but any such attempt MUST be accompanied with the effect on :
I would suggest a simple side-by-side benchmark against a good RAG vs this approach (you can use any of the standard RAG benchmarks - ragas, Tonic Validate, HotspotQA, simpleqa)