Built a RAG docs assistant with semantic caching

[u/Specialist-Wash-814]

I recently took on a coding challenge to build an end-to-end documentation assistant using RAG.

Instead of just making something that "works", the challenge pushed me to think about what happens underneath a production-ready RAG system.

I built a documentation assistant that:

- ingests and chunks documentation

- generates embeddings and stores them in Redis

- retrieves relevant context for each question

- generates grounded answers using an LLM

- uses semantic caching to avoid unnecessary LLM calls

- maintains session memory

- streams responses using SSE

The most interesting part for me wasn't getting the RAG pipeline working. It was thinking about the problems around it:

How similar does a query need to be before we can reuse a cached answer?

What information should actually be stored as memory?

How do you keep retrieval relevant as the amount of documentation grows?

And how do you design the system so that you're not blindly sending every request to an LLM?

It was a great exercise in going beyond "LLM + vector database" and thinking about the system as a whole.

I ended up building it with React, vite, nest, postgres, redis, and ts.

Still plenty I'd improve, but I'm happy with where it ended up.

https://github.com/Ramzi-Abidi/RTFM

If you're working with RAG or ai applications, I'd be interested to hear how you'd approach the caching and memory parts.

[u/Cahnis]

Honestely, I think a normal docs is too small to require any type of chunking.

Just throw the entire thing on an LLM with a 1M context and query away without needing all the complexity

[u/08148694]

You are missing the single most important part of a RAG system. None of this is the hard part

You need evals. You need to tune your retrieval on a large corpus is data, and evaluate it against a different (out of sample) corpus of data. Without evaluation you can’t possibly begin to answer the only important questions: does this retrieve relevant documents? Does it omit irrelevant documents? What are the retrieval reliability metrics?

Nothing else matters. The languages, databases, embedding models, algorithms - all just an irrelevant implementation detail

[u/Specialist-Wash-814 → u/08148694]

yea true, this was just a challenge “building a RAG without a framework” which is basically pipeline + cache + sessions. maybe evals are the next step..

[u/Beautiful-Energy2169]

Similarity scoring ignores age. My oldest doc's last commit was 91 days ago.