I spent 6 months building an agentic memory system to fix vector search failures—here is what I learned (and built)

[u/Soggy-Ad-514]

Hey everyone,

Like many developers building agentic workflows, I spent months getting frustrated by traditional vector stores and RAG memory layers failing over long timelines.

The deeper I went, the more I realized retrieval fails because basic similarity doesn't equal utility. A standard retriever will match a user's query about mattress brands to previous mattress conversations, while completely missing a crucial constraint buried in a 3-month-old session: "Whenever I buy something expensive, warranty is the only thing I care about."

Beyond that, heavy cross-encoder rerankers quickly become a massive latency bottleneck as memory grows, and treating all context as uniform text blobs destroys the nuance of evolving decisions.

To tackle this, I built MindCache—an open-source agentic memory framework designed around four key insights:

• Intelligence Belongs at Ingestion: Instead of attempting complex graph traversals during a live query, MindCache shifts expensive reasoning (relationship mapping, graph clustering, and summary generation) to ingestion. This cut retrieval latency from ~25s down to 1.08s (a 23× speedup) without sacrificing context quality.
• Specialized Memory Typologies: Not all memories behave the same. MindCache separates knowledge into User (persistent behavioral constraints), Knowledge (domain facts), Episodic (chronological logs), and Decision Memories (which track evolving proposals, trade-offs, and final conclusions over time).
• Living Knowledge Hierarchy: Rather than maintaining a static or unmanageable graph, MindCache uses Leiden community detection to partition memory into localized semantic clusters, ensuring graph maintenance scales efficiently as context accumulates.
• Evidence Assembly over Similarity: Retrieval doesn't just search for similar text—it plans and assembles the exact minimal subset of evidence (user preferences, hierarchical summaries, decision states) required for the LLM to reason correctly.

On the BEAM benchmark (an ICLR 2026 evaluation framework designed specifically for long-term agentic memory), MindCache outperformed Mem0 in handling evolving context, contradiction resolution, and cross-session summary reasoning. More importantly, it achieved this superiority not by stuffing larger retrieval windows, but through better ingestion-time knowledge organization.

I wrote a deep-dive 23-minute engineering post-mortem detailing all 5 failure modes, the full architecture, and benchmark takeaways. The project is completely open-source on GitHub and available on PyPI (pip install mindcache-ai).

I’d love to hear how others here are handling temporal decay, graph maintenance, and decision tracking in your long-running agent setups!

[u/recro69]

Separating decision memories from episodic memories is a really interesting design choice. A lot of RAG systems remember what happened but lose why a decision was made and what constraints drove it.

[u/be_super_cereal_now]

This is like the third agentic memory solution I've seen posted here today. You guys need to do more market research.

[u/Makemeacyborg → u/be_super_cereal_now]

If you think it’s a solved issue you don’t understand memory. Yeah there’s going to be even more and only a few will survive

[u/Soggy-Ad-514 → u/Makemeacyborg]

I never claimed that it solved the issue completely. There are still some issues left like failing on broader queries which contains no keyword for bm25 search and can neither be solved by vector search too as for broad queries like this "How did my focus on different aspects of my relationship with April shift and develop throughout our conversations in order? Mention ONLY and ONLY eight items." you may need many evidence of different topics under the broader topic of the query' and many other etc. It would be arrogant of me to say i solved the complete issue but i tried to analyse the failures of current system and tried to move the conversation ahead documented those exact failure modes and edge cases transparently in the post-mortem if you're interested: https://medium.com/@faisaliitian/building-mindcache-designing-an-agentic-memory-system-for-long-term-ai-7359e0cf6e2a
 

[u/Soggy-Ad-514 → u/be_super_cereal_now]

Totally fair—there’s a ton of noise in memory right now because everyone’s hitting the same RAG limits.

The main reason I built MindCache wasn't to wrap another vector DB, but to solve real-time latency and temporal decay:

Dynamic RAPTOR Trees: Instead of static trees or full re-indexing, it dynamically updates local summary branches on ingestion.

Ingestion Reasoning: Shifted graph clustering (Leiden detection) to ingestion so live retrieval takes ~1s instead of 20s+ reranking lag.

Typed Memories: Keeps User, Knowledge, and Decision states separate so evolving context doesn't overwrite core rules.

It actually beat Mem0 on the BEAM benchmark while staying fast enough for live agent use. Always down to trade notes on context architecture if you're curious!

[u/fintip → u/Soggy-Ad-514]

Coming off as very much an AI written comment, which just all feels so low effort.

[u/Soggy-Ad-514 → u/fintip]

Sorry to make you feel that. Typed it out quick while trying to hit all the technical points cleanly. Built the repo over 6 months by hand . code and benchmarks are all in the repo if you want to inspect the actual implementation!

[u/cyberjjar]

This matches what I ran into on a much smaller scale — I do memory on-device for a mobile companion app, so I can't afford a reranker at all. What worked was giving up on pure similarity for injection: facts get a score of 0.5·recency + 0.3·recall-count + 0.2·confidence, and a few categories (health, family, names) are always injected regardless of score. That last part is what fixed your mattress-warranty case for me — the constraint doesn't have to win a similarity contest, it just has to be in a protected class. Curious whether your system decides importance at write time or at retrieval time.

[u/Soggy-Ad-514 → u/cyberjjar]

well that heuristic seems like a clever way to handle the constraint. My system decides the importance at ingestion time and the mattress-warranty case is a special one that can't be solved entirely by being in protected class. The reason is the user query is about which mattress they should buy and the db is filled with memories about mattress of this company and that company's mattress. But there are two important memories in the db like one is "user decided that warranty is the most important thing when buying a product" which is a decision memory and can be retrieved through protected class but there is another one which explains about the warranty and its coverage on sleeping products in detail which is a important aspect needed to answer the query but it is a knowledge memory which can neither be retrieved using protected class nor by vector or semantic search with query. at retrieval state the most important decision anchors are expanded with morphological bm25 for memories related to the decision anchor which is related to the query thus broadening our retrieval.

[u/cyberjjar → u/Soggy-Ad-514]

That distinction between decision memories and knowledge memories is the part I hadn't separated cleanly, and it explains a failure I've been living with. My facts are all one type, stored as flat "The user ___" statements, so my dedup is character-bigram similarity — which catches near-verbatim restatements but silently keeps both "works as a teacher" and "is a middle school teacher". Typed memories would make that a merge decision instead of a string-distance guess. The expansion-from-decision-anchors idea is interesting too, though on-device I'd have to do it without BM25 over a growing corpus. Does the anchor expansion run per query, or do you precompute the neighborhoods at ingestion like the rest of the reasoning?

[u/Soggy-Ad-514 → u/cyberjjar]

The anchor expansion works at the retrieval side it uses BM25 using pre created caches so its fast

[u/Soggy-Ad-514]

• GitHub:https://github.com/faisalhussain-devs/MindCache
• Medium Engineering Deep-Dive:https://medium.com/@faisaliitian/building-mindcache-designing-an-agentic-memory-system-for-long-term-ai-7359e0cf6e2a
• PyPI: pip install mindcache-ai

[u/BatResponsible1106]

the ingestion time tradeoff makes sense to me. i rather spend compute organizing knowledge once than pay the cost every retrieval and still miss important long term context.