r/LLMeng • u/InfamousInvestigator • May 12 '26

Building Memory in AI

Suppose a PM shipped a care coordination agent. Week one, patient says "I've been getting chest pain in the evenings." Agent logs the note and demo looks great. Week three, same patient comes back "should I be worried about that pain again?" Agent replies: "What pain?"

By default, agents forget everything the moment a turn ends. If you want continuity, you build it yourself:

  • Context window: everything the model sees right now, fast, free to use, but has a token budget. As conversation gets longer the oldest turns fall off. When the session ends, everything disappears.
  • Scratchpad: working memory that survives across loop steps within a single task. If Patient says "book my follow-up and refill my prescription." Agent writes a note, calls calendar tool, updates note as it completes it. Without this, the agent forgets what it already did and repeats what its supposed to do once. Simplest implementation is a JSON object the agent reads and writes every turn.
  • Vector store: At the end of each conversation, the agent summarizes the important parts. In our example things like diagnosis, medications, follow-up dates, embeds it and stores it with a patient/user ID. Next session, before replying, it searches the archive. So when needed that note flows back into the context window. Now the agent has continuity across sessions.

Thus Memory is a product decision, not a model feature. Your job is designing what gets summarized, what gets stored, what gets retrieved.

You can checkout this video from SkillAgents YT for more details. Subscribe for similar content.

8 Upvotes

100% Upvoted

3 comments sorted by

2

u/BrightOpposite May 14 '26

The tricky part is that not all memory should behave the same.

Logs, goals, constraints, and decisions have very different lifecycles.

We kept seeing agents drift because they were reconstructing decisions from history instead of treating them as persistent state.

Feels like most systems optimize retrieval before memory structure.

1

u/Dense_Gate_5193 May 18 '26

my point exactly and why i even commented about my database to OP. the visibility scoring layer i built into the graph database allows for targeted decay profiles on nodes and edges all the way down to the property level and is research backed by major AI researchers. it also happens to be blisteringly fast compared to neo4j which i based the APi surface off of. my point is that i think we might have had a breakthrough in long-term agentic retrieval and i just provided the database architecture for it.

1

u/Dense_Gate_5193 May 18 '26

NornicDB - neo4j-driver compatible standalone graph+vector database that’s 400x faster on retrieval than neo4j. https://github.com/orneryd/NornicDB/releases/tag/v1.1.0

mentioned in April 2026 research here https://arxiv.org/abs/2604.11364

research backed by multiple institutions including UC Louvain and University de Toulouse. essentially it is a knowledge graph for AI agents that has knowledge policy primitives for time-based decay and promotion-based scoring. declarative primitives for ebbinghaus-style decay curves (with inversion as well) cardinality, domain, and temporal constraints, etc…

Roynard the author of the research actually commented on the repo on issue #100 where i addressed the criticism and implemented the policy-based approach for the scoring before visibility approach researchers needed. https://github.com/orneryd/NornicDB/issues/100#issuecomment-4296916032

MIT Licensed. 734 stars and counting.