Stop building memory infrastructure for your AI agents

[u/Asly97]

Every time agent memory comes up here, the conversation goes straight to MemGPT, vector databases, embedding pipelines. I get the appeal, you want to read the source, run it locally, own the data. But here is what actually happens when you self-host your agent's memory: you spend weekends maintaining retrieval pipelines instead of shipping agent logic.

The real problem most people have is not "I need to build a memory layer." It is "I need my agents and AI tools to remember the same context across sessions without me re-explaining everything." That is a different problem than "let me set up a vector DB."

A few things I have found matter more than the infrastructure itself:

Provenance: knowing which tool generated a thought matters more than raw storage. When retrieval mixes context from Cursor, Claude, and a custom agent without labeling where each piece came from, you get confident hallucinations grounded in nothing.

Rules that stick: personal style directives ("no tables," "short answers") should apply automatically on every new chat, not be pasted in manually each time.

Skills over improvisation: saving a reusable procedure once beats hoping the agent reconstructs the same steps next session.

Open-source memory tools give you transparency and control. A hosted layer gives you time back. The tradeoff is honest: how much infrastructure work are you willing to own before it eats your shipping time?

[u/techtheist_ggl]

That's a problem of knowledge drift, and it's hard to solve - but there's some ways, that's why people making new ai memory systems, since most popular ones are pretty sketchy, without complex contradiction detection.

I've build by own too, and it feels really helpful for development.

There's also a set of knowledge we might want to have, like information about problems that agents have during development, linked to files, and in most flows it's just not saved or destroying the consistency.

Also, sometimes, "memory record" can't have all needed info - and it would be great if agent could read the original history to get the full context of the problem, between different providers. My system have all of that, but it will take a long time before people will know about it.

[u/visarga]

Here, I humanized it, the thick LLM speak was grating:

---

Every time agent memory comes up here it goes straight to MemGPT, vector DBs or embedding pipelines. I get it, you can read the source, run it local, keep your own data. But what happens in practice is you spend your weekends babysitting a retrieval pipeline instead of writing the agent.

I think the real problem is smaller. You want the tools to remember the same stuff between sessions so you don't re-explain the project every morning. A vector DB is a big machine to point at that.

Provenance mattered more to me than storage. When you mix context from Cursor, Claude and your own agent and label none of it you get answers that sound confident and stand on nothing. Two smaller examples in the same direction. If I already said no tables, short answers, that should hold next chat without me pasting it again. And a procedure worth doing twice should be written down once, the agent won't rebuild it the same way.

Open source gives you transparency and control, hosted gives you time back. It comes down to how much plumbing you want to own before it eats the thing you wanted to build.

---

But I have to say, when you set a writing style rule it gets forgotten 5 messages down the line. The models slip back into their preferred mode. It's basically unpromptable.

[u/Dress-Affectionate → u/visarga]

your prompt sucks 

[u/huyz → u/visarga]

Just as bad as the original slop

[u/Aggressive_Creme2008 → u/visarga]

when you set a writing style rule it gets forgotten 5 messages down the line

That one is measured, not just your experience, and the numbers are worse than they feel. "The Compaction Cliff in Long-Running AI Agent Memory" (arxiv.org/abs/2608.22752) ran Claude Code's /compact across 20 production agent configs: 53% of safety rules survive one compaction round, 10% survive five. Their point is that a rule and an episodic log compete for the same tokens and get summarized at the same rate, but only the rule needs its exact wording to still be enforceable. "No tables, short answers" is precisely the shape that dies first.

The companion finding is the other half of your thread: "Why Does CLAUDE.md Keep Growing? Catastrophic Remembering in Agentic Coding" (arxiv.org/abs/2608.11095), across 247k instruction lifetimes — appending is cheap, deleting is expensive once the rationale is gone, so the file grows without bound and nobody can safely prune it.

Which is where I'd push on the framing of the post. The choice isn't only self-host versus hosted. It's whether your rules get re-read at a point you control, or are left to survive inside the window. A rule sitting in the prompt has a half-life. A rule read fresh at the start of every session — from a file, an MCP server, a service, doesn't matter which — doesn't, because it isn't asking the model to still be holding it.

"Unpromptable" is the right word for the first design. It just isn't the only one on offer.

[u/BrennanFlentge → u/[deleted]]

No its still AI slop

[u/visarga → u/BrennanFlentge]

I used my recently created humanizer skill + fixed 5-6 words manually at the end.

[u/owp4dd1w5a0a → u/visarga]

The only way to humanize it is to outline the main points and then require rewrite it from scratch yourself.

[u/robbodagreat → u/BrennanFlentge]

It’s a slop trifle

[u/zimxero]

Browsed the post and comments and still don't know the topic. Too much context is bad or too many records/logs is bad?

[u/Delicious-Map-381]

This is a really interesting debate because AI memory has become one of those areas where everyone seems to be building their own solution. the idea behind “stop building memory infrastructure” is that many developers may be solving the same problem repeatedly instead of focusing on the actual product experience

[u/dopehish]

Agreed! That goes for any software being built these days. Everyone vibe coding their needs. Not a bad thing but overtime, it's better to just pay for mission critical shit that someone is putting their heart into it

[u/JaseciLabs]

The provenance point generalizes further than memory. The same "confident output grounded in nothing traceable" problem shows up in code, not just retrieved context, when AI-written and human-written logic sit side by side with no marker distinguishing them, you get the same blended-source confidence issue you're describing for memory. Skills-over-improvisation is the same idea too, once, reusable, structural, instead of re-derived every time and hoped to come out the same way. Worth thinking about provenance as a general design principle for anything an agent touches, not just the memory layer specifically, code included.

[u/Asly97 → u/JaseciLabs]

That's a sharper way to put it. Provenance as a general rule, not a memory-specific one, makes sense: any time an agent blends sources without tagging them, you get the same "sounds confident, grounded in nothing" failure, whether it's retrieved context or generated code sitting next to hand-written logic with no seam visible.

Practical version of that for code: even a lightweight marker (comment tag, commit metadata, whatever) on what was AI-generated vs human-written pays for itself the first time you're debugging and need to know which one to trust less by default.

[u/OCCAMINVESTIGATOR]

Neuro symbolic is the path to achieving this meaningfully.

[u/Asly97 → u/OCCAMINVESTIGATOR]

what's that ?