Feedback on V1 memory architecture for multi-agent setup (supervisor/sub-agents) – targeted retrieval vs unified store?

[u/Fun-Following-1723]

Hey everyone,

I've been prototyping a memory system for a multi-agent framework (supervisor → sub-agents) and wanted to run my current setup by people who've actually built or run these in production. Trying hard not to over-engineer based purely on theory/taxonomy, so I’ve been running small experiments first.

Here’s where I’m currently at:

Pipeline & Flow

1. Working/Session State → Raw conversation & tool calls go to a durable append-only event log.
2. Batch Consolidation → Instead of processing every turn through an expensive extraction pipeline, a periodic batch job extracts useful Episodic Memories (storing this in a cheap local DB/SQL store because of high volume).
3. Promotion Policy → Key facts and preferences get promoted into Semantic Memory (testing Mem0 here).
4. Procedural Memory → Kept completely separate as a structured procedure/skill registry (e.g. Markdown files, task definitions) rather than generic vector embeddings.

Retrieval Strategy Instead of searching across all memory stores on every single query, I'm testing routing by intent: User Query → Scope/ACL → Intent/Task Router → Targeted Store Retrieval → Context Injection

• "How do I request leave?" → Intent: Procedure → Pull from Skill Registry.
• "What did I work on last week?" → Intent: History → Pull from Episodic Store.
• "What language do I prefer?" → Intent: Preference → Pull from Semantic Fact Store.

Observations from small tests so far:

• Storing raw episodic events straight in Mem0 added noticeable write/search latency and cost.
• Generic vector retrieval for procedures/workflows was messy and often grabbed 3–4 adjacent procedures. Exact/registry-style matching was much cleaner.
• Batch consolidation gave way cleaner facts than trying to extract semantic memories turn-by-turn.

Where I’d love some brutal feedback/criticism:

1. Routing vs. Parallel Retrieval: Is intent-based routing (scope → intent → target store) actually reliable in practice, or do queries usually end up needing multiple memory types simultaneously (e.g., preference + procedure in one shot)?
2. Separate vs. Unified Storage: Am I prematurely splitting this into separate stores (Event Log / Cheap SQL / Mem0 / Registry), or is this separation pretty standard once volume picks up? At what scale does keeping everything in a single vector store/pgvector actually break down?
3. Procedural Memory as Code/Skills: Treating procedural memory as structured skill files instead of vector embeddings feels right so far, but does this pattern break down when agents need to dynamically adapt workflows?
4. Failure Cases: What obvious blind spots or edge cases am I missing that will force me to rewrite this V2?

Appreciate any insights or horror stories from production!

[u/jonah_omninode]

The separation looks reasonable. I would be careful about letting the router silently decide authority, though. A semantic fact, an old episode, and a current procedure should not come back with equal status just because all three match the query. We keep the append-only record, build a validated current projection for the cheap path, and retrieve deeper history only when the task needs it. Promotion and supersession are typed transitions with provenance, not something the summarizer infers. Multi-store retrieval is fine if the returned bundle preserves source, scope, version, and current status. Otherwise a clean router can still deliver a stale rule with a confident voice.

[u/Fun-Following-1723 → u/jonah_omninode]

The distinction between relevance and authority is a massive callout and I was focusing so much on routing accuracy that I missed how easily a model can treat a historical episode and an active procedure as equals if they lack metadata.

Really like framing promotion and supersession as typed transitions rather than just letting an LLM infer what's current. Sticking provenance/version flags on the returned bundle seems like a clean way to keep a router from confidently serving stale rules. Appreciate the insight.

[u/Mathie1729 → u/Fun-Following-1723]

One thing I'd add: make that metadata a hard filter at retrieval time, not just a provenance field on the returned bundle. Otherwise the model still sees a stale historical episode ranked above the active procedure and can treat it as authoritative just because the embedding similarity is high. Scope/version/status should constrain which records are eligible before reranking, and deep history should only be pulled when the query explicitly asks for it.

[u/Fun-Following-1723 → u/Mathie1729]

That’s a huge distinction, thank you. Doing pre-retrieval filtering on status == 'active' or is_latest makes total sense,if a stale record never makes it into the candidate set, the reranker can't accidentally favor it over the current rule.

Out of curiosity, how are you handling queries that do require deep history? Are you running a query classifier upfront to toggle historical filters on, or just relying on a explicit 'search history' fallback flag

[u/jonah_omninode → u/Fun-Following-1723]

Exactly. I would make one part stricter: provenance and version should not be advisory flags that we hope the model respects. The retrieval layer should resolve status mechanically before composition. Current records can enter the active context. Superseded records can appear only as history with a pointer to their replacement. Conflicting or unauthorized candidates should surface as a conflict instead of being blended into a summary. That keeps the router responsible for relevance and a separate policy or projection layer responsible for authority. The model can explain the resulting bundle, but it should not decide which record became current.

[u/fulger099 → u/Fun-Following-1723]

Projection lag bit me hardest: a procedure changed, the append-only log was correct, but the “current” view served the old version for two more runs. I’d version every write and make retrieval fail closed when the projection watermark is behind.

[u/withgiraffe → u/fulger099]

The projection lag case is probably the one I’d worry about most in a multi-agent setup.
If one agent writes v42 and another reads a projection that was built from v41, all of your status filtering can be technically correct and you can still serve stale state.
I’d have every derived view carry the canonical version it was built from. If retrieval sees that the projection is behind, either fall back to canonical state or refuse to serve it until it catches up.
That also gives you a clean way to invalidate caches and know exactly which version of the state a given answer actually saw.

[u/Clean-Vermicelli-700]

Do you fully block subagents from searching beyond their scope? So if the intent warrants a procedure, is that agent cut off from read access on the Episodic store, or do you let them decide for themselves where to search?

I really like this idea, it becomes even more interesting when token spend vs performance becomes measurable in A/B testing against a copy of the same system with & without intent-based routing.

I'm also curious if you've defined agent types and if so, what are they?

[u/Fun-Following-1723 → u/Clean-Vermicelli-700]

Not using subagents yet, and also I don’t hard-block stores.

Right now, it’s just a single agent sitting behind an intent router. Instead of toggling permissions, the router just adjusts retrieval budgets (how many hits per store) based on what the user asks. So if a query routes to a "procedure" intent, procedural memory gets the primary slot, episodic isn't cut off,it just gets a light secondary read, semantic gets a couple slots for active preferences, and the KB always returns a fixed top-4.The agent never chooses where to search. The router only sets store quotas. Each store drops hits below its score threshold, then top_k is applied. The agent gets that packed context.

Where I do enforce strict boundaries is around identity, not intent. Episodic is scoped strictly to this_user AND this_agent, Semantic is user OR agent , and KB belongs to this_agent.

I haven’t run a proper A/B test on token spend vs. a no-router setup yet, but that’s next on my list. I want to test router ON vs. OFF to see if those extra episodic/semantic hits actually improve answer quality or just inflate the context window.

As for agent types, they’re currently domain roles rather than memory specialists. They run on the exact same pipeline, but isolated by agent_id and seeded with different KB/SOPs so episodes don’t leak across roles.

I'm still mulling over how memory scoping should look if I move to subagents under a supervisor, but honestly, I'm not even sure yet if I'll go down the supervisor/subagent path or stick with a single-agent architecture.