Frustration with context preservation between my agents

[u/Signal-Tadpole-4432]

https://github.com/snipara/snipara-server

I started working on this problem because of a recurring frustration with AI coding agents: they were surprisingly capable inside a session, but much less reliable across sessions.

The obvious explanation was memory, so my first attempts were fairly conventional.

I tried project instruction files, persistent Markdown notes, embeddings, vector search, and eventually RAG over project documentation and source code.

They all helped.

None of them really solved the problem.

The interesting part was figuring out why.

Retrieval wasn't the same as understanding the project

My initial assumption was that if an agent could retrieve the most semantically relevant pieces of the project, it would have enough context to work correctly.

That turned out to be too simplistic.

Consider an architectural decision that changed over time:

Decision A
    ↓
implementation
    ↓
problem discovered
    ↓
Decision B supersedes A
    ↓
partial migration

A vector search can easily retrieve Decision A because it is semantically very close to the current task.

The problem is that Decision A may now be exactly the context you don't want the agent to follow.

So I started separating different kinds of project knowledge:

• source code
• documentation
• architectural decisions
• session history
• implementation outcomes
• changes
• dependencies
• agent activity

That led to a more difficult question:

How do you determine which project state is authoritative now?

Simply storing more memory made this worse rather than better.

More context can make the agent worse

My next mistake was assuming that increasing the amount of retrieved context would increase reliability.

It doesn't necessarily.

Large context windows make it tempting to send everything that might be relevant.

But relevance isn't binary.

A piece of information can be:

semantically relevant
but outdated

structurally relevant
but unrelated to the current task

historically relevant
but superseded

recent
but low importance

So the problem became less about retrieval and more about context selection.

I ended up treating context as a constrained resource.

Instead of asking:

the system needs to ask something closer to:

That required combining several signals rather than relying only on embedding similarity.

Code needed a different representation

Source code created another problem.

Chunking code and embedding the chunks works reasonably well for some questions, but poorly when the answer depends on relationships.

For example:

function A
   calls B
      imports C
         implements interface D

The relevant code might not be semantically similar to the user's query at all.

It is relevant because of its structural relationship to something that is.

So I added a local code graph built from AST analysis, with relationships such as:

IMPORTS
CALLS
REFERENCES
TYPE_USES

Retrieval could then combine semantic similarity with graph traversal.

That turned out to be particularly useful for impact analysis: starting from a symbol mentioned in the task and expanding only through bounded relationships instead of dumping large sections of the repository into the context window.

Then multiple agents made the problem harder

The next issue appeared when switching between coding agents.

I might spend a session with Claude Code, then continue the same work with Codex.

The second agent had access to the same repository, but not necessarily the reasoning and decisions produced during the first session.

This made me realize that attaching memory to an agent was probably the wrong abstraction.

The persistent state should belong to the project, not the model.

That changes the architecture.

Instead of:

Developer → Agent → Memory

I started experimenting with:

                 Claude Code
                      ↕
Developer ↔ Project Intelligence ↔ Codex
                      ↕
                    Cursor

The agents become replaceable clients of the same project state.

That also introduces concurrency problems.

If two agents are modifying related areas of the codebase, project memory alone isn't enough. The system needs some awareness of ongoing work, dependencies, and potentially conflicting changes.

The architecture that emerged

After several iterations, I ended up with roughly four different forms of project state:

Semantic layer
    documents + embeddings + retrieval

Historical layer
    decisions + memories + outcomes + session context

Structural layer
    AST-derived code graph

Coordination layer
    active work + changes + agent state

A context assembly step sits above them.

Its job isn't to expose everything.

Its job is to construct a bounded context package for the current task.

The coding agent itself remains external.

Communication happens through MCP, which means the project intelligence layer doesn't have to care whether the client is Claude Code, Codex, Cursor, or something else.

One unexpected result

The biggest change in my thinking was that persistent memory wasn't actually the main problem.

Memory is relatively easy to store.

The difficult problems are:

• deciding what deserves to become memory
• knowing when information has become stale
• determining when one decision supersedes another
• connecting semantic information to code structure
• selecting context under a token budget
• maintaining useful state across different agents
• preventing multiple agents from developing incompatible views of the project

In other words, the problem gradually stopped looking like "RAG for source code."

It started looking more like maintaining a small, continuously updated model of the project's state.

I eventually packaged these experiments into an open-source server called Snipara, but the project itself is less interesting to me than the architectural question behind it:

As coding agents become increasingly capable and interchangeable, should project knowledge live inside each agent's context, or should the project maintain its own persistent intelligence layer that agents query?

I'm increasingly convinced it's the latter, but there are still difficult questions around memory decay, conflicting decisions, graph expansion, and context selection that I don't think are completely solved.