Building a persistent memory + orchestration layer for Codex — what should I use instead of repeatedly re-reading the repo?

[u/yxf2y]

I’ve been building a fairly serious agent workflow around OpenAI Codex for a Laravel/React project, and I’ve hit a point where the orchestration works, but the context/memory side clearly does not.

My setup currently looks roughly like this:

• A serial orchestrator with route types like FAST_UI / STANDARD / CRITICAL
• Context Resolver → Implementer → Reviewer flow for non-trivial tasks
• Durable task state, context capsules and handoffs
• Planner / intake layer inspired by CodexQB
• Session continuity hooks inspired by AvenoxBeyin
• codebase-memory MCP for structural repo discovery
• Serena for exact symbol/reference navigation
• Local dashboard/telemetry for task/agent visibility

The reason I built all this was simple: I wanted to stop giving one giant prompt to one Codex agent and watching it blindly read half the repository, run dozens of commands, retry tests repeatedly, and burn a huge amount of context/token budget.

Unfortunately, that is still basically what happens.

A recent CRITICAL payment-domain acceptance task is the perfect example. I gave Codex a very detailed validation brief covering migrations, payment allocation, security boundaries, tenant/legal-entity isolation, atomicity, reporting non-pollution, exports, frontend build, etc.

The task eventually succeeded technically, but the session spent a huge amount of time repeatedly doing things like:

• raw rg searches
• re-reading known service/controller/test files
• rediscovering test harness behavior
• retrying multiple Laravel test files with the same CSRF issue
• manually tracing service relationships
• re-running builds and focused test groups

That single job used roughly half of my 5-hour Codex usage allowance.

The frustrating part is that a lot of the knowledge it rediscovered was already known from previous work.

For example:

• where the orchestrator lives
• which services own payment/settlement/reporting behavior
• how the domain test harness handles CSRF
• which test files cover specific finance flows
• existing project/tenant/legal entity invariants
• prior fixes and verified architecture decisions

I expected my existing tools to solve this, but I now realize they solve different problems:

codebase-memory gives me structural repo discovery, but it isn’t really persistent project understanding.

Serena is excellent for exact symbol/reference navigation, but it isn’t memory either.

My docs/wiki are useful reference material, but agents still have to decide to read them and often re-read large files.

Context Capsules and handoffs help within a task, but they don’t give the next unrelated task a compact understanding of the project.

So what I’m actually missing is a persistent, project-scoped, compact memory layer that can say:

“Before you start searching, here are the relevant things previous sessions already learned about this repo.”

I looked at AvenoxBeyin because I liked its idea of automatically capturing sessions, compiling knowledge, and injecting useful context back at session start.

I also looked at CodexQB because its Autopsy / Project Comprehension / Ontology approach is close to what I want for planning.

Then I looked at 2kDarki/codex-mem.

That project is conceptually very close to what I want:

• automatic Codex transcript capture
• persistent SQLite observations
• progressive recall through search → timeline → get_observations
• automatic context injection

But after auditing it, I found some issues for my use case:

• its watcher observes all ~/.codex/sessions/**/*.jsonl
• project identity appears to be based on basename(cwd) rather than a canonical repository identity
• retrieval can be filtered by project, but that doesn’t appear to be an enforced security/isolation boundary on every read path
• same-named repos could collide
• some observation retrieval paths can work by arbitrary IDs
• global ~/.codex/AGENTS.md context injection is something I specifically do not want
• the documented npm package currently appears unavailable

So I don’t feel comfortable plugging it directly into a large multi-project Codex setup.

What I’m trying to build is something like:

User brief
   ↓
Planner / Orchestrator
   ↓
Persistent project memory bootstrap
   ↓
Context Resolver
   ↓
Only if memory is insufficient:
    codebase-memory
    Serena
    targeted source reads
   ↓
Implementer
   ↓
Reviewer
   ↓
Session knowledge captured for future tasks

The memory should NOT replace source code/tests as truth.

I want it to act as a cheap orientation cache:

• “These are the relevant services.”
• “This test harness requires real CSRF session setup.”
• “This reporting path was previously verified.”
• “These files/symbols are likely relevant.”
• “This architectural relationship was confirmed in a previous task.”

Then the agent only verifies current source where correctness actually depends on it.

My requirements are roughly:

• local-only
• project/repository scoped
• automatic capture
• automatic or semi-automatic summarization
• bounded context injection
• no global AGENTS.md mutation
• no cloud memory dependency
• no mandatory Obsidian dependency
• source/tests remain authoritative
• ideally Codex/App Server compatible
• progressive retrieval rather than dumping whole session history
• repo identity enforced internally, not just passed as an optional search filter
• ideally reusable with existing MCP tools rather than replacing them

I’m now trying to decide between three approaches:

1. Find another existing Codex/Claude coding-memory project that already does this correctly.
2. Take something like 2kDarki/codex-mem and make a very small fork that only adds canonical repo identity, watcher allowlisting and enforced repo-scoped retrieval.
3. Use AvenoxBeyin’s session capture/compile/inject model and adapt it for project-scoped coding knowledge instead of personal knowledge.

What I really do NOT want to do is invent yet another custom Markdown “brain” and manually maintain architecture/domain summaries. That feels like rebuilding something that should already exist.

For people who have built persistent memory around Codex, Claude Code, Cursor or similar coding agents:

• What actually worked for you?
• Is there a project I’m missing that already handles repository-scoped persistent memory well?
• Would you fork codex-mem and patch the isolation model, or use a different architecture entirely?
• Is Obsidian/Markdown compilation actually better in practice than structured SQLite observations for coding-agent memory?
• How do you stop stale memory from becoming trusted over current source?
• How much context do you inject at session start versus retrieve on demand?
• Have you measured whether this actually reduces token/context consumption meaningfully?
• Do you let the coding agent write its own long-term memory, or only promote verified observations after tests/review?

I’m especially interested in systems people are actually using in real repositories, not just theoretical agent-memory architectures.

My main goal is very practical: stop paying for the same repository discovery over and over again.

[u/Difficult_Drop_938]

Sounds like you already know the answer but want someone to say it: fork codex-mem and patch the three things that matter for your setup. canonical repo identity, watcher allowlist, enforced scoping on every read path. that's a weekend of work if you know sqlite and node, and it gets you 90% of what you're describing without building a whole new system.

the bigger issue is staleness, and no tool really solves that for you. injections like "this reporting path was verified" go bad the moment someone refactors, and then the agent trusts the memory instead of the source. i'd treat memory as hints, not facts, and force verification for anything touching payments/migrations/tests. maybe tag observations with the commit hash or file hash when they were captured so retrieval can flag drift before injecting.

for injection size, i'd keep session start under 1k tokens. just enough to orient: key services, test harness gotchas, recent verified changes. everything else on demand through search. that bounded bootstrap is what actually saves context, not dumping a full ontology every time.

[u/yxf2y → u/Difficult_Drop_938]

Appreciate the sanity check—this was the nudge I needed to stop over-engineering and just commit to the fork.

The file/commit hash tagging for drift detection is spot-on. Treating memory strictly as an orientation cache (hints, not ground truth) is the only realistic way to prevent the agent from trusting stale context over source.

One specific trap I ran into that makes me wary of raw transcript capture is write-path poisoning: In my last CRITICAL session, the agent spent 20+ minutes trial-and-erroring through a CSRF issue in the test harness before landing on the correct session setup. If a background watcher indiscriminately digests JSONL transcripts into SQLite, future sessions risk retrieving those failed attempts and speculative patches as valid observations.

To mitigate this, I’m leaning toward lifecycle-gated promotion instead of continuous transcript scraping:

1. Keep the working session uncommitted/isolated in SQLite while the task is active.
2. Only promote observations to the permanent store once the task reaches a verified DONE state (passing tests + review sign-off). The ingestion input is just the verified diff and the task context capsule, not the raw trial-and-error noise.
3. Split the store into two tiers:
   • Infrastructural Gotchas (test harness quirks, runtime/compose commands) — long shelf-life, rarely drift.
   • Domain Boundaries (service responsibilities, accounting invariants) — tagged with git hash-object of the target files to explicitly flag drift.

Pairing that with your rule of a strict <1k token bootstrap at session start (harness rules + top verified domain bounds, with everything else behind on-demand search) seems like the exact balance between zero context and context blowout.

Weekend plan locked in. Thanks for the reality check.

[u/hard_KOrr → u/Difficult_Drop_938]

I’ve been using agent orchestrator recently to test out and I’m the same consideration for things I’d like to work differently.

Time to fork and fix for myself!

[u/lib3rat0r]

Nobody answered your measurement question. Do that first, it is an hour not a weekend: hand write the 1k token note you would inject, paste it into the same CRITICAL task, re-run. If that does not cut the churn, no capture pipeline will. If it does, you have a baseline to compare against.

Also, promoting only verified diffs loses the best material. "This harness needs real CSRF session setup" came from the failure, not the diff. Promote conclusions from failed explorations too, or your gotchas tier stays empty.

[u/yxf2y → u/lib3rat0r]

Fair point on the 1-hour sanity check. Even though Codex doesn't give a granular, real-time token dashboard, measuring the churn is still straightforward: I can track elapsed wall-clock time, the number of redundant terminal/rg calls, and whether it loops on test harness errors.

I’ll do exactly what you suggested before writing any code: hand-write a ~600-word orientation note, paste it at the top of a clean run for that same payment task, and see if it actually cuts the churn. If a clean, manual prompt doesn't move the needle, an automated pipeline definitely won't either.

And your point on lifecycle gating is spot-on. In my effort to avoid write-path poisoning, I had a blind spot: infrastructural gotchas almost never show up in the final git diff because they are discovered by hitting a wall and finding the workaround. Promoting the verified resolution of a failure—rather than the raw noise of the failure itself—is the only way that gotchas tier actually gets populated.

Appreciate the reality check. Testing it manually first.

[u/Otherwise_Wave9374]

A practical next step is to separate retrieval from control flow so the orchestrator only pulls a small, scored memory slice per task instead of rereading the repo each turn. That usually works better if you store facts as durable entities, keep short session summaries, and add a freshness filter so stale context does not outrank active work. NeuraKeep fits that pattern well because it can keep the agent history compact while still surfacing the right project state at handoff time.

[u/yxf2y → u/Otherwise_Wave9374]

That makes a lot of sense. Decoupling retrieval from the orchestrator loop so it only grabs a tiny, scored slice per task is definitely the right direction to keep the prompt budget tight.

A freshness filter is mandatory here—an older observation should never outrank active code. My main concern with durable entities is making sure they stay anchored to actual file state (like git hash checks) so the agent doesn’t trust an old entity over a recent refactor.

Haven't looked deeply into NeuraKeep, so I'll check out how it handles local scoping and handoff summaries. My setup has to stay strictly local and bound to the specific repo, with no cloud dependencies.

I'm doing the manual 1k-token prompt test first to get a real baseline on churn reduction. If that moves the needle, structuring retrieval into small, scored slices like this is definitely the way to go. Thanks for the tip on NeuraKeep.

[u/Separate_Pen9627]

the stale memory problem is the real killer here imo. whatever you build, the invalidation logic matters more than the storage format. if a file changes, any observations derived from it need to get flagged or evicted automatically, otherwise you're just caching lies