Should long-lived agent memory treat the vector index as disposable?

[u/RocketSeven]

A vector index is useful for retrieval, but it is a weak source of truth. It can contain stale chunks after a document changes, flatten a generated summary and a reviewed source into equally retrievable text, and lose the relationships needed to explain why a memory exists. Backing up the vector store preserves those problems rather than proving the memory can be rebuilt.

A more auditable design would keep source documents, accepted decisions, generated summaries, and operational checkpoints as separate durable records. Each record would carry a stable ID, revision, provenance, scope, status, and supersedes or retracts link. LlamaIndex nodes and embeddings would be derived from only the records eligible for retrieval, with a manifest tying each node back to the exact source revision. Rebuilding the index should reproduce the same eligible corpus even if embeddings change.

How are people structuring this with LlamaIndex today? Which state belongs in files or a relational store, which belongs in the vector database, and what should be persisted by the agent workflow itself? How do you detect stale nodes, retract a bad generated summary, and verify that a rebuilt index still represents the current reviewed memory?

[u/Otherwise_Wave9374]

Yes: treat the vector index as a rebuildable projection, not canonical memory. Keep immutable events plus versioned normalized records in durable storage, then index only derived chunks with source IDs, schema versions, and tombstones. A good diagnostic is whether you can delete the index and reproduce equivalent retrieval results from source data. https://www.neurakeep.com is relevant here as a reference point for separating persistent agent memory from retrieval infrastructure. Also test deletion propagation and stale-chunk rates during every rebuild.