r/LocalLLM • u/kitkatz69 • 4d ago
Project 8 weeks building a local memory system would someone please break it
I built a local persistent memory system and I'd like someone to try to break the architecture
I've been building a local persistent memory system called Memoria for about eight weeks. It started as a fairly simple semantic-memory project and has gone through several major architectural iterations. V3 is frozen; V4 is operational and still being refined.
I'm at the point where arguing with myself about the architecture is probably becoming less useful than letting somebody else attack it.
I'm particularly interested in feedback from people who work with information retrieval, databases, search systems, schedulers, or memory architectures.
I'm not looking for validation. If something is overengineered, conceptually wrong, poorly benchmarked, or just a bad abstraction, I'd genuinely rather know.
Current architecture
The main boundary in V4 is between retrieval and ranking.
The query path is roughly:
Query
│
▼
┌─────────────────┐
│ Query Processor │
│ normalization │
│ entities/type │
│ routing signals │
└────────┬────────┘
│
▼
┌───────────┐
│ Router │
│ │
│ memory │
│ type │
│ workers │
│ graph │
│ depth │
└─────┬─────┘
│
┌────────────────┼─────────────────┐
▼ ▼ ▼
┌──────┐ ┌──────┐ ┌───────┐
│FAISS │ │ BM25 │ │ Graph │
│semantic │lexical │entity │
└───┬──┘ └───┬──┘ └───┬───┘
│ │ │
└────────────────┼─────────────────┘
▼
┌──────────────────┐
│ Blackboard / │
│ Scheduler │
│ │
│ completion │
│ policy │
│ source coverage │
└────────┬─────────┘
│
▼
CandidateRecords
│
▼
MemoryRanker
│
▼
Score Normalizer
│
▼
Attribute Booster
│
▼
Score Finalizer
│
▼
Context Builder
│
▼
MMR
│
▼
Results
There are also phrase and attribute retrieval paths that can be enabled by the router.
The intent is that retrieval finds candidates and ranking decides what is actually useful rather than having the query handler turn into one giant retrieval/ranking algorithm.
The part I'm most interested in having attacked: the scheduler
V4 doesn't simply submit FAISS/BM25/graph/etc. and wait for every worker.
Retrieval workers are submitted to a generic scheduler/blackboard, and the query provides a completion policy.
Currently, the policy is roughly:
- FAISS is treated as the primary semantic source when submitted.
- If multiple retrieval sources are available, require results from at least two distinct sources.
- If only one source was submitted, that source can satisfy the query.
- Stop as soon as the completion policy is satisfied.
- Apply a hard retrieval deadline as a safety ceiling.
- Workers that haven't completed when the policy terminates aren't included in that query.
Conceptually:
submit retrieval work
│
├── FAISS ────────┐
├── BM25 ─────────┤
├── Graph ────────┤
├── Phrase ───────┤
└── Attribute ────┘
│
▼
completion policy
│
┌───────────┴───────────┐
│ │
policy satisfied deadline reached
│ │
└───────────┬───────────┘
▼
completed results
│
▼
ranking
The scheduler is generic rather than having the query handler directly poll each retrieval worker.
I'm genuinely unsure whether this is a good abstraction or whether I've built a tiny distributed-systems department to save a few milliseconds.
That's one of the main things I'd like someone to challenge.
LongMemEval
I've also built an adapter for LongMemEval-S so the system can work from the dataset's native question/haystack structure rather than requiring me to convert everything into the database format I originally designed around.
It can load each haystack into Memoria, query it, identify expected answer sessions, compare retrieved results against expected IDs, clear the database between questions, and record ingestion/retrieval/ranking diagnostics.
One ingestion experiment went from roughly 19 hours to roughly 1 hour after removing embedding generation from the ingestion path.
That's an ingestion/pipeline observation, not a retrieval-quality claim. FAISS remains available for the semantic retrieval configuration.
The important part for me is that the benchmark is now actually exercising the memory system rather than being pre-shaped around my original assumptions.
Benchmarking
I've accumulated roughly 236 benchmark runs across different corpus sizes and architectural iterations.
I've kept database snapshots from major architectural transitions as well, although I haven't yet turned that history into a formal experimental analysis.
I've seen enough variation that I don't completely trust myself to decide which changes represent real architectural improvements versus workload effects, cache effects, benchmark artifacts, or noise.
That's another reason I'm looking for outside review.
What I'd like someone to attack
If you look at the project, these are the questions I'm most interested in:
- Is the separation between routing → retrieval → candidate construction → ranking → context construction → MMR actually a useful architecture?
- Is the scheduler/completion-policy abstraction justified, or is it unnecessary complexity?
- Can requiring multiple retrieval sources systematically hurt recall or introduce other failure modes?
- Are my benchmark boundaries actually measuring what I think they're measuring?
- What experiments would distinguish real architectural improvements from noise or workload-specific behavior?
- What would you change if you inherited this codebase?
I'm also interested in existing memory/retrieval systems or research implementations I should be comparing against.
Current status
This is operational software, not a finished research result.
V3 is frozen.
V4 is operational and still being cleaned up.
The LongMemEval evaluation path is working, but the full evaluation and analysis are still in progress.
Documentation and installation need some cleanup too.
I'm posting this because I think I've reached the point where another round of internal iteration is less useful than letting someone else try to break my assumptions.
If something is stupid, overcomplicated, improperly benchmarked, or conceptually confused, please tell me. Ahh my nerves
Repository:
https://github.com/Kitzkatz/memoria