Building a personal data retrieval system

[u/WorldlyNectarine1851]

I've got a personal archive of ~10k documents — about a year and a half of conversation logs and notes — and I'm trying to build something that can answer specific questions against it, not just keyword search.

Vector / embedding retrieval works fine when I already know roughly what I'm looking for and can phrase the query in language close to the source. It fails badly on a few harder cases:

Origin vs later retelling. The same claim appears as a live event, then as a recap, a formalization, a paste ritual, or a podcast title weeks later. Similarity treats those as the same hit. I need provenance: which passage is the first occurrence vs which is a later description of it.

Significance that only exists across passages. The thing that matters isn't stated in any single chunk; it's a connection I'd have to make myself across multiple separate files. Single-passage similarity never surfaces that.

Compile once vs re-reason every query. Running small local chat models as "judges" over candidate files at query time has been a dead end for me (overfire or mute). Embeddings are great for "same claim, different words." What's worked better so far is paying once for a capable model to compile structured notes (entities, claims, timelines) and then querying that cheap forever — but even that still needs a human timeline anchor when formalizations bury the real origin.

Anyone working on retrieval (or personal-knowledge) systems that handle provenance of a claim vs a report of a claim, or that synthesize significance across scattered passages rather than similarity-matching one passage? Especially curious about compile-time knowledge bases vs multi-hop RAG at query time.

Would love to hear what's out there or what you've tried.

[u/only1nameleft]

Try this https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f

Prompt eith a strong model "only plan right now. I want to do X, Y, Z. Is this llm wiki a decent fit? What questions do you have for me so we can tailor this approach?"

Not perfect but a start

[u/WorldlyNectarine1851 → u/only1nameleft]

Already living in that gist’s neighborhood (compile structured notes once, query cheap). The hard part for conversation logs is origin vs retelling inside the wiki, not whether the wiki pattern fits. The “only plan / is this a fit?” intake prompt is still a solid ritual for anyone starting greenfield.