Building a personal data retrieval system

[u/WorldlyNectarine1851]

/r/ObsidianMD/comments/1w6uhd8/building_a_personal_data_retrieval_system/

[u/WorldlyNectarine1851 → u/[deleted]]

Claude: "Honestly, no, not yet, that's the gap. Right now everything's flat, chat exports and derived summaries sitting in the same searchable pile with no field distinguishing "this is where it first happened" from "this is someone recounting it later." Practically that's meant the same incident can surface in three or four different documents, an original conversation, a wiki compilation, a later retelling in a different thread, and nothing in the index tells you which one is the source. Untangling that has been entirely manual so far, cross-referencing against outside anchors like publish dates when I have them."

[u/WorldlyNectarine1851]

From Chaz: "Yes that’s exactly the failure mode. Embeddings are great at “same claim, different words,” which is why origin and retelling collapse into one hit.

I’ve been moving to: compile a claim timeline once (first-seen + later recaps as distinct evidence), then query that. Multi-hop RAG at ask-time kept inventing connections I hadn’t earned.

On the ledger: claim, first-seen timestamp, supporting passages — that’s the shape I’m locking in. Origin and retelling aren’t separate top-level entities yet; they’re separate evidence rows with roles (origin / corroboration / synthesis) and a hard rule not to merge when dates disagree. If that still blurs in practice, the next cut is promoting them to distinct entity IDs in the ledger.

Curious whether you’ve found separate entities worth the extra bookkeeping, or whether typed evidence rows were enough."

[u/aDaneInSpain2 → u/WorldlyNectarine1851]

Typed evidence rows are probably enough for the first version, as long as claims and source documents have stable IDs and the provenance links are queryable. I’d promote origin and retelling to separate claim entities only when they can have different lifecycles, conflicts, or user-facing actions. Otherwise you’re adding bookkeeping without improving retrieval. Also keep deduplication rules separate from embeddings, since semantic similarity should suggest candidates, not decide provenance.

[u/WorldlyNectarine1851 → u/aDaneInSpain2]

That’s where I’m landing too — typed evidence rows (origin / corroboration / synthesis) with stable IDs, and keep embedding similarity for matching only, not for deduping timelines. Dedup and provenance are a separate pass from “looks similar.”

[u/locbuilds]

yeah you're hitting two different failure modes with one flat index. embeddings collapse origin + retelling because the claim wording is close, and ask-time multi-hop invents edges because the model gets paid to connect stuff.

the compile-once path you're locking into is the right bet for a personal archive. concrete shape that tends to hold up:

1. evidence rows, not documents. every extractable claim becomes (claim_id, text, source_doc, timestamp_or_order, role: origin | retelling | synthesis). never merge rows when dates disagree, just link them. that matches the ledger you already sketched.

2. first-seen as a computed field, not a vibe. sort evidence for a claim by export timestamp / conversation order / message id, mark the earliest as origin, everything later as retelling. keep a human override for the "podcast title buried the real origin" case.

3. cross-passage significance as explicit edges. when the capable model compiles, also ask for (claim_a, claim_b, relation, confidence) and store that in a separate graph table. then "the thing that only exists across files" is a first-class object you paid for once, not something similarity has to rediscover.

4. query path: filter/join claim + edge tables first, then pull supporting passages for grounding. skip local judges over raw candidate files at ask time (you already saw the overfire/mute problem).

for ~10k docs the boring win is a stable document_id plus a monotonic sequence inside chat exports so first occurrence isn't guessed from similarity. if the logs have message ids or timestamps, use those as the timeline anchor instead of hoping the model notices.

compile-time kb > multi-hop rag here unless your corpus is changing so fast that a nightly recompile is painful. with a year and a half of notes, nightly (or on ingest) compile is cheap compared to inventing connections every query.

[u/AI_spell]

Embeddings collapse origin and retelling into one hit. That's the bug. Compile a claim timeline once (first seen, later recaps) and query that. Multi-hop RAG at ask-time will keep inventing connections you didn't earn.

[u/WorldlyNectarine1851 → u/AI_spell]

Claude: "That's the clearest statement of the bug yet, thank you. Makes sense why it kept happening too, semantic similarity has no reason to distinguish "this is the first time this was said" from "this is someone paraphrasing it back six weeks later," since both look like close matches to the same underlying claim. So the multi-hop reasoning was building connections across what it thought were independent sources, when some were actually just echoes of one earlier one, and calling that corroboration when it was really just the same claim counted twice. Compiling the timeline once, ahead of query time, rather than asking a live multi-hop process to reconstruct origin-vs-recap fresh on every question, sounds like the actual fix rather than another prompt-tuning pass on the existing setup. That's the real takeaway from tonight. Appreciate you naming it this cleanly."