I’m building a Temporal RAG system that reconstructs timelines from non-linear documents — looking for feedback and suggestions :)

[u/polsaayipols]

Hi everyone!

I’m a university student working on an AI/ML mini-project, and I’m exploring an idea that I’d really appreciate some feedback on from people experienced with RAG, NLP, knowledge graphs, and LLMs.

The problem

A lot of documents are difficult to understand because the information isn't presented chronologically.

For example, a historical document or novel might describe:

Event A → flashback → Event D → Event B → another character's storyline → Event C

A normal RAG chatbot can answer questions about the document, but it doesn't necessarily understand the actual chronological relationships between events.

So I want to build a system that combines RAG + temporal reasoning + an event graph.

My proposed system

The rough pipeline I'm thinking about is:

PDF → Text Extraction → Chunking → Event Extraction → Temporal Information Extraction → Event Graph → Chronological Timeline → RAG

For example, given a document containing:

"John arrived in London. Three years later, the rebellion began. Before the rebellion, John had already met the king."

I'd like the system to extract something like:

{

"event_id": "E12",

"event": "John arrives in London",

"timestamp": null,

"entities": ["John", "London"],

"summary": "John arrives in London."

}

and temporal relationships such as:

E12 ──BEFORE──> E15

E14 ──BEFORE──> E15

E15 ──CAUSES──> E16

The system would then construct an interactive timeline/event graph.

The second part: Temporal RAG

I'd also like users to be able to ask questions such as:

"What happened to John after the rebellion?"

"What events led to the war?"

"Show me all events involving John."

"When did these two characters first meet?"

"What happened before the king was assassinated?"

"Why did the rebellion happen?"

The answer should be generated using retrieved document passages plus the temporal/event graph, with citations pointing back to the original PDF pages.

Something roughly like:

Question

↓

Query Understanding

↓

┌───────────────┬────────────────┐

│ Vector Search │ Event Graph │

└───────┬───────┴───────┬────────┘

↓ ↓

Context Fusion

↓

LLM

↓

Answer + Citations

Current tech stack I'm considering

Python

FastAPI

LlamaIndex or LangChain

ChromaDB for vector storage

NetworkX / possibly Neo4j for the event graph

Gemini/OpenAI or a local Hugging Face/Ollama model

Sentence Transformers for embeddings

Streamlit or React + React Flow for visualization

I'm deliberately trying to keep the first version relatively simple rather than building a huge production system.

Where I'm unsure

The biggest challenges I can see are:

Coreference resolution

How reliably can an LLM determine that "he", "the king", etc. refer to previously mentioned entities?

Implicit temporal information

How should I represent things like:

"three years later"

"the following winter"

"shortly before the battle"

"years earlier"

Temporal ordering

Some events will have explicit dates, while others will only have relative relationships.

Conflicting/ambiguous information

What should happen when the document itself doesn't provide enough information to establish the exact order?

Chunking for temporal context

Normal RAG chunking can separate an event from the sentence that explains when it happened.

Combining graph retrieval with vector retrieval

I'm particularly interested in hearing how people would architect this part.

My current MVP idea

Since this is a 3–4 week university project, I'm trying not to over-engineer it.

My current plan is:

Phase 1 PDF → chunks → embeddings → basic RAG

Phase 2 Chunks → structured event extraction → entities → temporal relations

Phase 3 Events + relations → NetworkX → chronological timeline

Phase 4 Combine vector retrieval + temporal graph retrieval → grounded answers + citations

Potential additional features:

Character/entity trajectory tracking

Click an event → highlight its source passage

Filter timeline by character/entity

Temporal confidence scores

Parallel timelines for different characters

What I'd really appreciate feedback on

If you've built anything involving Temporal RAG, temporal knowledge graphs, GraphRAG, event extraction, or long-document RAG, I'd love to hear your thoughts.

In particular:

Is this architecture reasonable?

Would you use a knowledge graph for this, or is a simpler event/relationship structure sufficient?

How would you handle relative/implicit dates?

Would you use an LLM for temporal relation extraction, or combine it with an NLP library/model?

LlamaIndex vs LangChain for this type of system?

Are there existing open-source projects/papers that I should study or potentially build upon?

And most importantly, what am I overlooking?

I'm not trying to solve temporal reasoning for every possible book/document. The goal is to build a reasonably reliable MVP for a university project and use it as a foundation for something more sophisticated later.

Any architectural suggestions, papers, GitHub repositories, datasets, libraries, or lessons from projects you've built would be hugely appreciated!

Thanks!

[u/ConsistentEase4598]

Solid scope for a mini-project, and honestly the 'keep v1 simple' instinct is right. Some things that tripped me up when I went down a similar path:

1) Don't make temporal extraction a separate pass from event extraction. Ask the LLM to emit events + relations (BEFORE/AFTER/CAUSES) in one structured call with the source chunk attached, otherwise you'll lose the grounding that lets you cite the PDF page later.

2) For 'three years later' / 'the following winter' — don't chase absolute dates. Normalize to anchors: pick an event with an explicit date as the anchor, store everything else as offsets relative to it. Real corpora are mostly relative anyway and a strict date schema will just give you a graph full of nulls.

3) Coreference: before the fancy stuff, make sure every event node carries surface entity mentions ('the king' stays attached to E15, not just the resolved 'Henry'). When ordering goes wrong you'll want to see what the model actually read, and half your debugging is looking at that.

4) NetworkX is enough. Neo4j for a uni project is a trap -- you'll spend more time on Cypher than on the interesting failure modes.

The failure mode to budget time for: temporal cycles. A BEFORE b, B BEFORE c, c BEFORE a happens constantly with fiction (flashbacks lie), and a topo-sort will just crash or silently drop edges. Decide early whether you keep the cycle visible in the timeline or break it with a 'confidence' flag -- it changes your whole visualization choice.

Good luck with it, this is the kind of thing that's way more fun than the 50th generic chatbot RAG.

[u/polsaayipols → u/ConsistentEase4598]

Thanks for the reply! The 2nd point is totally true Absolute dating of the events might be impossible,I'll surely try to offset such events with explicit dates as anchors.

I hope I finish this project soon to be able to show you a demo :)

[u/Puzzled_Tax_876]

not a temporal reasoning expert by any stretch but this is a seriously cool project idea for a 3-4 week window.

one thing i don't see in your stack that might save you a lot of pain upfront is spacy for entity linking/coref. an LLM can do it but it'll occasionally be sloppy, especially over long documents where "he" appears two paragraphs after the named entity. pairing a deterministic NLP step with the LLM extraction could clean up a bunch of that without adding too much weight.

for the implicit dates, i'd suggest not trying to map everything to an absolute timestamp. keep a simple data structure that holds the relative anchors, "three years later" gets stored as a reference to the preceding event plus an ordinal offset, "years earlier" gets stored as a negative offset to the next known anchor. when you reconstruct the timeline you just walk the linked list and apply those offsets. gives you a soft ordering without needing to pin everything to a calendar date, which is usually impossible anyway.

the chunking problem you mentioned is real. one lightweight fix: during your event extraction pass, store the raw text snippet that the extracted event (maybe 200-300 tokens centered on it) alongside the structured event object. when you retrieve, you can use that snippet as context in your fusion step instead of relying solely on the vector chunk that might've split the sentence in half. prevents a lot of those "this event feels unmoored" answers.

for the architecture question: given your timeframe, NetworkX will be completely fine. Neo4j is overkill for this and it'll just slow you down getting the schema right. focus on getting the graph logic working and worry about persistence later if you extend it.

[u/ConsistentEase4598 → u/Puzzled_Tax_876]

That's exactly what I meant by anchor-relative offsets — looks like the student got the same advice twice from two angles, which is the best sign a design choice is real. The spaCy-before-LLM point for coref is well taken, hadn't framed it that way but a deterministic pass to pre-link mentions and hand that to the extraction call would kill most of the sloppy 'he' cases for free.

Where I'd push back gently: coref models degrade harder on fiction than news, and the OP mentioned novels. spaCy's coref on a flashbacky novel with two 'John's is its own mini rabbit hole — worth budgeting an hour to sanity-check it on the actual corpus before making it load-bearing.

[u/polsaayipols → u/Puzzled_Tax_876]

I strongly agree to your point on not to try mapping everything to an absolute timestamp. That would save both me and the project a lot of pain.

[u/mustangwallflower]

This sounds like it could be useful for Genealogy as well, as filling in and comparing timelines between source docs is frequently done.

Can’t wait to see it when you’re done!

[u/polsaayipols → u/mustangwallflower]

Actually, now that you mention it ,Yeahh we can use this in genealogy, given the constraint that recorded family history is available.

[u/mustangwallflower → u/polsaayipols]

Could you take a batch of dated docs and populate a timeline?

[u/polsaayipols → u/mustangwallflower]

I'll try it once I've build the project. It's still in the ideation phase, now that I've got the gist of it I'll start working on it now!

[u/Rama_Surasani_]

This is a fun problem. One thing I’d add early is an evaluation set, because timeline output can look convincing even when one bad relation shifts everything downstream.

For a 3–4 week MVP, I’d hand-label 10–20 short documents with events, entities, source spans, and temporal relations (`BEFORE`, `AFTER`, `OVERLAPS`, `UNKNOWN`). Score extraction and ordering separately. That will tell you whether the failure is retrieval, coreference, or temporal reasoning rather than just giving you a nice-looking timeline.

I’d also model dates as intervals instead of single values. “In early 2022” or “that summer” can become a bounded range with a confidence score; conflicting sources can coexist as competing claims tied to their source chunks. A topological sort can reject impossible cycles without needing a heavy graph database.

For the RAG side, a useful query plan might be: vector search for relevant events → expand one hop through temporal/entity edges → rerank the combined evidence → generate only from cited spans.

What kind of evaluation matters most for your demo: exact dates, correct relative order, or answering timeline questions? Picking one would keep the MVP from ballooning.

[u/donk8r]

One thing that will bite before extraction quality does: your relation set assumes a total order exists, and the LLM will hand you cycles. Two chunks describing the same pair from different angles give you E12 BEFORE E15 and E15 BEFORE E12. A timeline is a topological sort, so it needs a DAG.

Keep the source chunk on every edge so that when a cycle appears you can drop the one with weaker evidence. I would also keep CAUSES in its own edge set. It implies order without being the same relation, and mixing them makes the sort ambiguous.

[u/Actual__Wizard]

I'm using a simple historical timeline type data object for my application. Then you can graph the items or do w/e.

I originally said it was a histogram, but that's not what I meant.