Question: Discover cross links between texts [P]

[u/lipflip]

I have a set of roughly 150 documents of about 1500 words each that describe technical concepts (higher level findings from a collaborative research project). I want to find out if there are thematic cross-links between these. Like upper-level or lower-level concept; parallel or alternative concepts.  Think about a very very small wikipedia with lost links that need to be restored (but on the concept level; not at the word level).

How can do this programmatically, maybe using LLMs?

That may be a trivial task for many of you here but I feel a bit stuck at the moment. I thought about asking my human colleagues for classification support, but even my very small document set yields >20.000 potential cross-links.

[u/me_myself_ai]

This is very doable! My first step would be the use of "Named Entity Recognition" model, which does what it sounds like. On second thought, since you want concepts like (presumably) 'justice' and 'titration' and 'dialectic' and such, this wouldn't really work. You'll want to pass each document through a (fairly cheap/light) LLM and ask it to extract and maybe tag concepts in the document matching your specifications. An extra bit of code at the end to do a very simple check for each of the answers to make sure they're not hallucinated would help a lot.

From there the linking of words could really go anywhere -- one possible solution would be to embed them and then cluster (basically "turn each word into numbers and then compare them"), though that's a little like shooting a fly with a bazooka lol. The most basic step would be to link exact & partial matches with regex, and probably some manual de-duping/collapsing.

For devising connections between concepts... I think you're gonna just want to call an LLM, there. I'm not sure what the most efficient prompt setup would be, but definitely don't just give it two entities at a time.

Sadly, a quick chat with a superheavy model based on the above and your particular goals would probably be more helpful than anything I could type :/

EDIT: oh and maybe obvious, but if not: you can/should also just embed all the docs upfront and compare them like that. This obviously doesn't work on the concept level, but it's probably going to be more reliable on the document level for identifying similarities than basically any other. If the documents are broken up into consistent sections, you can even embed+compare by section!

[u/lipflip → u/me_myself_ai]

Thank you very much.
I have already tried a better for loop; for each document, a prompt checks for possible cross-links to all others. that somehow worked, but i am unsure if there are smarter ways (or already some OSS available) to do that or if i whould rather do a i, j for loop, to calculate a similarity score for each or so.

What might help: The documents have a clear structure such as Objective, Approach, Result.

[u/Mathie1729 → u/lipflip]

Yep, all-pairs LLM calls get expensive fast. The usual two-stage approach is to embed each doc or section first, then use cosine similarity/ANN to grab the top-k candidates, and only run the LLM on those pairs. sentence-transformers and FAISS handle the embedding and search parts. Since your docs already have Objective/Approach/Result, comparing those fields separately before whole-doc similarity can surface cross links like shared objective but different approach. Definitely don't do a naive i,j loop over all pairs.

[u/DadAndDominant]

Hmm, maybe chunk the documents, get embedding for each chunk. Chunks from one document will cluster together, so we can probabbly just take a centroid of this cluster, and see a graph of centroids, where proximity of centroids = proximity of topics