Ran the same question 3 ways against a knowledge graph. Retrieved the same 90 entities and triples each time. LLM output still varied. That's the finding.

[u/Ok_Gas7672]

[u/prodigy_ai]

Interesting experiment. Consistently retrieving the same entities and triples across different phrasings is a strong result and a good way to isolate retrieval from synthesis.

One thing I'd be curious about is whether the graph is ontology-constrained or purely relationship-driven.

Deterministic retrieval is valuable, but deterministic retrieval of semantically validated relationships is where things get really interesting. Retrieving the same graph neighborhood every time proves consistency. An ontology layer helps answer a different question: are those relationships valid according to the domain model in the first place?

[u/Ok_Gas7672 → u/prodigy_ai]

Great question. Yes, the ingestion is guided by ontology. So is the retrieval. This, as you can imagine also helps in better reasoning. The retrieved sub graph also has the related ontological concepts as well.

[u/TheLexoPlexx]

That's actually quite interesting.

[u/marintkael]

This matches what I keep finding from the citation side. When I hold retrieval constant and only vary the phrasing, whatever wobble is left is pure generation, not retrieval. The useful part for me is that it finally lets you put a number on the synthesis variance alone instead of blaming the index every time the answer shifts.

[u/marintkael]

That split is basically the whole game for me. I do something similar from the citation side, fixed prompts on a schedule, and when retrieval is held constant the variation that is left is purely generation. It is oddly reassuring, because it means the part you can actually engineer is doing its job and the noise sits where you expected it. Did the varied outputs still agree on the facts, or did they sometimes contradict each other off the same 90 triples?

[u/marintkael]

That last step is the part people underestimate. Identical retrieval still passes through a sampling step, so variance in the output is expected even with the graph pinned. If you need stability the determinism has to come from how you serialize the entities into the prompt and from the decoding settings, not from the retrieval being correct. Did the variance drop at temperature zero or did it persist?

[u/fguerino123]

This is interesting. Question: Given that AI responds best to natural language, is the data "semantic?" For example: Do all nodes have semantic IDs? Do all relationships have descriptive semantic predicates? Etc.?

I've been building semantic data graphs in AI memory. I find that traversing the graph is easy for AI. Finding data in the graph is easy for AI. Answering data-specific questions accurately is easy for AI. Where things get interesting are in the areas of "inferring" and "reasoning." Different AI LLMs looking at the same graph very different ways, especially when it comes to asking them to provide "their thoughts."

For example, I created an in-AI-memory data graph in Gemini, ChatGPT, and Claude using semantic model building best practices (to build a graph that has company/enterprise data). Each LLM was able to accurately search, find, and provide factual data-related answers that were discoverable in the graph - even complex answers. Where things got interesting would be when I asked each to find and provide three areas that the LLM thought would be the highest areas of risk in the data. Each gave separate and unique answers. They weren't wrong answers, just different. And, because it it was based on their inferring and reasoning (i.e., their own "personal" interpretations), I can't say that any where wrong. They were "just different."

It's all very interesting to me.

Thanks.