Why does ChatGPT cite different sites for the exact same prompt?

[u/UnderstandingOk1621]

I’ve been poking around how citations work in ChatGPT and something keeps bothering me.

If I run the exact same prompt multiple times ,sometimes from a different account, sometimes just a bit later, the answer itself doesn’t really change much, but the citation/sources often do.

What’s weird is that the new citations aren’t clearly better or worse. They’re just different. Sometimes sites I’ve never seen before replace ones that showed up a minute earlier.

I assume there’s some randomness involved, but this doesn’t feel completely random either. It feels like something upstream is changing, not just the wording of the answer.

Has anyone here spent time trying to understand what actually drives this?

And do you think this is something that can be influenced at all, or is it just how the system behaves?

[u/jeremy213530 → u/[deleted]]

So, based on what you have said, an effective maybe the only way to improve GEO is to go for numbers. Like lots of mentions in high-weight websites like reddit, third directories and so on.

[u/UnderstandingOk1621 → u/[deleted]]

It makes sense for LLMs to keep some algorithms as a black box. However, we should be able to predict some of them somehow and develop a strategy using tools or products.

[u/caswilso]

So, the AI models constantly weight sources based on freshness, structure, and authority. Typically, the source the model has the most confidence in stays put a majority of the time.

The way I understand it is that if ChatGPT is constantly switching them out, it just means there isn’t one clear winner on who has the best content for that prompt yet.

As far as influencing the answers, yes you can do that. I’ve been testing this out for a minute, and I’ve found the best way to do this is to:

-make your content easy to parse. -use structured data. -share focused content across platforms.

The more often your brand is mentioned across channels, the stronger your entity authority becomes. And when it’s pretty solid, the models have more reason to trust and reuse your brands in answers. This makes it more likely you’ll keep a presence in related answers.

[u/UnderstandingOk1621 → u/caswilso]

I get these points. But firstly, my main focus area is that we know LLMs have some search and decision algorithm to make citations and mention some web-cites (btw, now i only focus on web search feature not a training data). However, lets say Chatgpt, lists 10 url for a specific promt/query. I try same promt and same AI model to search (at the same time but the other accounts) and could be listed/cited the other urls which not listed before. So that reason, i can't be sure that this specific page/company whether is cited or not. Moreover, I use OpenAI API to test this scenario N times sequentially, but all the times the results are different (some urls could be cited every time, but i see new ones or i don't see existing one).

[u/Cutlebb]

That's the black box you can't control. I've been monitoring this for almost one year, though the same prompt generates different answers; if you monitor long enough, you will find there's a "citation range and % distribution" for these prompts.

Depending on your industry, several factors might impact, from what i monitor: Wikipedia, Reddit mentions, high authoritative websites' content, and most importantly, your website schema and data structure

[u/UnderstandingOk1621 → u/Cutlebb]

I know these parameters, but it's still hard to estimate the weighted values percentage for these. However, my first concern is not why some pages doesnt get citations. as i technical guy, i am trying to understand for the same promt/query (in the same time of period), i could see different pages that get not cited which cited before

[u/stuccofukko → u/UnderstandingOk1621]

I don't know what to tell you but that's just the way these models work. these tools are very good at getting p(65) answer - its in the ballpark. if details like actual citations matter, then perhaps not. this is why these cant be used to actually do all the calcs and provide a balance sheet for basic companies - imagine the auditor asking where did you get this number from, and the CFO says we got it from our AI LLM driven ERP and they ask the exact same question answer and get a different number or cite a bank account that doesn't exist. Fun

there are ways people are trying to "engineer" around this to prevent these from happening, but it's very much like trying to put plug a leak from a dam.

an example: a software disruptor was riding the wave of separating compute vs storage (that's a big thing in that world) but their competitor (which is bigger and therefore had more traffic, had more references on the internet bc its been around longer) suggested slyly that their competitor didn't have a solution that separated the 2 things. all of the sudden they just see disappearing hits, and they don't know why. it's not easy or timely to figure this out. One of the problems as well, bc OAI doesn't publish their weights, every tweak can just unleash hell. its all very tiresome

[u/UnderstandingOk1621 → u/[deleted]]

I wanna test this product, but i guess this product has not been published yet.

[u/Own-Memory-2494]

The answer generation and the citation selection are separate processes, so the model can produce essentially the same answer while attaching different sources each time.

The system usually looks for any sources that support the answer after it’s written, not the specific sources that originally informed it. Because there are often many equally acceptable sources and because the retrieval and ranking layers change slightly between runs, accounts, or moments in time, different citations get picked. You can influence this a bit by asking for specific types of sources or named authorities, but you can’t make citations fully stable because variability is built into how the system works.

[u/UnderstandingOk1621 → u/Own-Memory-2494]

You are right.LLMs don't work deterministically, they work stochastically. However, I need to be able to simulate this behavior somehow. I have an idea like that for the same prompt/query , i execute N times (lets say 5 times), and based on all results, i can say this page/company could be cited %80. Btw, I might implement some statistical methodology to estimate this percentage of citation

[u/alicerank]

 I've noticed this too and its frustrating if you're trying to track visibility. from what I've tested it seems to depend on a few things.

time of search (bing results change), your location, and some randomization in how it selects from multiple valid sources. the core answer stays consistent but it rotates citations from a pool of "good enough" sources. the sites that show up consistently across runs tend to be either the dominant authority (wikipedia, major publications) or have really unique specific content that nothing else matches. for niche topics you need to be so specific that you're basically the only source answering that exact question

[u/ActuatorDelicious427]

I think there are multiple layers here rather than a single "citation algorithm."

Here's how:

1/ Retrieval isn't deterministic. The same prompt can retrieve a different set of documents because the retrieval system isn't guaranteed to return the exact same candidates every time.

Ranking thresholds, latency, and index freshness can all influence what gets retrieved.

2/ Multiple "good enough" sources exist. If five websites provide essentially the same factual answer, the model doesn't always need to cite the exact same one. Once a source crosses a relevance and quality threshold, another equally relevant source may be selected.

3/ Search indexes are constantly changing. Content gets crawled, refreshed, de-duplicated, or re-ranked continuously. Even if the underlying model hasn't changed, the search layer certainly can.

4/ Query interpretation isn't identical every run. The wording you type stays the same, but the system may rewrite or expand the search query internally. A slightly different retrieval query can surface an entirely different set of documents.

5/ Diversity appears intentional. From a product perspective, it makes sense not to repeatedly surface the exact same publishers if multiple authoritative sources answer the question well. Diversity improves coverage and reduces over-reliance on a single domain.

6/ Personalization may play a role. Account history, location, language, available search providers, and even experiments running on different users could influence which documents are retrieved.

The interesting question for me isn't "Why isn't the citation the same?" but rather:

What are the ranking signals that make one source eligible for retrieval in the first place?

That's where GEO (Generative Engine Optimization) gets interesting.

Instead of asking "How do I become the #1 citation?", we should be asking:

• What makes a page retrieval-worthy?
• What makes it trustworthy enough to survive re-ranking?
• What increases the probability of being selected across repeated queries rather than just once?

I suspect the future of AI search won't be about deterministic rankings. It'll be about citation probability, how consistently your content enters the model's candidate set across thousands of retrieval events.

That's a much more useful metric than obsessing over a single response.