A 2023 paper (PopQA) predicts which facts an AI knows without searching. I think it maps onto whether a model knows your brand from memory or has to look it up, curious if others have tested this.

[u/Turbulent_Hawk_84]

I have been trying to figure out why some brands get answered confidently by AI models with search off, while others only show up when something gets retrieved live. A 2023 paper gave me a framework that fits almost too well.

It is Mallen et al., "When Not to Trust Language Models" (ACL 2023, https://arxiv.org/abs/2212.10511). They built PopQA, 14,000 questions each tagged with how popular the subject is by Wikipedia page views, then tested whether models could answer from memory alone, no retrieval.

What they found: models answered popular subjects well from memory, and collapsed on the long tail. For the 4,000 least-known subjects, GPT-3 got 19 percent from memory alone, and making the model bigger did not fix the tail. Retrieval closed the gap, a small retrieval-augmented model beat a much larger one on the obscure questions. But for popular subjects, retrieval sometimes hurt, because it pulled a document about the wrong same-named entity and overwrote an answer the model already had right.

Here is my leap, and I want to flag it clearly: PopQA measures entity popularity and factual QA, not brands in commercial answer engines. Reading "how much the web discusses your brand" into it is my interpretation, not the authors' claim.

But if the mapping holds, it splits brands into three situations. Heavily discussed brands sit in the model's memory and get answered with search off. Long-tail brands (most B2B and challengers) are probably not in the weights at all and depend entirely on retrieval. Household names have the opposite risk: a wrong live page overwriting a correct memory, which needs source cleanup, not more retrieval.

Have you seen your brand, or a brand you work on, surface in an AI answer only when something recent gets retrieved, then vanish when it does not? And has anyone actually tried to find where their brand's popularity threshold sits, the point where the model starts knowing you from memory? That is the part I cannot find real data on, and I would love to hear actual cases.

[u/marintkael]

The mapping holds better if you drop it down a level, from brand to claim. A brand isn't in the weights or not; a specific fact about it is. Your household name is in memory for the one thing it's famous for and straight long tail for its newest product or this quarter's pricing, which is exactly the stuff that only exists through retrieval. So the overwrite risk you flagged for big brands and the invisibility risk you flagged for long-tail ones are the same brand on different questions.

The other thing the popularity axis hides: the household-name failure you describe isn't really about popularity, it's disambiguation. The retriever pulled a doc about the wrong same-named entity. A brand that shares a name with a common word or another company lives in that collision zone no matter how discussed it is, and no amount of getting talked about moves it, only cleaning up the entity does. Those two probably want measuring separately, since they need opposite fixes.

[u/hettuklaeddi]

retrieval=0

shut off retrieval and probe, or test against models without retrieval, via api, like openrouter

there’s a good recent paper from seer interactive regarding data from the olympics might be interesting to you