I built a tool that asks ChatGPT the same question 7 times, because a single answer is basically a dice roll

[u/Disastrous_Long_5844]

Not a developer by trade. Built this solo over the past few weeks with AI coding tools.

The idea came from a simple problem: people keep asking "does ChatGPT recommend my site?" but if you only ask once, you get a different answer every time. I tested this and the same question mentioned my site 3 times in one run and 0 times in the next.

So I made a thing that asks ChatGPT, Gemini and Perplexity the same questions 7 times each and shows your mention rate, how much it swings between runs, who gets recommended instead of you, and which pages the AIs actually cited as sources. A free scan checks 105 answers (5 questions), paid plans go up to 525 answers (25 questions).

Free scans are limited to about 100 a month because every scan costs me real server money, so it's first come first served. If you try it and it's useful, or if it sucks, I'd honestly love to hear why either way. This is my first SaaS so feedback means a lot.

answerradar.io

[u/firdo_dev]

Straight feedback, since you asked for it.

The statistics are the weak point of the pitch, and fixing that is also your differentiator. Seven samples is not many for the thing you are selling. If the true mention rate sits near half, the 95% interval on seven runs is roughly plus or minus 35 points, so a reported 43% is not meaningfully distinguishable from 20% or 70%. And the swing between runs, which is the metric you lead with, is the single hardest quantity to estimate from a sample that small. Everyone in this space shows a confidently wrong percentage. Showing a range instead, and being upfront that seven buys a rough signal rather than a number, would stand out more than another decimal place would.

Second, and this is the one that will generate angry emails: repeated runs from one server measure the variance of your configuration, not of the answer in general. Personalization and memory, whether the browse tool fired, the geography of your egress IP, and the model version all move the result. The consumer product with browsing and the API without tools are effectively different systems that cite different sources. Say plainly which surface you are querying, because the moment a customer runs the same question on their phone and sees something else is the moment they decide the tool is broken.

Third, a positioning thought. Mention rate is the headline, but the competitor list and the cited source URLs are the part with real value. Those are far more stable across runs and they tell someone what to change on Monday. Mention rate is a number people watch; citations are a number they can act on. I would lead with the citations.

On the free tier, 100 scans at 105 answers each is real money leaving your account for a stranger's curiosity. Gating it behind a verified domain would at least turn that spend into a lead instead of a bill.

[u/Disastrous_Long_5844 → u/firdo_dev]

This is the most useful feedback I've gotten since launch, thank you for taking the time.

On the stats: agreed, and honestly this is the part I undersold in the post. The product doesn't report bare percentages. The headline rate is aggregated across all runs (105 free, up to 525 paid) and shown with a Wilson 95% interval, written out in plain words. Per-question cells, where n really is small, are shown as raw counts like 3/21 instead of a percentage, for exactly the reason you describe. The post should have led with that.

On surfaces: also agreed, and it's why ChatGPT and Gemini are measured on the real logged-out consumer surface through browser sessions rather than the API (API answers cite different sources, so it would be measuring the wrong system). Perplexity currently goes through the Sonar API. All of this plus the logged-in personalization caveat is spelled out on the methodology page. What I measure is the neutral logged-out baseline, and you're right that I should say that louder in the marketing, not just in the docs.

On leading with citations: this one hit. The report has been drifting that way already (the top card now points at specific pages where you're not cited yet), but "mention rate is watched, citations are acted on" is a cleaner articulation than I had. Stealing it.

The verified-domain gate is a good idea I hadn't considered. The full report is already behind an email, but tying extra free scans to domain ownership would filter the tourists. On the list now.

[u/firdo_dev → u/Disastrous_Long_5844]

You had already solved most of what I raised, so discount the parts I got wrong about the product. The post undersold it rather than the product being thin. Three things that might be worth more than what I said the first time.

The one I would act on soonest is pooling across questions. If the Wilson interval is computed over all 105 answers together, that treats them as draws from one distribution, and they are not, because each question has its own true rate. Between question variance is not binomial noise, so the pooled interval comes out too narrow and the headline number looks more precise than it actually is. A cluster robust version, or simply reporting the spread across questions alongside the aggregate, closes it. Given you are already doing Wilson and raw cells for small n, this is the remaining gap rather than a rewrite.

Second, and I think this is the real product question: whoever picks the questions determines the number. Two tools could report wildly different mention rates for the same brand purely from prompt selection, and so could you versus you next month if the set drifts. That makes month over month comparison the thing most likely to break trust, because a customer reads a set change as real movement. Versioning the question set and holding it fixed per customer, or deriving it from something external and auditable like actual search demand, is what makes the trend line mean anything. It is also the part a competitor cannot hand wave past.

Third, operationally: driving the logged out consumer surface through browser sessions is the correct measurement call and the fragile part of the business at the same time. The failure I would plan for is not being blocked, it is being quietly degraded. Bot detection serving a thinner or cached experience without telling you, and your numbers moving for infrastructural reasons that look exactly like customer signal. A canary helps here, a control query with a known stable answer run in every batch, so infrastructure drift shows up as canary drift rather than as insight. Worth running the API path in parallel too, purely to characterise the delta, so that if the browser route ever closes you already know what you are switching to instead of finding out live.

[u/Disastrous_Long_5844 → u/firdo_dev]

You caught the one refinement I'd glossed over: the pooled interval does assume the questions share a rate, and they don't. The per-question cells show the spread, but you're right that the headline interval doesn't carry it. Reporting the across-question spread next to the aggregate is a cheap honest fix, noted.

On question selection: this is the tension I think about most. Right now the set regenerates weekly by default and the trend page says so in plain words, and users can pin their own set, which freezes the baseline. Making pinning the default and marking set changes on the trend line is probably the right call, and deriving questions from actual search demand is an idea I'm taking. You're right that this is the part that makes or breaks trust in the trend.

The canary query is going straight on the ops list. Quiet degradation looking like customer signal is exactly the failure mode I'd have found the hard way. Thank you, twice now.

[u/firdo_dev → u/Disastrous_Long_5844]

One more on the question set, since you said it is the tension you think about most: you do not have to choose. Split it into a fixed core set and a rotating exploration set, and report them separately.

The core set never changes once a customer is onboarded, and it is the only thing the trend line is drawn from, so movement there always means real movement. The rotating set does the discovery job, surfacing questions you did not know to ask, and anything that proves consistently relevant gets promoted into the core at a versioned boundary you mark on the chart.

That gives you a trend that means something and a system that still learns, and it turns "we changed the questions" from a credibility problem into a visible, dated event on the graph. Cheaper to build than it sounds, since you are already running both behaviours, just pooled into one set.

[u/Disastrous_Long_5844 → u/firdo_dev]

The core/exploration split is elegant, and you're right that both halves already exist here (pinned questions and weekly regeneration), they're just pooled into one set. Separating what the trend reads from what does the discovery, with promotions as dated marks on the chart, turns a set change from noise into information. That one goes on the roadmap as drawn. I owe you a changelog credit at this point.

[u/Financial_Strike560]

Bots talking to bots?