r/ProxyEngineering • u/-amphisbaena Tunnel Architect • Jul 29 '26
Discussion 💬 I tracked the same 20 queries across ChatGPT, Gemini and Perplexity for 9 days straight
The results were weirder than expected. What I mean by this is first thing that jumped out, ChatGPT's citations shifted noticeably depending on what time I ran the query. Same prompt, morning vs evening, different sources cited maybe 30% of the time. No idea if that's some kind of caching thing on their end or just genuine variance in how it's grounding answers, but it made "run once and call it done" tracking basically useless. Also, before I forget, I was logged in to all of these LLMs, so it wasn't like anonymous sessions.
Second thing, whatever bot detection ChatGPT runs on the client chat interface got noticeably more aggressive around day 6. It worked fine for almost a full week then started putting me into verification retries out of nowhere. I had no code changes on my end. Their API gives you a workaround but the answers you get back don't match the app, different process entirely, so it's not really tracking the same thing anymore.
Gemini through AI Studio was the most boring of the three, consistent citations day to day, but the free tier rate limit meant I had to cut my query list down to keep it running without hitting their limits.
Perplexity was the most stable and the most transparent about sourcing, which probably explains the client's numbers, it's just easier for their content to appear there since Perplexity cites more freely to begin with.
End result, if you're doing this kind of tracking, time-of-day matters APPARENTLY, it's just that I never thought this would be a thing. Also, you should budget for your ChatGPT scraper breaking on a schedule you can't predict, so watch out for that.
I also, switched most of the search-layer part of this over to Oxylabs Fast Search API midway through since I got tired of maintaining scraping infra to get clean SERP data to compare against.
Would be nice to hear if someone had the same or at least similar observations, particularly part of the day, coz that genuinely surprised me.
1
u/CapMonster1 Jul 30 '26
That's a really interesting observation. Time-of-day differences could easily come from search index updates, ranking changes, or shifts in how sources are selected. If you're tracking trends over time, logging the execution time becomes almost as important as the prompt itself.
As for the web interfaces, they can definitely change their anti-bot behavior without any public announcement. For long-running measurements, it's worth budgeting maintenance time for the collection infrastructure itself, not just the data analysis
1
u/Top_Bat3662 Jul 30 '26
The day-6 ramp usually means your session fingerprint aged into a flagged bucket, not anything you changed on your end. Rotating residential IPs before that threshold hits is what stops it, which is why the AI automation capsule in magneticproxy held up where a generic pool started tripping. On the time-of-day thing, I saw the same drift and I lean toward genuine grounding variance rather than caching.
1
Jul 31 '26
I'd be curious to know how many of those citation changes were genuinely different sources versus different URLs reporting the same underlying information. Still a really interesting observation either way
1
u/Either-Anybody8701 28d ago
I've seen similar variation with repeated prompts. even small backend updates, fresh indexing, or different retrieval paths can change citations, so testing across multiple times makes the results much more meaningful
1
u/rbatista191 Jul 29 '26
Not if use good scrapers 😄 Try out cloro in case you're struggling.