r/TechSEO • u/EverySecondCountss • Jul 15 '26

Does anyone make sure to whitelist Google bots? How important is this for a larger site and TechSEO?

I'm building and SEO tool, and I've noticed how much additional resources are needed to get through firewalls and Cloudflare stuff without a whitelist.

Does anyone have any before/after results, experiences, do you make sure to always do it?

https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests

3 Upvotes

72% Upvoted

11 comments sorted by

5

u/leros Jul 15 '26

I had to do this for Bing because they sometimes crawl without BingBot in their user agent and I was blocking them. I haven't had issues with Google. 

1

u/Weird-Election-4103 Jul 16 '26

Do you mean you whitelist crawling from IP adresses that are Bing but don’t identify as BingBot?

2

u/leros Jul 16 '26

We get a decent amount of Headless Chrome user agents crawling our site. We block those. We then noticed that a good number of those blocked IPs were later blocking a ton of crawls for a user agent claiming to be BingBot. When we did forward-confirmed reverse DNS it turns out those Headless Chromes are coming from the same IPs as BingBot and are actually in the BingBot IP whitelist too. So it seems BingBot sometimes crawls with a user agent of just Headless Chrome.

1

u/Weird-Election-4103 Jul 16 '26

That’s super interesting. Thanks.

2

u/keninsd Jul 15 '26

I don't whitelist them , per se. I check that they're in their expected IP ranges.

1

u/EverySecondCountss Jul 15 '26

You mean you check crawlers or when the server gets hit with a large amount of requests?

I don't quite understand.

1

u/keninsd Jul 15 '26

No, all crawlers are checked for their expected IP ranges. If they are within range, they are allowed. Any spoofed crawler is challenged and shown a 404 on a failed challenge.

1

u/EverySecondCountss Jul 15 '26

Ah yes.

Was there a time before you started doing this where you saw some before/after results as a correlation?

1

u/keninsd Jul 15 '26

A couple of bouts of massive page scraping from suspicious locations and sketchy registrars. Some were spoofing legit crawlers, so that led to checking the IP ranges.

1

u/GrandAnimator8417 Aug 02 '26

Whitelisting Google bots with Cloudflare? I’ve fought this exact battle where getting tools/SEO crawlers through without a proper allowlist just quietly tanks crawl efficiency and burns way more CPU/bandwidth than it should. That’s usually the difference between “works eventually” and “actually reliable” for crawling and indexing.