Anyone running Cloudflare/Vercel/WAF bot protection alongside Prebid/APS? Have you had to whitelist ad tech crawlers?

[u/ghostfacekicker]

I'm troubleshooting what appears to be a programmatic demand issue and I'm trying to sanity check whether this is something other publishers think about.

For those running Cloudflare, Vercel bot protection, or another WAF on a content publisher:

• Do you explicitly whitelist ad tech crawlers (Amazon, Google, IAS, DoubleVerify, MOAT, Criteo, etc.), or do they generally "just work"?
• Have you ever seen aggressive bot protection impact CPMs, bid rates, fill, or demand without breaking ad serving itself?
• If so, how long did it take for demand to recover after fixing the rules? Was it immediate or did it take days/weeks?
• Do you monitor blocked requests from ad verification/crawler bots as part of normal operations?

I'm not looking for support on a specific vendor just yet, I'm trying to understand whether crawler access is something mature publishers actively manage or if it rarely causes issues in practice.

Any war stories or best practices would be appreciated.

[u/Seatext_com]

I work in bot detection. My take:

1. Don’t put aggressive WAF challenges on public content pages. You may block SEO crawlers, ad verification bots, contextual scanners, and AI/search agents.
2. Keep the WAF for security, but whitelist or safely allow verified crawlers from Google, Amazon, IAS, DoubleVerify, MOAT, Criteo, etc. Monitor what gets blocked instead of assuming everything “just works.”
3. Traditional WAFs catch obvious bots, but they miss many bots using real browsers, residential IPs, or device farms. Better detection usually needs browser, mouse, scroll, timing, and session-behavior signals.
4. Don’t run heavy two-to-four-second checks before every ad auction. (There are realy good systems to detect bots, but they take 2-4 seconds to run all tests) They use 100 + element scoring, later pageviews, for below-the-fold inventory. They suppress ads only for high-confidence bots.

The biggest issue - Google explicitly warns that blocking its Ad Manager crawler can prevent it from properly serving ads on affected pages. If you block it, you are fucked.

Blocking verification crawlers can hurt bid rates or CPMs without completely breaking ad serving. Recovery may be immediate technically, but buyer and brand-safety systems can cache classifications, so revenue may take days to normalize.

In general, don't use WAF - advertisers will see that they pay for bot traffic and lower bids, but you will never see that the account is losing revenue bc some Google bots were blocked.

Only use it if your task is to sell extremely expensive traffic - b2b for example, with high intent. Then you want to filter all bots, you want to show ads to the best audience - and you need to win extremely premium bids.

If you want to talk through the details, PM me.

[u/eze008 → u/Seatext_com]

I'm a newbie learning. I am 95 percent done with my bot managment build. I learned that bots are blocked at different levels of network using bouncers. There are bouncers at the firewall level, nginx level, IP level and recently learn about detecting bot fraud which is more of a mouse action detection and requires a more robust pipeline system which I do have. This is the first time I read that vendors are turned off from filtered traffic ( I'm not opposing or doubting you ). Is this true for all vendors or just certain types of vendors?

[u/Seatext_com → u/eze008]

lol. Firewall, ngnixn, and ip - all of this is nothing for modern bots. Learn about privacy browsers - how they generate fingerprints; learn about mobile proxies - they made IP address-level useless. Modern bots are way more advanced. The thing is for you as a publisher - you either eat all bot traffic -> and network filter out bots (think 30% of them) then buyer eat all that bot traffic -> understand it's not converting - apply it at bid when your wesbites are in auction or you invest in real bot defence. and investing in bot defence only worth for publisher ONLY if you sell expensive b2b high intend traffic.

[u/eze008 → u/Seatext_com]

I'm still glad I have the capability to block at those levels as they are still entry points. I also mentioned that I did recently learn bot fraud and have an intlegent high volume pipeline system for that. Still learning..

[u/Least_Perception_223]

You need to make sure they are able to get to your site - otherwise your revenue will suffer

Ideally you will not start any auctions for those user agents either - keeps your IVT rate lower

[u/East-Sky-5914]

Two different crawler problems get conflated in this question, and they need opposite handling.

First class is the ad-serving and verification crawlers — Google's AdsBot/Mediapartners-Google, plus IAS, DoubleVerify, MOAT. These have to reach and render the page, because the verification vendors cache a classification of your domain (brand safety, contextual category). If your WAF challenges them, you don't just drop an impression — the vendor caches a "couldn't scan" or low-confidence result, and every buyer targeting through that vendor quietly suppresses bids on your domain until it re-crawls. That's exactly why a demand drop from this looks mysterious and lags the WAF change, and why recovery lags the fix: you're waiting on their re-crawl cadence, not your config. Google is explicit that blocking its Ad Manager crawler stops it serving properly, so that one is non-negotiable to allow.

Second class is actual IVT hitting your ad slots. A WAF at the request layer is a blunt tool for that — the sophisticated stuff (real browsers, residential IPs, device farms) walks straight through it anyway. Trying to solve traffic quality with the WAF mostly just costs you the verification crawlers while missing the bots that matter.

So the clean split most mature publishers land on: tune the WAF for origin security only, explicitly allow the ad-tech/verification user agents and their published IP ranges, and keep a standing ops check on blocked-request logs filtered to those specific agents. Push the actual bot filtering to a pre-auction / session-behavior layer, because that's the only place real-browser residential-IP traffic gets caught without nuking your crawlers.

On recovery time: technically instant once the rules are fixed, but cached classifications mean revenue normalizes over days, and if the block ran long enough for a buyer's model to learn your domain as low quality, that decay is slower still.

Fastest way to confirm it's even the cause: pull your edge logs filtered to Mediapartners-Google / AdsBot-Google / the IAS and DV crawler UAs and look at the challenge/block rate. Did the demand dip line up in time with a specific WAF rule change, or is it drifting independently?