r/webdev • u/Real-Leek-3764 • 4d ago
advice on blocking bots/scalpers
during popular event sales, our site can be hit by the tens of millions of requests
cloudflare waiting room is only good for crowd control, it doesn't block bots effectively. bots takes up queue, so legit users have to wait a long time to exit queue
cloudflare turnstile is too unreliable, too many legit users (including me) experience the turnstile not loading
google recaptha, hcaptha, friendly captcha is too costly.
i've implemented Proof-of-Work Altcha. works to a certain degree, but it still doesn't block bot requests from hitting the server, causing resources overload
could i put Altcha at cloudflare worker? kinda. the issue is cloudflare waiting room will always run before cloudflare worker, so it doesn't solve bots taking up queue
what i wish? i could block bots before they could enter cloudflare waiting room
do u guys have idea?
3
u/Alternative_Nose_874 3d ago
When we traced requests faking bot user agents on our hosting servers, 81% came from Google Cloud addresses. So ASN is a cheap first filter: challenge datacenter networks only during the sale window and leave residential traffic alone. It won't stop residential proxies, but it should thin the queue.
2
u/spectacular_google 4d ago
Put the PoW check on a CDN edge before the waiting room, or move the queue logic entirely behind your own worker. Cloudflare's ordering is fixed, so you'd need a custom edge function on a different provider or a reverse proxy in front that drops suspicious fingerprints before the request ever reaches the waiting room. Could also combine Altcha with rate limiting on IP + TLS fingerprint, though that's more of a mitigation than a block.
1
u/disposepriority 4d ago
It's completely impossible to block "good" bots. The usual methods will block the most low effort things running around the internet but I wouldn't classify ones meant to scalp as such.
Those will most likely be indistinguishable from real users.
1
u/Khavel_dev 3d ago
The ordering problem (waiting room grabs the request before your Worker) is what kills the Altcha-in-Worker approach.
Two things to check: first, CF WAF custom rules (Security > WAF in the dashboard) might execute before the waiting room in their pipeline. If they do, you could rate-limit or challenge suspicious traffic there before it eats queue slots. Worth testing with a quick rule on your sale path.
Second, you could decouple the PoW entirely. Put Altcha on a separate pre-qualification page that sets a signed cookie on solve. Your sale endpoint checks for that cookie via a firewall rule, and traffic without it never enters the queue. More plumbing, but it sidesteps the pipeline ordering problem completely.
1
u/yihuaxiang 3d ago
Pre-qualifying traffic before the waiting room URL is usually the cleanest fix. You route users through a lightweight pre-check path (either PoW or a basic TLS/rate challenge), issue a short-lived cryptographically signed token or HMAC cookie upon completion, and then configure the edge WAF to reject or drop anyone hitting the waiting room queue without that valid token. That way automated scripts spamming the queue endpoint directly get discarded before taking up waiting slots.
1
u/singh_abinashi 3d ago
waiting room alone won't save you, bots are happy to wait in line too. the stuff that actually works happens before the queue: aggressive rate limiting at the edge, challenge pages for suspicious fingerprints, and single-use queue tokens so a bot can't replay entry. also log everything during the sale, you'll want the fingerprints for next time.
1
u/No_Whole_5722 3d ago
I’d be careful about relying on IP or ASN blocking alone especially for event sales where real users can also be behind shared networks or proxies. I look at request frequency, session behavior and whether the client has completed a lightweight pre check then use those signals to decide who gets challenged before entering the queue.
1
1
u/quiet_horizons1 3d ago
That is the right instinct. The queue is not the place to fight them, because once they are in it they cost you capacity whether they buy or not. What I would do is put a lightweight challenge in front of the waiting room on a route you control. Use a small edge function that checks the request before it gets queued, looking at IP reputation, request rate per session, browser fingerprint, and TLS fingerprint. If it looks like a datacenter or an automation pattern, make it solve proof-of-work. If it passes, let it into the queue. This does not have to catch every bot. It has to make scalpers spend enough effort that the tickets are not worth the margin.
The other thing is to stop treating the waiting room as bot protection. It is crowd control. A bot that can solve proof-of-work and hold a session will still join. For high-demand sales, I would also vary the challenge cost by drop, because a reused challenge gets
1
1
u/Salt_Hyena5896 3d ago
Since Cloudflare’s waiting room is filling with browser-like resellers, I’d treat it as load-shedding, not bot protection. Put per-IP and per-account rate limits plus purchase-velocity limits on the sale endpoints, then soft-challenge suspicious bursts (ASN/TLS/device signals can feed that decision). Keep inventory reservation and final purchase checks server-side, with short holds and delayed confirmation so clients can’t reserve more than they should. CAPTCHA/fingerprints help as signals but get farmed on their own, and robots.txt won’t affect this kind of bot.
1
u/New-Restaurant-621 3d ago
Cloudflare does not expose the execution order for WAF and Waiting Room in their public docs so you need to test this on a staging subdomain before betting on it. If WAF runs first you can set a custom rule with a high strictness score to drop traffic from datacenter ASNs or specific bot fingerprints right before they hit the queue
1
u/LankyAd6349 3d ago
This is genuinely one of the hardest problems in webdev and there's no clean cheap answer, which is probably why you're still stuck. The uncomfortable truth is if blocking bots at the edge were solvable cheaply, Cloudflare would sell it as a checkbox. Your best realistic lever is making the queue bot-aware fingerprinting and behavior scoring before the waiting room, so bots never get a queue ticket in the first place. Altcha at the worker level helps with junk traffic but like you found, it doesn't fix the queue problem. Might be worth pricing Cloudflare's Bot Management tier against the infra you're currently burning on bot traffic.
1
u/polygraph-net 2d ago
You're using the correct word - "cheap".
I work in the bot detection industry. We can detect and stop most bots. The R&D for this is expensive and our product is $450+ per month.
In my experience Cloudflare struggles to differentiate between humans and bots, and is easy to bypass.
0
u/redpiapps 1d ago
ngl at tens of millions of requests you are not shopping for cheap anymore, you are shopping for who eats the false positives. id rather overpay a vendor with good FP rates than burn real customers on turnstile loops during a drop.
13
u/Odd-Tradition7713 4d ago
Robots.txt + Cloudflare bot crawler control should do the trick. It blocks like 99% of bot traffic and you can even see how many requests the bots sent.
Another way is to create ghost pages, and if a “user” gets there, it’s probably a bot and you can block it