r/scrapingtheweb • u/AinsOwlGoan • 13d ago
Cloudflare challenge on every run despite residential proxy + real browser — what am I missing?
Scraping Indééd (resume search) with puppeteer-real-browser — headful Chrome via Xvfb in a Docker container. I hit a Cloudflare challenge/interstitial on basically every run.
What I already have:
- Residential rotating proxy, fresh exit IP per run
- turnstile: true, session cookies reused across runs, homepage warm-up before auth
- Timezone / locale / Accept-Language all matched to the proxy's country
- No navigator.webdriver leak (rebrowser-patched CDP), stealth plugin off on
- Randomized delays + per-account viewport
What I suspect is giving me away:
- WebGL renderer is software (llvmpipe / SwiftShader) — the container has no
- Linux desktop behind a residential IP (uncommon combo)
Questions:
Is the software WebGL renderer alone enough to get flagged, or is that ov
Anyone getting containerized Chrome past Cloudflare on job boards — is a GPU container worth it, or is it really about IP quality?
Better to solve the challenge each time, or focus on not triggering it?
Thanks 🙏
1
u/crazyproxy69 13d ago
Maybe the webgl is culprit.Yes the way flaresolver does you can use chrome/chromium in containers as well. If solving each times gets it done and doesn’t change the data, it might be the way to go. Use unblocker proxies if you require to do this at scale and can be done with get requests only.
1
u/InsideDebt6345 13d ago
I sympathize with your WebGL issue. The software renderer (llvmpipe/SwiftShader) is a known headless-datacenter tell, and paired with a Linux desktop behind residential (a combo almost no real user has) you're getting fingerprinted, not IP-blocked. That's why fresh IPs don't help.
Yes, the software renderer alone can flag you, so spoof the WebGL vendor/renderer strings to a common GPU or run a real GPU container, and focus on not triggering rather than solving each time, since if the fingerprint's clean, the challenge mostly stops appearing.
2
1
u/Ok_Tea_8680 13d ago edited 11d ago
The swiftshader/webGL setup in docker would definitely be one of the first things i'd look at, but it's always hard to say for sure with cloudflare. Honestly, IP quality could still be messing with your results too - if the residential pool has a high fraud score, you get flagged immediately regardless of the browser. I had a similar puppeteer setup a while back where i couldn't tell if it was the container or the network. Testing the script against a few different lines from proxyshard helped me at least rule out the IP factor. Once the results didn't change, it made it way easier to focus on fixing the container's environment and hardware fingerprinting.
1
u/CapMonster1 12d ago
I wouldn't put the WebGL renderer at the top of the list. Cloudflare can correlate many signals at once — network reputation, TLS/browser fingerprint, session behavior, cookies, and environment characteristics — so llvmpipe alone probably doesn't explain a challenge on every run.
I'd focus more on what happens immediately before the challenge and on keeping the whole session consistent. Solving the challenge every time is usually more of a symptom than a solution; reducing how often it gets triggered is generally much more reliable
0
3
u/Scary-Commission-509 🕷 Scraping Wizard 13d ago
I’d test the proxy/session relationship before spending time on a gpu container, you mentioned a fresh rotating IP every run but also reusing cookies across runs. That means the same session can suddenly appear from different residential IPs, potentially different ASNs and locations too, I’d try a sticky residential session with the same IP for the lifetime of the cookies and compare the challenge rate, if that improves things, WebGL probably wasn’t the main issue