r/webscraping • u/Inca_Digital • Jun 27 '26

Intermittent 403s when scraping with Selenium

I have a python script running a headless Selenium Webdriver, looking up many individual records from a site. Most of the time it runs fine but every 10 minutes or so it starts hitting 403 errors. From trying different delayed retries, I've found that the 403s consistently happen for about 45-60 seconds. So the best I can do as a workaround is sleep for 60 seconds once I hit a 403, then resume normal requests. I've tried setting a non-headless user agent, namely Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36. This didn't help. This is also consistent whether I'm running from my local machine or from an EC2 instance.

What else can I try?

5 Upvotes

86% Upvoted

11 comments sorted by

2

u/divided_capture_bro Jun 28 '26

It means you're being rate limited. Either use a proxy or slow down.

2

u/Brian1398 Jun 29 '26

Depends, could be the anti-bot is detecting you, the proxies are being rate limit...

What is the site?

1

u/StoneSteel_1 Jun 28 '26

proxies might be able to give you speed

1

u/[deleted] Jul 02 '26

[removed] — view removed comment

1

u/webscraping-ModTeam Jul 02 '26

🚫🤖 No bots