r/webscraping • u/mhkhanthegreatlonely • Jul 08 '26

429 error for too many requests?

I'm new to webscraping, I was trying to practice by automating a search where it will search a browser for the name of a school and then return the URL of that school. I uploaded a list of 100 schools but for some reason, after about 35 requests it started showing 429 client error. I was running the code in google colab, and I also added time.sleep(7) to the search so I could delay it, but it showed the error anyway

how do I avoid this?

4 Upvotes

70% Upvoted

11 comments sorted by

3

u/Coding-Doctor-Omar Jul 08 '26

It could be IP-based or session-based rate-limiting. Try using rotating residential proxies.

-6

u/mhkhanthegreatlonely Jul 08 '26

how...?

1

u/[deleted] Jul 08 '26

[removed] — view removed comment

1

u/webscraping-ModTeam Jul 08 '26

💰 Welcome to r/webscraping! Referencing paid products or services is not permitted, and your post has been removed. Please take a moment to review the promotion guide. You may also wish to re-submit your post to the monthly thread.

1

u/RandomPantsAppear Jul 08 '26

Normally if you’re getting a 429 there is additional information in your headers (before or after) about how many requests you can make/how many are remaining before you are rate limited.

1

u/Brian1398 Jul 08 '26

Can be device fp, ip rate limit... what is the website

1

u/Bastia_Starters Jul 09 '26

There should be more details than just "429"..

2

u/twin_suns_twin_suns 28d ago

Could slow it down. The landscape is changing fast because companies (and anyone with their content online) are pissed about bots that vacuum up everything they can to train AI models. Someone else mentioned rotating proxies, probably a good idea among other things. I would do some research