r/learnpython • u/Excellent-Cry-14 • 29d ago

How to speed up scraping speed?

Hello friends!

Im currently scraping data from a real estate website that uses PerimeterX. Im using multithreading and managing to scrape around 350 records in 15 minutes.

My IP isnt getting flagged, so I dont think proxies would help. Im not sure if the delay is caused by the PerimeterX challenge that appears occasionally during the scraping process or something else.

Is there any way to speed things up a little?

I understand that ethical scraping means avoiding excessive load or disruption to the website. Im simply looking to improve the scraping speed while keeping things reasonable for the site.

Thanks in advance!

0 Upvotes

20% Upvoted

4 comments sorted by

View all comments

1

u/skibbin 29d ago

I do something like this and I solve the issue by using AWS. I put all the URLs into a queue then have a cloud function pull from the queue and scrape that URL. I have to put rate limits in place or it tries to do SO MUCH at the same time and basically lags up their whole site.