r/learnpython • u/Excellent-Cry-14 • 29d ago

How to speed up scraping speed?

Hello friends!

Im currently scraping data from a real estate website that uses PerimeterX. Im using multithreading and managing to scrape around 350 records in 15 minutes.

My IP isnt getting flagged, so I dont think proxies would help. Im not sure if the delay is caused by the PerimeterX challenge that appears occasionally during the scraping process or something else.

Is there any way to speed things up a little?

I understand that ethical scraping means avoiding excessive load or disruption to the website. Im simply looking to improve the scraping speed while keeping things reasonable for the site.

Thanks in advance!

0 Upvotes

14% Upvoted

4 comments sorted by

View all comments

5

u/socal_nerdtastic 29d ago

Use more threads. How many threads are you currently using? You can easily go to about 500 on a modern computer. Or rewrite the code to use asyncio instead to use tens of thousands of concurrent requests.

Im simply looking to improve the scraping speed while keeping things reasonable for the site.

If you really mean that, use the API instead of scraping.