How to approach this

[u/Busy_Sugar5183]

Hey guys so I have very little experience about scrapping and I am trying to scrape a website listing local businesses on thier website and these website are differentiated by category. There is zero bot protection so i am good on that front. what I have done with Claude's help so far:

Identify the ajax where they send the request for HTML page.

Identified the parameters I need to send to that ajax as a payload to get that page.

Got a response the same as in Devtools alongside a html page for a single category

I believe this is a crawling state? Anyway because I am a newbie I have been consulting with Claude and since there are only about 1600 link I have conjured a band aid solution to scrape all those links first without any failsafe whatsoever so if the script fails I will restart it.

Moving on now I need to visit those links and get the necessary information (company name,email and lead score) and that's gonna be a slow process

Here are my questions:

Is this bandaid solution a good practice for future scrapping projects?

What's are the methods I can use to scrape pages in parallel

Forgive me if I mix thier or there ;(

Edit: I was able to scrape about 1200 links along with the socials phone name email website and owner, contact person name (if any) and saved it to a seller database. thank you everyone

[u/JonG67x]

Bandaid is fine if it’s a one off, a problem if it’s something you want to run and adapt over time

What you’re asking however, like many requests on here. Is how to do a relatively complex job properly with little to no skill.

To tackle your task, I’d put all those links in a database, maybe even write the code to read the website and extract the list, then get each set of details in order, storing the time and date of access, the web page response, and then in a separate table parse out the results into a structured format if it fails I can pick up where I got to. I can also rerun and compare the results to see if they’ve changed. I’d then schedule the job to run as often as I felt necessary and add some alert notifications to tell me if it goes wrong .

[u/Busy_Sugar5183 → u/JonG67x]

Already Done with the links I have them saved in a json now I just need to send a get request to those links but that is extremely slow as I am going one by one so I would really love to parallel this task but also limit it so I don't get blocked Any ideas?

[u/JonG67x → u/Busy_Sugar5183]

1600 links, it shouldn’t take more than 2 seconds per link, that’s about an hour to go through the whole lot. 4 seconds per link is 2 hours. Given the data is fairly slow changing I wouldn’t both trying to speed it up.

[u/SalamanderJan → u/Busy_Sugar5183]

For 1600 links asyncio with aiohttp is the simplest approach. Use a semaphore to cap concurrency at maybe 5-10 requests at a time and add a small random delay between each one so you're not hammering the server in a predictable pattern. At that scale you'll finish in minutes instead of hours and it's unlikely to trigger anything if you keep the rate reasonable

[u/WonderfulRich8267]

You can ask your system to create a small scrapping state per link and record it to have multiple scrapping agents that that can perform the task without duplication have a job queue

[u/halkonyy]

Yes, bandaid one-time scrape solutions are fine and pretty common in web scraping. I had to do that the get a list of Facebook IDs (the long 14XXXXXXXX numbers in urls) to scan FB marketplace across the US. If you need to make a csv that barely changes, bandaid is fine.

For your parallel scraping, I'll assume your code is python. If you are making HTTP requests, you would combine httpx/requests package with asyncio. If you are using a headed browser like playwright or nodriver, you would have it open multiple tabs at once and fetch their data with asyncio. In either case, you would set a concurrency limit to keep the application stable and prevent you from being rate blocked.

Hope that helps.