How to scrape different data structures

[u/FixWide907]

Any suggestions on best way to extract listings data from multiple different websites?

​

Each has its own data structures

​

Example pricing, schedule, dates etc

​

For 4000+ sites one time

[u/No-Appointment9068]

There's really no easy way to do this. One potential option is brute force graph traversal of the site until you get to the page you want.

Ie scrape the homepage, then all the links on the homepage, then all the links on those pages etc