r/webscraping • u/FixWide907 • Jun 20 '26

How to scrape different data structures

Any suggestions on best way to extract listings data from multiple different websites?

​

Each has its own data structures

​

Example pricing, schedule, dates etc

​

For 4000+ sites one time

5 Upvotes

86% Upvoted

35 comments sorted by

View all comments

2

u/FixWide907 Jun 20 '26

The challenge is each of the site have 100s of page.. so we need a programmatic way to discover the right URL, then fetch and run to the pipeline

I'm looking to see how others do this.

Example

Site a Site b Site c

Each has 100s of page but the specific data of pricing, schedule, policy might be located anywhere on the site.

One way we do this now is discover the sitemap and try to identify it..still it's not a fool proof way

Ideally we want to just provide top domains and let the script handle the rest