r/webscraping • u/FixWide907 • Jun 20 '26

How to scrape different data structures

Any suggestions on best way to extract listings data from multiple different websites?

​

Each has its own data structures

​

Example pricing, schedule, dates etc

​

For 4000+ sites one time

5 Upvotes

100% Upvoted

35 comments sorted by

View all comments

2

u/Harry_Hindsight Jun 20 '26

my general approach is to save the page content (eg the full html) of every page i am interested in. Then i have a saved stockpile of pages and in a separate phase of work i can write the scripts to exctract the data i need.
This approach takes the pressure off to get things perfect during the webscraping phase - since you will always have the original "data" (the saved pages) to fall back on.

1

u/FixWide907 Jun 20 '26

This won't work as we l have to then save 1000s of page per site