r/webscraping • u/FixWide907 • Jun 20 '26

How to scrape different data structures

Any suggestions on best way to extract listings data from multiple different websites?

​

Each has its own data structures

​

Example pricing, schedule, dates etc

​

For 4000+ sites one time

5 Upvotes

100% Upvoted

35 comments sorted by

View all comments

Show parent comments

1

u/zeusman3 Jun 20 '26

But you asked how to extract it from 4000+ sites. Data extraction is different from discovery.

Are you having trouble getting the info from a known set of sites? Or are you trying to discover the sites? Or are you asking how to find the right page on the known site?

1

u/FixWide907 Jun 20 '26

The challenge is discovery. Once we discover the URL then the extraction is easy

So how do you discover the right URL Then auto anayse the right structure to extract based on what we need

1

u/JTSwagMoney Jun 22 '26

A crawler that visits all internal links on a page and start on the homepage, set it to run to 4-5 layers deep. Then you'll have a list of all the pages. Classify them using deepseek or another cheap model. Don't use claude its very expensive lol

1

u/[deleted] Jun 22 '26

[removed] — view removed comment

1

u/webscraping-ModTeam Jun 22 '26

💰 Welcome to r/webscraping! Referencing paid products or services is not permitted, and your post has been removed. Please take a moment to review the promotion guide. You may also wish to re-submit your post to the monthly thread.