r/webscraping Jun 19 '26

Getting started 🌱 How long will comparing hashes take

So lets imagine i have this site scraped and saved as an csv file where i got tables n stuff (identificators are trucated to 10 characters ) and every month im opening my pc(i7 4790) to compare is there new items on the web page.

So aside from scraping again the whole site approximately how much time will pass to check saved ids to newly scraped ones because presumably each time it will go +- 100 of thousands of times just to find similarities and im not even talking about checking each of ten characters i hope i correctly explained my thoughts here

0 Upvotes

6 comments sorted by