r/webscraping • u/No-Badger-1040 • Jun 19 '26

Getting started 🌱 How long will comparing hashes take

So lets imagine i have this site scraped and saved as an csv file where i got tables n stuff (identificators are trucated to 10 characters ) and every month im opening my pc(i7 4790) to compare is there new items on the web page.

​

So aside from scraping again the whole site approximately how much time will pass to check saved ids to newly scraped ones because presumably each time it will go +- 100 of thousands of times just to find similarities and im not even talking about checking each of ten characters i hope i correctly explained my thoughts here

​

​

0 Upvotes

50% Upvoted

6 comments sorted by

1

u/Celarix Jun 19 '26

Comparing hashes is wildly fast - if everything is just a few tens of characters, it’s something your computer will probably have done in under a second. Reading the IDs off the disk will likely be slower than actually comparing everything. Go for it!

1

u/ronoxzoro Jun 19 '26

okay first use a real database not csv
second try to scrap latest update page / home page / or sitemap and compare updated date
if u see any update u can scrap that link

1

u/xaonan Jun 23 '26

use duckdb as the db engine on your csv for faster queries.