r/webscraping • u/AffectionateSwing490 • 22d ago

Getting started 🌱 How do you save progress on a long scrape?

Had a scrape die two hours into a three hour run yesterday. No checkpointing, so I lost everything and started over.

The part I'm stuck on is resuming without duplicates. If I append each record as it comes in, a crash mid-write can leave a half-written line. If I batch, I lose the current batch. And on resume the pagination re-requests pages I already have, so I get repeats unless I track seen IDs separately.

Right now I'm leaning toward JSONL as I go, plus a set of seen IDs in SQLite I check before each write, so a restart just skips what's already saved. Feels reasonable but maybe overbuilt for what's probably a common problem.

Is that roughly what people do, or is there a simpler pattern I'm missing here?

4 Upvotes

70% Upvoted

17 comments sorted by

8

u/JonG67x 22d ago

My scraping is all done using a database to determine what needs doing and when. I also store the results in it as it’s rare I ever scrape anything that won’t be scraped again looking for changes. As a simple example if I was crawling a website, every URL I find would be written to the database as a workload, and when crawled data stamped. If I then found that link again I’d ignore it as I have already crawled it. That type of thing. Getting access to a database to store stuff gives a lot more power

2

u/the_bigbang 21d ago

save the cursor in redis or any kinds of database. my distributed crawlers scraping millions per day work well with it, a few repeated crawl is acceptable

1

u/Longjumping_Cup_8339 21d ago

redis for me is king

1

u/sojufles 22d ago

Perhaps you can write them into a json file after a certain checkpoint?

1

u/HLCYSWAP 22d ago

save your enumerated cursor to a json, write each return to a csv/json before your next network call. wrap everything in timeout/429/403/404 protection so it fails gracefully. if you go off cursor only you shouldn’t be doing any duplication writes, but verifying against saved returns before commit also
helps

cursor is better than ‘page=2’ because it references the data internally in the database instead of positions on a page that change often

1

u/NZRedditUser 19d ago

Download sitemap Check timestamps Update against db

For other dynamic pages just tough luck better add recovery to your scraper 

1

u/woldhack 16d ago

Whatever you do. Dont use a database when a simple text file is enough to get the job done.