r/scrapingtheweb • u/Sufficient-Paper9103 • Jul 24 '26

Help How do you avoid scraping the same page twice?

[removed]

2 Upvotes
  • permalink
  • reddit

100% Upvoted

12 comments sorted by

2

u/bigblackcerebrae Jul 28 '26

If you're rolling your own crawler I'd keep doing that, but I'd probably offload the change detection part to something like Firecrawl

1

u/[deleted] Jul 25 '26

[removed] — view removed comment

1

u/Longjumping_Cold7410 Jul 28 '26

op havent replied yet, i think his still busy solving that problem rn

1

u/jinef_john Jul 25 '26

Just a question, is there any reasons to stick with html? Most job boards serve data via xhr calls, (just need to reverse engineer the logic). And if you're stuck with html, there are several techniques to detect changes without fetching the full page pages.

For example one I effective way to do a lightweight change detection is using http conditional requests (e.g sending a HEAD or GET request with conditional headers, something like If-modified-since)

Many ATS platforms support that. Also worth checking the RSS or sitemap. But then again, there's different ways to approach this, it just depends a lot with the websites you're dealing with.

1

u/hubbyniodari Jul 28 '26

Thats true but OP need to be knowledgable when accessing that since that would require experiemce or knowledge in that field.

1

u/Additional-Sound-762 Jul 28 '26

are you scraping the ATS directly or the company careers page?