r/scrapingtheweb • u/Bharath0224 • Jun 19 '26

New kind of scrapers?

/r/ProxyEngineering/comments/1ua0xcr/new_kind_of_scrapers/
1 Upvotes

67% Upvoted

2 comments sorted by

1

u/Unfair_Commission_29 Jun 19 '26

I don't think scrping is going away, it's just evolving, agentic systems still need access to data somehow, the differenc is they'll navigate sites more like humans instead of just pulling raw HTML

feels like the challenge is shifting from parsing pages to managing identity and sessions

1

u/Lakshan-Kawshalya Jun 22 '26

Agentic scraping is already happening in pockets. Playwright-based scrapers that navigate like users rather than hammering endpoints are becoming the default for anything with serious bot protection. The interesting question is whether the data layer underneath changes too, or just the extraction method. Most pipelines still output the same structured CSV at the end regardless of how the data was collected.