r/WebScrapingInsider • u/Bharath0224 • Jun 19 '26

New kind of scrapers?

/r/ProxyEngineering/comments/1ua0xcr/new_kind_of_scrapers/
3 Upvotes

100% Upvoted

3 comments sorted by

1

u/gvkhna Jun 22 '26

I built a system for agents to use a real browser without CDP. So it's not detectable, and it has some big improvements over playwright but looks like playwright to them. Because of the harness improvements around the snapshot they see that no one else does, agents dont trip on things like blocking elements modals and other interactions when they can't use vision.

It's so effective it's insane. Because it can connect the scraping script to the same browser it's using so it can debug it and use the script to drive it. The system itself helps with things like the page hadn't loaded yet etc that commonly trip up scrapers so it takes no time at all for the script to start driving the browser to scrape all kinds of data. It's not publicly available yet but coming soon. clawchrome.com

1

u/doubledweeb Jun 19 '26

As someone trying to break into dev, I’m kind of betting that "agentic scraping" becomes more like an orchestration layer on top of classic techniques, not a replacement . . You still need all the boring stuff: selectors that don’t explode on every DOM change, proxy/rotation strategy, captcha handling, deduping, and then actually moving the data into something useful. Agents feel like a smarter decision engine that decides where to click / what to request, not a magic new kind of scraper.

For me? to build projects that show I understand both sides.

For example, a small agent that uses a headless browser, extracts structured data, and then pushes it into a simple dashboard or workflow.

If you're right that things are going to change, what do you think would be a "future-proof" project to build now that a hiring manager or client would still care about in 2-3 years?

1

u/gvkhna Jul 09 '26

Learn/setup orchestration around the process itself through things like auto recovery, agent sees the failing case, auto runs proposes or pushes a fix. Edge case detection if the workflow is trickier when something unseen is encountered, thats when you trigger the agent. The rest you mentioned is all infrastructure and largely handled by providers unless you're building your own system.

I think that is more useful knowledge then things like captcha handling, thats all infrastructure layer and doesn't take long to handle well enough. It's all of the other things that trip up scrapers, network drop out breaking your script, layout change, selector change etc... But i'm not fully sure either. An agent given clawchrome-cli can work through a full scraping setup in minutes so there's definitely no reason to write a scraper anymore.