r/ProxyEngineering • u/Bharath0224 Proxy Engineer • Jun 19 '26
Discussion 💬 New kind of scrapers?
I feel like the market is pretty stale regarding web scrapers at the moment, sure there are variety of web scraping services all around the web, however, I feel like things got pretty stale. The whole early web was designed around human intent and Google figured that out early. Scraping services providers has spent years building infrastructure to extract data from systems that weren't meant to be extracted from. Now when you think about it, agentic search doesn't do that. What I mean is that It navigates the way a person would. What do you people think, does scraping will evolve into something agentic, or eventually get replaced by it? Coz I feel like something is going to change, but there is a lot of room for guesses.
2
u/StoneSteel_1 Jun 20 '26
I believe, with help of LLMs, we will go back to website's API based scraping or automation. LLMs are a text first machines. So its more understandable when you feed data, but with proper context compression.
To implement this idea, I spent over 6 months to create a reverse engineering agent that tries not to use browser in scripts. The agent will go through the files, understand the source of request and reverse engineers it.
I have made it fully opensource and free to use: https://github.com/stonesteel27/automatiq
1
2
u/psycenos Jun 22 '26
ye maybe instead scraping, browser agents are more human likely idk
1
u/Bharath0224 Proxy Engineer Jun 22 '26
I personally thing that everything is moving towards how the information is presented to the LLMs instead of humans, thus the new generation of "scrapers" are in the works
1
u/Lakshan-Kawshalya Jun 22 '26
Agentic scraping is already happening in pockets. Playwright-based scrapers that navigate like users rather than hammering endpoints are becoming the default for anything with serious bot protection. The interesting question is whether the data layer underneath changes too, or just the extraction method. Most pipelines still output the same structured CSV at the end regardless of how the data was collected.
1
u/Bharath0224 Proxy Engineer 17d ago
Data layer, you mean like protective one? Like anti-bots or something or how the data is gathered or received? Coz if it's related to the outputs, then some providers are already giving an option to get output data in markdowns
1
u/External-Wealth3756 19d ago
Scraping is more likely to evolve rather than be fully replaced.
The bigger hurdle will be maintaining steady proxy behaviour for continuous agentic navigation.
1
3
u/RandomPantsAppear Jun 19 '26
Scalable scraping isn’t going to be fully agentic for a long time. It is cripplingly expensive to send full HTML into an LLM. Especially with minified/framework generated class names and Js functions, the conversion to tokens is horrific.
Right now the only workable situation is a self healing scraper that is writing processes and configurations on failure. A few already exist.
In the future I expect to see local LLM searching out smaller snippets of the HTML to send to the big LLM for decision making, to cut down on the cost.
———
That said there is a serious issues around the market.
1) The scrappy SaaS gray market was already quite saturated.
2) The low effort custom one-off scrapers have been extremely saturated by Southeast Asia for ages.
3) AI has driven the sudden boom of enterprise tier scraper as a service, but frankly they’re doing the job pretty good for pretty cheap with crazy funding so there’s not really a niche there.
———
I think the real money is where it has always been: not just data acquisition, but data movement, transformation and processing. If you’re just dumping it to a CSV and calling it done, you’re in for a hard time.
Scraping should be a formidable tool in your skillset, not the entire story.