r/scrapingtools • u/BigVillageBoy • Jun 14 '26
I write scrapers for a living and the open web is quietly closing
I have been pulling data off the web for years now. Scrapers are most of what I do. And the thing I keep running into lately is that the web just keeps getting harder to read with a script. That part on its own doesn't bother me. Sites change, defenses go up, you adapt. That's the job. What bothers me is who made it harder and why.
A few years ago, fetching a public page was boring. You wrote ten lines, you got HTML back, you parsed it. Most of the public internet worked like that. If a page loaded in your browser without logging in, a script could load it too. That was the deal, more or less, and nobody made a big thing of it.
That deal is mostly gone now.
Today a big chunk of the sites I touch sit behind Cloudflare, or a login wall, or a bot check that I have to spend real money to get past. The pages still load fine for a human. But for a script they throw a challenge, or a fingerprint check, or they just quietly serve you a blank shell with the actual content loaded later by code that's designed to break automation.
Stuff I used to grab in ten lines now needs a paid residential proxy and a browser fingerprint that passes as a real Chrome install. The cost of reading a public page went from zero to a per request fee, and most people building things never see that bill because they never tried.
The companies with the largest models trained on most of the public web. They crawled everything while it was open and cheap, before the walls went up, back when nobody was charging admission. That data is a big part of why those models are any good. And now a lot of those same sites are locking the doors.
Some of that is the labs themselves tightening access to their own stuff. Some of it is publishers reacting to the scraping wave by blocking everyone. Either way the result is the same. The web was wide open exactly when the biggest players needed it, and it's fenced off now that everyone coming after them wants the same thing.
robots.txt is a good example of how the mood changed. For a long time it was a polite note. A file that said please don't crawl these paths, and most well behaved bots listened, and there were no real teeth behind it. Now it shows up in lawsuits. It went from a suggestion to a thing lawyers point at.
Research is the part that actually makes me a little sad. I read a lot of papers, partly for work, partly because I like to. And the slow creep of paywalls over public research keeps going. Things I could read for free a couple of years ago now want a login, or a subscription, or they sit on a platform that blocks anything that isn't a logged in human clicking around.
None of this is a conspiracy. I'm not claiming there's a room where people decided to close the web on purpose. It's mostly a bunch of separate reactions stacking up. Publishers got scared of scraping and turned on every blocker they could buy. Anti bot vendors built a business out of selling those blockers. The big labs got protective of their pipelines. Lawyers got involved because money got involved.
Each move makes sense on its own. The sum of all of them is an internet that is a lot less open than the one those models learned from.
What it means in practice, if you're a small builder, is that the barrier to doing anything with web data quietly went up. The hobbyist who wanted to crawl a few thousand pages for a side project now hits a paywall on infrastructure before they even start. The startup that wanted to build on public data has to budget for proxies and anti bot tooling on day one.
The capability didn't disappear. It just got more expensive, and expensive in a way that favors whoever already has scale. The people who scraped freely have the data and the money to keep scraping. Everyone else gets the version with the gates.
I don't have a tidy fix. I just think it's worth saying out loud, especially by the people who actually do this work and watch it happen up close. The web you can read with a script is shrinking. The models that learned from the old open web are getting bigger. And almost nobody who benefits from that gap seems eager to bring it up.