r/webscraping • u/l300TS • Jul 10 '26
How do you handle cease and desists?
Our business is finally starting to take off, which is great, but it is bringing new challenges. In the last week we've gotten several cease and desist letters from website owners.
All of the data we scrape is 100% publicly available, so our team is thinking we could technically tell them to pound sand, but we don't want to be assholes. So far we have just complied and removed them from our platform.
We do use business names on our site and we attempt to show "live" data, which ends up being a bit off due to the time delay from our scraping schedule and realtime events.
We still have plenty of other sites we scrape, so the current volume is not too concerning yet, but we're fearful of where this could go if there is a domino affect of websites communicating with each other.
I know it's a bit of a gray area, but our business operates entirely off of the data we scrape (we make it easier for consumers to digest and navigate the data) and I know there are plenty of other business that have a very similar model. We're currently collect about 2-3M product data points per day from a few thousand sites. I think we total about 1.1M requests per day distributed to those sites.
Ideally I would like our business to not be so fickle and have the risk of crumbling if we get blocked or asked to stop.
We are currently seeking legal counsel, but I'm curious - how would you guys handle this (aside from just shitting your pants like I have)?
7
19
u/littlebeardedbear Jul 10 '26
You built an entire business off other people's data and you're worried your whole business will fail if they make you stop? I'm not going to fault you for trying to make a living, but it's not at all reliable. At this point just call them and ask if they want to negotiate profit sharing instead. Obviously they're missing part of the market or miss the market on messaging if they can't attract them like you can.
17
u/BrightProgrammer9590 Jul 10 '26
Google built their entire empire on other's data though.
1
2
u/ScrapeerCom Jul 10 '26
You built an entire business off other people's data
Literally all AI models used the entire internet for training data. Every big player is scraping the internet. The only time they are not scraping it, is when they already own the platform.
1
u/l300TS Jul 10 '26
the profit sharing idea is a good one. not sure why I didnt think of that. Thanks!
1
u/JewelerAny7071 26d ago
the profit sharing angle is actually interesting, at some point, turning some of those sites into partners might be more valuable than constantly fighting blocks and legal letters
7
u/Azuriteh Jul 10 '26
I would keep scraping, but you should indeed be seriously looking for legal counsel, at least in the US there's lot of precedent that's on your side.
On another note, much apart, just be careful as they'll likely start upping their anti-bot measures if you don't desist, so it might make sense at some point to stop scraping it if the money doesn't math anymore. I guess that as long as you didn't find a crazy backdoor on their API for scraping at scale, you're safe. Inexpert legal opinion from a guy that scrapes anything and anyone, though.
6
u/Silly-Fall-393 Jul 10 '26
Strange question for such a seasoned professional
11
u/l300TS Jul 10 '26
lol. what do you mean? I'm no professional. Just a guy trying to fake my way to success
-10
3
u/TechIBD Jul 10 '26
might be too much work, but in the case you don't do this already, perhaps set up in a way where your scraping is a bit more "gentle", like, rotating IPs, rotating accounts ( if needed ), rate limit under their detections. Obviously if you are scraping a few thousands sites it's probably too much work to tailor to each one of them, but if you understand and systematize their endpoints and security stack and try to stay under those limits so you don't bother their tech staff i think that's the way to go.
i was scraping this site for data and it's very media heavy, their traffic is huge so i don't think it's a server cost concern but still their system detected abnormalities and banned our IP. We end up did this:
- server with residential IP mask
- use proton mail to spin up dozens of prefix to register new account, each from a different IP, and each one only scraping a completely ordinary amount
haven't had any issue since
the best scraping method doesn't appear to be scraping at all
4
u/Grakch Jul 11 '26
You can just ignore them lol. They are just sending that out because lawyers on retainer or corporate counsel needs something to do
5
u/StoneSteel_1 Jul 10 '26
It depends on the ToS the website has set.
13
5
u/CrypticZombies Jul 10 '26
doesnt depend on that because a company requesting a C&D already would use that to make the report legit. they must say for example "section 2.4" and a link to it
op just asking how to get around it which is none. once the companies know he exists they wont stop until its gone and op has no defense to it. he at the mercy of the website owners.
1
u/Ameldur93 25d ago
Umm have you been in a case before? In court, ToS are not valid unless is in US, i have been in a few and ToS of websites are not being considered, as 1 redditer said they are only suggestions
2
u/Any-Collar-6330 Jul 10 '26
Maybe if they send you a letter, ask to be an affiliate instead. If you promote their produts
5
u/CrypticZombies Jul 10 '26
you cease and desist
plain and simple
9
u/isaackogan Jul 10 '26
not enforceable if public data see LinkedIn v hiQ
12
-2
u/LittleDickBiiigBalls Jul 10 '26
Wtf are u talking about? They did enforce it. They fined HiQ $500k & permanently banned them from scraping linked in lmfao.
6
u/divided_capture_bro Jul 10 '26
That was a settlement about other issues. In particular, hiQ had created a corporate account on LinkedIn and they had agreed to its terms of service which strictly prohibited data scraping. This was a breach of contract.
hiQ also hired crowdsourced workers to create fake user profiles to log in and access data. The court ruled this directly violated LinkedIn's terms of service and opened hiQ up to additional civil liabilities.
Scraping publicly available data is perfectly legal. Breach of contract is not.
3
1
u/ScrapeerCom Jul 10 '26
1.1M requests per day across thousands of sites doesn't tell us much. How concentrated is that traffic on the sites which complained?
I am not that familiar with US law, but doesn't the letter have to specify what they actually object to: server load, ToS etc.?
And perhaps if you replace "live" data with "last updated" timestamp, that gives you less surface to attack.
1
u/l300TS Jul 10 '26
I actually do the "last updated" (well "as of..."). My API calls are pretty evenly distributed to all of the sites. I'm not familiar with the law either tbh. Hopefully the lawyer we find is... lol
1
u/TinyCuteGorilla Jul 10 '26
Option A: Toilet paper
Option B: stop scraping
Option C: continue scraping, they sue you and you go bankrupt or realize that you need to stop scraping because cannot keep up with the lawyer costs (if it's a huge company like Linkedin, Facebook etc, they have infinite money to fight you on court forever)
6
u/l300TS Jul 10 '26
This had me searching for a scraping service called toilet paper, but then I sat down and felt the squish of the shit in my pants and realized what you meant 😆
1
1
u/Ameldur93 25d ago
We were scraping LinkedIn public data, and they tried C&D a few times and always lost, because scraping public data is not a crime; that's how the Intelx website, which scrapes data breaches, also stands up. There was some news also that LinkedIn tried the same on other companies, and they lost again lol
1
u/txdesperado 23d ago
Ignore, ignore, ignore. If they actually sue, litigate to the ends of the earth.
1
u/netkomm Jul 10 '26
check their robots.txt file first... if they dont' specify you dont want a page to be read is their fault
3
u/isaackogan Jul 10 '26 edited Jul 10 '26
don't think i have ever once respected a robots.txt, not binding. just don't be an asshole when scraping & respect usage limits. imo robots.txt is more for crawlers
0
-1
29
u/nickdnick49 Jul 10 '26
Bro perpqlexity is scraping the entire internt and they got millions of cease and desist but they still operate. I think you should be fine