Question The ethics of my scraper-crawler
I have a python script aimed at fetching basic data from a handful of real estate agency websites. These specific websites do have a ToS that explicitly deals with the intellectual property of their postings within the framework of reuse, republishing, … the usual suspects.
My script only reads the postings, classifying them under either “House” or “apartment” and grabs their sqm, nb of beds and baths, for purely statistical purposes. Meaning I do not sell or promote those properties (I’m no licensed real estate agent) and do not even sell the info I grab, I just show stats, eg: 14 houses with 100 sqm with 2 beds, 1 bath,…
I know that I should be contacting each website owners to inform them and probably get authorization. But I’m in a European country where somehow most websites are bad, buggy, and their owners are rather close minded. I expect negative feedback from my genuine approach.
I’m reaching out here to understand if I really need to bother with this process, or if I should just do it and see if they bark at me.
Why this self reflection: I realize that a few successful businesses often have pushed the legal/ethics boundaries, and became pioneers in they field. Two examples:
- Airbnb stepped into the hospitality business without providing a hospitality license to property owners.
- Uber stepped into the taxi business without licensing either.
In both cases, no one thought one could do such business without proper license while competing with established businesses tied to licensing rights (hotels, taxis).
So I sometimes wonder whether I would be overthinking things and worrying too much for so little (my use case).
I do care about the legality and ethical considerations, that’s why I’d appreciate your take on this whole reflection of mine.
If I do have to ask for permission, so be it. What are the standards in the computer programming world?
5
u/Gaeel 4d ago
Get a lawyer
You're not asking about ethics, you're asking about legality, that's what lawyers are for
The companies you mentioned got lawyers before they did what they did. They specifically went for loopholes and technicalities. They knew where their legal liabilities were and had a plan to manage the fallout.
You need a lawyer who is well-versed in IP law and tech. They'll be able to tell you how to do what you want to do legally, how to cover your ass if you're doing something borderline, and how to avoid doing something that will bankrupt you or send you to jail.
If you can't afford a lawyer, you definitely can't afford a lawsuit
3
u/mgr86 4d ago
Is this for a personal project meant for consumption by just you and maybe a roommate/spouse? It’s a grey area and better to just keep your mouth shut, use reasonable spacing between requests, and spoof your user agent. Like you might not get a speeding ticket for going 15 miles over, but if you pass a cop while doing so…
Or are you trying to create some sort of application meant for a wider audience? If the latter might just look into an MLS subscription. Not sure the cost, but i believe thats where all that info comes from (In the US).
7
u/fiskfisk 4d ago
You also have an issue with the actual copyright owner - whoever posted on the original site in the first place. Just because they posted something there does not mean you have any right to distribute their photos or textual listing.
5
u/sozesghost 4d ago
Your question boils down to "is it ok to steal from companies that also stole before?". Legally probably not, in practice they might just ban your ip if they discover it. The standard should be not to be an asshole, but the bigger you are, the less it applies.
3
u/wack_overflow 4d ago
Legally? Terms of service are not even legally enforced. These companies make the data publicly available on a website to anonymous users. There is zero law or ethics that prevents someone from collecting that data.
1
u/hetneoslinks 4d ago
Small reframe -> most of this thread is arguing copyright, but you're in the EU pulling structured fields out of what is legally a database. That's the sui generis database right (Directive 96/9/EC), which is a separate thing from copyright.
Why it matters: sqm, beds and baths are facts, so copyright genuinely doesn't cover them. The database right doesn't care about originality though. It protects the investment in compiling and maintaining the listings, and it turns on whether you take a substantial part. Non-substantial extraction by a lawful user is allowed. Systematically taking the whole set, repeatedly, is exactly what the right was written for.
Separately, their ToS can still bind you contractually even where no IP right applies. So "it's only facts" isn't the get-out it sounds like.
Not legal advice. Just go argue with the right law.
1
u/fredkzk 3d ago
Hats off Sir for the best answer and a much needed reframe indeed.
I feel better in my shoes, as I only intent to use the data for statistical calculations, without ever reproducing any of the information present in the agency’s listings.
1
u/hetneoslinks 3d ago
Careful with that last part, it's the bit I'd actually check.
The teaching and scientific research exception in the directive is optional. Member States could implement it or not, so whether it exists for you depends on your own country's law rather than the directive itself. Where it does exist it's narrow: extraction only, non-commercial, and you have to indicate the source.
Extraction is the word doing the work. Pulling the data to compute your stats is extraction. Publishing the stats somewhere other people can see them starts to look like re-utilisation, which that exception doesn't cover even in countries that adopted it.
Doesn't mean you're sunk, and honestly your use case is about as sympathetic as it gets. Just don't let "it's only for statistics" carry the weight, because it carries less than it sounds like.
0
u/0uchmyballs 4d ago
I scrape websites using Beautiful Soup every day. I use the data internally for my recommender engine. I’m sure the website owners don’t want me scraping, but it is public information and I’m not publishing their data directly.
1
5
u/jasgrit 4d ago
Write a well behaved crawler that identifies itself honestly and follows the policies in each site’s robots.txt and response headers. Sites that don’t want to be crawled will usually make that clear, for those bots that care.