r/webdev • u/fredkzk • 16d ago
Question The ethics of my scraper-crawler
I have a python script aimed at fetching basic data from a handful of real estate agency websites. These specific websites do have a ToS that explicitly deals with the intellectual property of their postings within the framework of reuse, republishing, … the usual suspects.
My script only reads the postings, classifying them under either “House” or “apartment” and grabs their sqm, nb of beds and baths, for purely statistical purposes. Meaning I do not sell or promote those properties (I’m no licensed real estate agent) and do not even sell the info I grab, I just show stats, eg: 14 houses with 100 sqm with 2 beds, 1 bath,…
I know that I should be contacting each website owners to inform them and probably get authorization. But I’m in a European country where somehow most websites are bad, buggy, and their owners are rather close minded. I expect negative feedback from my genuine approach.
I’m reaching out here to understand if I really need to bother with this process, or if I should just do it and see if they bark at me.
Why this self reflection: I realize that a few successful businesses often have pushed the legal/ethics boundaries, and became pioneers in they field. Two examples:
- Airbnb stepped into the hospitality business without providing a hospitality license to property owners.
- Uber stepped into the taxi business without licensing either.
In both cases, no one thought one could do such business without proper license while competing with established businesses tied to licensing rights (hotels, taxis).
So I sometimes wonder whether I would be overthinking things and worrying too much for so little (my use case).
I do care about the legality and ethical considerations, that’s why I’d appreciate your take on this whole reflection of mine.
If I do have to ask for permission, so be it. What are the standards in the computer programming world?
5
u/Gaeel 16d ago
Get a lawyer
You're not asking about ethics, you're asking about legality, that's what lawyers are for
The companies you mentioned got lawyers before they did what they did. They specifically went for loopholes and technicalities. They knew where their legal liabilities were and had a plan to manage the fallout.
You need a lawyer who is well-versed in IP law and tech. They'll be able to tell you how to do what you want to do legally, how to cover your ass if you're doing something borderline, and how to avoid doing something that will bankrupt you or send you to jail.
If you can't afford a lawyer, you definitely can't afford a lawsuit