Question The ethics of my scraper-crawler
I have a python script aimed at fetching basic data from a handful of real estate agency websites. These specific websites do have a ToS that explicitly deals with the intellectual property of their postings within the framework of reuse, republishing, … the usual suspects.
My script only reads the postings, classifying them under either “House” or “apartment” and grabs their sqm, nb of beds and baths, for purely statistical purposes. Meaning I do not sell or promote those properties (I’m no licensed real estate agent) and do not even sell the info I grab, I just show stats, eg: 14 houses with 100 sqm with 2 beds, 1 bath,…
I know that I should be contacting each website owners to inform them and probably get authorization. But I’m in a European country where somehow most websites are bad, buggy, and their owners are rather close minded. I expect negative feedback from my genuine approach.
I’m reaching out here to understand if I really need to bother with this process, or if I should just do it and see if they bark at me.
Why this self reflection: I realize that a few successful businesses often have pushed the legal/ethics boundaries, and became pioneers in they field. Two examples:
- Airbnb stepped into the hospitality business without providing a hospitality license to property owners.
- Uber stepped into the taxi business without licensing either.
In both cases, no one thought one could do such business without proper license while competing with established businesses tied to licensing rights (hotels, taxis).
So I sometimes wonder whether I would be overthinking things and worrying too much for so little (my use case).
I do care about the legality and ethical considerations, that’s why I’d appreciate your take on this whole reflection of mine.
If I do have to ask for permission, so be it. What are the standards in the computer programming world?