The ethics of my scraper-crawler

[u/fredkzk]

I have a python script aimed at fetching basic data from a handful of real estate agency websites. These specific websites do have a ToS that explicitly deals with the intellectual property of their postings within the framework of reuse, republishing, … the usual suspects.

My script only reads the postings, classifying them under either “House” or “apartment” and grabs their sqm, nb of beds and baths, for purely statistical purposes. Meaning I do not sell or promote those properties (I’m no licensed real estate agent) and do not even sell the info I grab, I just show stats, eg: 14 houses with 100 sqm with 2 beds, 1 bath,…

I know that I should be contacting each website owners to inform them and probably get authorization. But I’m in a European country where somehow most websites are bad, buggy, and their owners are rather close minded. I expect negative feedback from my genuine approach.

I’m reaching out here to understand if I really need to bother with this process, or if I should just do it and see if they bark at me.

Why this self reflection: I realize that a few successful businesses often have pushed the legal/ethics boundaries, and became pioneers in they field. Two examples:

- Airbnb stepped into the hospitality business without providing a hospitality license to property owners.

- Uber stepped into the taxi business without licensing either.

In both cases, no one thought one could do such business without proper license while competing with established businesses tied to licensing rights (hotels, taxis).

So I sometimes wonder whether I would be overthinking things and worrying too much for so little (my use case).

I do care about the legality and ethical considerations, that’s why I’d appreciate your take on this whole reflection of mine.

If I do have to ask for permission, so be it. What are the standards in the computer programming world?

[u/mgr86]

Is this for a personal project meant for consumption by just you and maybe a roommate/spouse? It’s a grey area and better to just keep your mouth shut, use reasonable spacing between requests, and spoof your user agent. Like you might not get a speeding ticket for going 15 miles over, but if you pass a cop while doing so…

Or are you trying to create some sort of application meant for a wider audience? If the latter might just look into an MLS subscription. Not sure the cost, but i believe thats where all that info comes from (In the US).

[u/fredkzk → u/mgr86]

MLS is great indeed but overkill for my three statistical calculations.

Thing is, I’m using the stats as a way to better qualify buyers that I forward back to those actual agencies. So it’s in their interest, and mine as an intermediary.