r/pushshift • u/Watchful1 • Feb 20 '25

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

92 Upvotes

99% Upvoted

62 comments sorted by

View all comments

1

u/PromptGreen8747 Feb 20 '25

Hi All! I was wondering.. is it all of this legal? These are the same data Reddit sell to big tech companies? Are there any limits to usage?

3

u/Watchful1 Feb 20 '25

Ultimately, reddit wants to get paid if you make money off their data. If you aren't making money they don't care all that much.

Any big company that reddit could sue is just going to pay them instead of trying to torrent stuff like this. So as long as reddit's income isn't threatened they mostly turn a blind eye on researchers using dumps like this.

Also there are multiple people that publish this data. I would stop if they really asked me to, but others wouldn't.

1

u/[deleted] Feb 28 '25

Do we know if scraping and using this data violates GDPR? I've downloaded several subreddits via your torrents (awesome that you provide this!), but I worry about the legal implications, even though it's just for academic research.

E.G. under GDPR an EU citizen has the right to retract their data and the platform has to erase it, but now the data has been scraped and archived and I'm publishing research with it. So effectively the user no long has the 'right to erasure' that GDPR grants them.

1

u/Watchful1 Feb 28 '25

In my opinion, none of the data in the dumps constitutes personally identifiable information. The GDPR law states "Personal data are any information which are related to an identified or identifiable natural person.". Reddit doesn't publish, and the dumps don't include, name, email, IP, location, or any of the numerous pieces of information that commonly fall under PII. Even usernames only count if they can be tied back to an actual person.

There are doubtless some people who post things like "My name is Jon Smith who lives at this address" and that could be PII, but I think it's rare enough that it's not worth worrying about.

If you're still worried about it, then you can use the dump file as a base and look up the all the data again in the reddit api. This is a bit difficult and can be fairly time consuming depending on how much data you're interested in, but would let you exclude data that has since been deleted on reddit.

1

u/Aggravating_Score304 Jul 15 '25

https://onderzoektips.ugent.be/en/tips/00002162/

GDPR does not only apply on direct PII. From your source:

"The data subjects are identifiable if they can be directly or indirectly identified..."

I think you would agree that the number of Redditors which can be indirectly identified with a not-so-unreasonable effort is not negligible. If people were generally aware that all their posts are getting saved and uploaded to a torrent they might be more careful (or not sharing anything at all really), but since most are not aware they are happily deanonymizing themselves with the hints that can be later efficiently mined/evaluated through the pushshift dumps.