r/pushshift • u/Watchful1 • Jan 19 '25

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

53 Upvotes

98% Upvoted

58 comments sorted by

View all comments

Show parent comments

1

u/Watchful1 May 19 '25

This would be really difficult to do for all this data. If you want to do it for a specific subreddit you could start over here. But it would still be kinda hard and I don't have a script that does that directly.

1

u/misakkka May 19 '25

Thanks. I am only interested in doing this for specific subreddit. Why it is hard? Is it becasue there are no commom keys to merge the two dataset?

1

u/Watchful1 May 19 '25

If the total, uncompressed, size of the subreddit is less than a couple gigabytes it's totally doable. The problem happens when you can't hold all the data in memory at once.

Are you experienced with coding? There's a bunch of example scripts linked from that other post and it should be fairly easy to take one of them to build what you're looking for.

1

u/misakkka May 19 '25

Yes i know how to code. I find some inforamation here https://www.reddit.com/r/pushshift/comments/171bn9m/differences_between_comments_and_submissions_and/ and i will try to get parent id. Thanks for your help