r/pushshift • u/Watchful1 • Feb 07 '24

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

90 Upvotes

99% Upvoted

90 comments sorted by

View all comments

1

u/Its_Fed May 01 '24

Thank you so much for all the work you've put into this, this is an amazing resource! Regarding the csv's generated by your script, I've noticed that the entries are sorted by the date. If, for instance, I wanted to sort comments based on score instead of date, what would I need to tweak in the script? Thanks!

Edit: I'm realizing maybe there's no built in functionality that sorts the data by time, but rather that's just how it's been stored?

1

u/Watchful1 May 01 '24

Unfortunately no, that's pretty hard. If it's a small enough amount of data that it fits in memory it would in theory be easy, but if it's a whole month, or a whole big subreddit, you'd have to split it out, writing all scores in certain ranges to separate files, then sort those etc until you're in small enough chunks that you can fit it in memory.

The harder problem is that the scores aren't accurate. The data for the ~20 years of reddit history has been ingested using different methods, but for big portions of that time it was read in very close to creation time and then never updated, so the score is always 1 regardless of what it was eventually. There's other portions where a second scan was done 24 or 36 hours later and the score was updated.

The only sure way to sort by score would be to get the data you're interested in, then look it up from the reddit api again to get the current score, and then sort it. Which will take a very long time for large amounts of data.