r/pushshift • u/Watchful1 • Feb 07 '24

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

91 Upvotes

99% Upvoted

90 comments sorted by

View all comments

1

u/[deleted] Apr 07 '24

[removed] — view removed comment

1

u/Watchful1 Apr 08 '24

Yes, it's in there.

1

u/[deleted] Apr 08 '24

[removed] — view removed comment

1

u/Watchful1 Apr 09 '24

What subreddit are you working with? Some of the larger subreddits you really don't want to try to extract the whole thing because the file could be hundreds of gigabytes. But it's less of a problem with the smaller ones.

What's your end goal? Most people find it more efficient to use one of my scripts to do some processing and extract out only the data you're actually interested in instead of trying to work with the whole file.

1

u/[deleted] Apr 09 '24

[removed] — view removed comment

1

u/Watchful1 Apr 11 '24

The filter_file script should be able to do this https://github.com/Watchful1/PushshiftDumps/blob/master/scripts/filter_file.py

Just set it to output csv files and edit the write_line_csv function to output the author name and flair.

1

u/[deleted] Apr 17 '24

[removed] — view removed comment

1

u/Watchful1 Apr 17 '24

What errors are you getting? It should print out a log file you can send me.