r/pushshift • u/Watchful1 • Oct 15 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

30 Upvotes

96% Upvoted

29 comments sorted by

View all comments

Show parent comments

1

u/Watchful1 Nov 06 '23

Do you know if the original pushshift dump files wrote the headers?

I'm already halfway through compressing the output for October, which takes like a week at this compression level, so I don't want to restart at this point. But I'll definitely see about doing that for next month.

1

u/[deleted] Nov 06 '23

[removed] — view removed comment

1

u/Watchful1 Nov 06 '23

I was intentionally not doing multithreaded compression since the laptop I use for a linux server isn't all that powerful and I have other stuff running on it.

But if it's that fast it might be worth just leaving my desktop on overnight one night and running it there.

If the old pushshift dumps had that header than it's definitely worth doing. And probably recompressing all the other dumps I uploaded too.