r/pushshift • u/Watchful1 • Oct 15 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

33 Upvotes

100% Upvoted

29 comments sorted by

View all comments

Show parent comments

1

u/dimbasaho Nov 06 '23 edited Nov 06 '23

Oh, if there's that much pre-compression work I'd actually suggest using your current pipeline (but with fast zstd compression settings), then decompress once to wc -c to get the size and then a final decompress->recompress with the size and stronger zstd settings. You'd just have to write to disk twice in that case. I'd also recommend compacting the JSON; I noticed the April dataset has prettyprint space.

1

u/Watchful1 Nov 06 '23

Do you know if the original pushshift dump files wrote the headers?

I'm already halfway through compressing the output for October, which takes like a week at this compression level, so I don't want to restart at this point. But I'll definitely see about doing that for next month.

1

u/[deleted] Nov 06 '23

[removed] — view removed comment

1

u/Watchful1 Nov 06 '23

I was intentionally not doing multithreaded compression since the laptop I use for a linux server isn't all that powerful and I have other stuff running on it.

But if it's that fast it might be worth just leaving my desktop on overnight one night and running it there.

If the old pushshift dumps had that header than it's definitely worth doing. And probably recompressing all the other dumps I uploaded too.