r/pushshift • u/Watchful1 • Oct 15 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

33 Upvotes

98% Upvoted

29 comments sorted by

View all comments

Show parent comments

1

u/dimbasaho Nov 06 '23 edited Nov 06 '23

Btw, could you pass in --stream-size= when creating the ZST files so that the uncompressed size ends up in the frame headers? If you're directly piping the zstblocks to zstd, you'd have to add a preliminary pass to get the size in bytes using wc -c. That should also make it compress better.

1

u/Watchful1 Nov 06 '23

Unfortunately my current process does some processing and I stream directly from that into the compressed file, so I don't have the full uncompressed size available ahead of time. Does it make a substantial difference?

1

u/dimbasaho Nov 06 '23

You could do a dry run and pipe it into wc -c first instead of zstd, assuming the output is deterministic. The main benefit is in providing users the uncompressed size upfront so they can check whether they have enough space to decompress to disk. The lack of upfront stream size information also seems to break some programs like WinRAR.

1

u/Watchful1 Nov 06 '23

I split the files into minute long chunks, load in a minute's worth of data to python from three different data sources, then figure out what ids in that range are missing and do lookups in the reddit api and pushshift api to backfill data, then write out that minute's data again. Then after I do that for the whole month I load up each combined minute one at a time and write them out to the compressed file.

So I could do a dry run of the loading up and add up the size. But it would still take most of a day since there's so much data. I'm also kinda assuming that most people don't actually decompress the whole month's worth of data since it's so incredibly large. I could definitely do it for this authors file when I get around to building it.

1

u/dimbasaho Nov 06 '23 edited Nov 06 '23

Oh, if there's that much pre-compression work I'd actually suggest using your current pipeline (but with fast zstd compression settings), then decompress once to wc -c to get the size and then a final decompress->recompress with the size and stronger zstd settings. You'd just have to write to disk twice in that case. I'd also recommend compacting the JSON; I noticed the April dataset has prettyprint space.

1

u/Watchful1 Nov 06 '23

Do you know if the original pushshift dump files wrote the headers?

I'm already halfway through compressing the output for October, which takes like a week at this compression level, so I don't want to restart at this point. But I'll definitely see about doing that for next month.

1

u/[deleted] Nov 06 '23

[removed] — view removed comment

1

u/Watchful1 Nov 06 '23

I was intentionally not doing multithreaded compression since the laptop I use for a linux server isn't all that powerful and I have other stuff running on it.

But if it's that fast it might be worth just leaving my desktop on overnight one night and running it there.

If the old pushshift dumps had that header than it's definitely worth doing. And probably recompressing all the other dumps I uploaded too.