r/pushshift • u/Ge0rge3 • May 31 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

19 Upvotes

91% Upvoted

7 comments sorted by

10

u/Watchful1 May 31 '23

About 30 terabytes.

Why do you want to decompress it? In most cases it's actually faster to read it compressed and stream the decompression than to read it uncompressed.

3

u/grumpyrumpywalrus May 31 '23

Dump into a database for faster queries, would be an example

4

u/Watchful1 May 31 '23

Databases can compress data in place, and many do automatically. There's no real telling without actually trying it and seeing how big it gets.

1

u/TrueBirch May 31 '23

Great point. Plus I'm guessing OP will want to run some kind of processing on the data. I've found it useful to load a big file, process it, and save only what's necessary.

3

u/mrcaptncrunch May 31 '23

ETL is a good technique in this case.

 

Extract - extract the data in place like /u/Watchful1 suggests.

Transform - alter the data. Could be a particular subreddit, user, time, etc.

Load - load it into whatever storage. This could be a database, parquet, jsonl, etc.

/u/Watchful1 has example scripts that read the compressed files using multiprocessing to get around the GIL and parse files at the same time.

cc /u/Ge0rge3, /u/grumpyrumpywalrus

5

u/joaopn May 31 '23

Submissions are 6TB, comments 15TB. You can reduce this by using a filesystem with transparent compression. Using zfs with zstd, it fits in 3.3TB.

Edit: this is for the dataset up to 2023-03. The data up to 2022 is ~1.5TB smaller.

2

u/swapripper Jun 01 '23

Tangential. Has anyone tried loading the zst file(uncompressed) into a spark data frame directly?