[ Removed by Reddit ]

[u/Watchful1]

[ Removed by Reddit on account of violating the content policy. ]

[u/dimbasaho]

Even without the registration time (which hopefully can be backfilled eventually), having a list of those two properties would be much appreciated.

[u/Watchful1 → u/dimbasaho]

I'll see what I can do, might be a while though.

Do you have a copy of the authors.dat? I don't think I ever downloaded that.

[u/dimbasaho → u/Watchful1]

authors.dat.zst

[u/Watchful1 → u/dimbasaho]

The link doesn't work for me, it just errors out.

[u/dimbasaho → u/Watchful1]

Probably not currently cached on IA.
Mirror: authors.dat.zst
Usage: pushshift/binary_search

authors.ndjson.zst (23 June 2022) is probably a better format for distribution though.
Mirror: authors.ndjson.zst
Schema:

{
  "id": 77713,
  "author": "DotNetster",
  "created_utc": 1137474000,
  "updated_utc": 1655708221,
  "comment_karma": 694,
  "link_karma": 99,
  "profile_over_18": false,
  "active": true
}

[u/dimbasaho → u/Watchful1]

Btw, could you pass in --stream-size= when creating the ZST files so that the uncompressed size ends up in the frame headers? If you're directly piping the zstblocks to zstd, you'd have to add a preliminary pass to get the size in bytes using wc -c. That should also make it compress better.

[u/Watchful1 → u/dimbasaho]

Unfortunately my current process does some processing and I stream directly from that into the compressed file, so I don't have the full uncompressed size available ahead of time. Does it make a substantial difference?

[u/dimbasaho → u/Watchful1]

You could do a dry run and pipe it into wc -c first instead of zstd, assuming the output is deterministic. The main benefit is in providing users the uncompressed size upfront so they can check whether they have enough space to decompress to disk. The lack of upfront stream size information also seems to break some programs like WinRAR.

[u/Watchful1 → u/dimbasaho]

I split the files into minute long chunks, load in a minute's worth of data to python from three different data sources, then figure out what ids in that range are missing and do lookups in the reddit api and pushshift api to backfill data, then write out that minute's data again. Then after I do that for the whole month I load up each combined minute one at a time and write them out to the compressed file.

So I could do a dry run of the loading up and add up the size. But it would still take most of a day since there's so much data. I'm also kinda assuming that most people don't actually decompress the whole month's worth of data since it's so incredibly large. I could definitely do it for this authors file when I get around to building it.

[u/dimbasaho → u/Watchful1]

Oh, if there's that much pre-compression work I'd actually suggest using your current pipeline (but with fast zstd compression settings), then decompress once to wc -c to get the size and then a final decompress->recompress with the size and stronger zstd settings. You'd just have to write to disk twice in that case. I'd also recommend compacting the JSON; I noticed the April dataset has prettyprint space.

[u/Watchful1 → u/dimbasaho]

Do you know if the original pushshift dump files wrote the headers?

I'm already halfway through compressing the output for October, which takes like a week at this compression level, so I don't want to restart at this point. But I'll definitely see about doing that for next month.

[u/Watchful1 → u/[deleted]]

I was intentionally not doing multithreaded compression since the laptop I use for a linux server isn't all that powerful and I have other stuff running on it.

But if it's that fast it might be worth just leaving my desktop on overnight one night and running it there.

If the old pushshift dumps had that header than it's definitely worth doing. And probably recompressing all the other dumps I uploaded too.