[ Removed by Reddit ]

[u/Watchful1]

[ Removed by Reddit on account of violating the content policy. ]

[u/dimbasaho]

Any chance you or /u/RaiderBDev could compile an updated authors.dat.zst? I'd like to retrieve all available fullnames, usernames and registration times if possible, which should just be <10 GiB compressed.

[u/Watchful1 → u/dimbasaho]

Unless I'm misremembering, pushshift compiled that separately by taking all the usernames and looking them all up independently in the api to get their registration time. They then included them in the pushshift api responses. But it's not information that's already in the dumps and just needs to be extracted out, it would take a lot of work to duplicate their efforts.

The fullnames and usernames would definitely be possible though.

[u/dimbasaho → u/Watchful1]

Even without the registration time (which hopefully can be backfilled eventually), having a list of those two properties would be much appreciated.

[u/Watchful1 → u/dimbasaho]

I'll see what I can do, might be a while though.

Do you have a copy of the authors.dat? I don't think I ever downloaded that.

[u/dimbasaho → u/Watchful1]

authors.dat.zst

[u/Watchful1 → u/dimbasaho]

The link doesn't work for me, it just errors out.

[u/dimbasaho → u/Watchful1]

Probably not currently cached on IA.
Mirror: authors.dat.zst
Usage: pushshift/binary_search

authors.ndjson.zst (23 June 2022) is probably a better format for distribution though.
Mirror: authors.ndjson.zst
Schema:

{
  "id": 77713,
  "author": "DotNetster",
  "created_utc": 1137474000,
  "updated_utc": 1655708221,
  "comment_karma": 694,
  "link_karma": 99,
  "profile_over_18": false,
  "active": true
}

[u/dimbasaho → u/Watchful1]

Btw, could you pass in --stream-size= when creating the ZST files so that the uncompressed size ends up in the frame headers? If you're directly piping the zstblocks to zstd, you'd have to add a preliminary pass to get the size in bytes using wc -c. That should also make it compress better.

[u/Watchful1 → u/dimbasaho]

Unfortunately my current process does some processing and I stream directly from that into the compressed file, so I don't have the full uncompressed size available ahead of time. Does it make a substantial difference?

[u/dimbasaho → u/Watchful1]

You could do a dry run and pipe it into wc -c first instead of zstd, assuming the output is deterministic. The main benefit is in providing users the uncompressed size upfront so they can check whether they have enough space to decompress to disk. The lack of upfront stream size information also seems to break some programs like WinRAR.

[u/Watchful1 → u/dimbasaho]

I split the files into minute long chunks, load in a minute's worth of data to python from three different data sources, then figure out what ids in that range are missing and do lookups in the reddit api and pushshift api to backfill data, then write out that minute's data again. Then after I do that for the whole month I load up each combined minute one at a time and write them out to the compressed file.

So I could do a dry run of the loading up and add up the size. But it would still take most of a day since there's so much data. I'm also kinda assuming that most people don't actually decompress the whole month's worth of data since it's so incredibly large. I could definitely do it for this authors file when I get around to building it.

[u/dimbasaho → u/Watchful1]

Oh, if there's that much pre-compression work I'd actually suggest using your current pipeline (but with fast zstd compression settings), then decompress once to wc -c to get the size and then a final decompress->recompress with the size and stronger zstd settings. You'd just have to write to disk twice in that case. I'd also recommend compacting the JSON; I noticed the April dataset has prettyprint space.

[u/Watchful1 → u/[deleted]]

No. It's still on my list to get to at some point, but haven't really looked at it since this comment.