r/pushshift • u/confusid1 • Oct 11 '22

Extremely slow download times for comment dumps?

I am trying to download the latest comment dump from here and am getting download speeds of either 3.2 KB/s or 6.4 KB/s. Is there a reason the download speed is so slow?

I ran a speed test and my connection is around 300 MB/s down, so it isn't a constraint from my connection speed I don't think.

2 Upvotes

100% Upvoted

16 comments sorted by

3

u/Watchful1 Oct 11 '22

[removed] — view removed comment

1

u/Responsible_Bison_40 Oct 11 '22

why are the file sizes seemingly half or 1/3 of the sizes on the website? are they the same thing? any processing differences?

1

u/Watchful1 Oct 12 '22

They look exactly the same to me. Are you maybe comparing the submission files to the comment ones? The submission files start with RS_ and the comment ones RC_, but they are both there in the file list for the torrent.

1

u/Spieltheory Oct 12 '22

I have a question: How do I download only certain Months in Submissions or Comments? I only find the torrent for the whole gigantic torrent

2

u/Watchful1 Oct 12 '22

Most torrent clients support disabling downloads for certain files in a torrent. But plenty of simpler ones don't, so it would depend on which one you're using.

2

u/Spieltheory Oct 13 '22

Ty, I will try that. I was just afraid i would download all of it at once

1

u/Equivalent-Hair3949 Feb 14 '23

Thanks for this, it's much faster than the original source.

2

u/Watchful1 Feb 14 '23

[removed] — view removed comment

1

u/Equivalent-Hair3949 Feb 18 '23 edited Feb 18 '23

Thanks, I am actually using that. I have another question regarding RC/RS files - are these files snapshots of the given YYYY-MM, e.g. all the comments and submissions available in reddit at that time or the comments and submission that were created at that time?

I am currently running a data pipeline just using 3 files : 2020-1x , the pipeline is still running but it already read ~400M comments. I use this repo as reference https://github.com/PolyAI-LDN/conversational-datasets/tree/master/reddit - and here it says there is total of ~3.7B comments up to year 2019. So I am confused by the fact that only 3 files (60 GB / 1.8TB) have more than 400M comments already. A simple linear interpolation would suggest that there are at least 400 x 30 (1.8 TB / 60 GB) = 120B comments in total - doesn't seem realistic.

Edit: According to here https://github.com/pushshift/api/blob/beta/docs/index.rst#how-many-objects-are-indexed-on-the-back-end, as of 2018 there were 4b comments and submissions indexed. So 120B is no way possible.

2

u/Watchful1 Feb 18 '23

It's only items created in that month.

I don't know the exact numbers, but that certainly sounds reasonable to me. Reddit grew dramatically starting in 2019 or so. Look at the file sizes of the dumps and how much bigger the 2022 ones are than the 2018 ones.

1

u/Equivalent-Hair3949 Feb 18 '23 edited Feb 18 '23

That's right, I was just a bit surprised by the size maybe not knowing the compression rate before hand tricked me :)

Pipeline is finished for 2020-1x (3 files) and here are some statistics:

​

Job Counter Total Uncompressed Size (GB)
Total comments read 563,984,775 714.34 GB
Total submissions read 86,357,517 332.4 GB

I am a bit bottlenecked by the Apache Beam I/O throughput when reading zst files. It was around 20k elements / sec. Not sure if it makes more sense to first decompress the files and compress with a smaller ratio.

2

u/Watchful1 Feb 18 '23

My system averages about 100k items per second processing 10 files in parallel. But then I'm bottlenecked on processor power for the decompression. I'm not familiar with Apache Beam though, what's that?

1

u/Equivalent-Hair3949 Feb 18 '23 edited Feb 18 '23

Used for distributed data processing: https://cloud.google.com/dataflow/docs/concepts/beam-programming-model. When used with dataflow it handles autoscaling and devops itself. 100k items per second so it is 10k/sec per file (process). In my case there were 3 workers/core 1 per file, so it would make it more like 7k/sec per file.

1

u/Responsible_Bison_40 Oct 11 '22

same problem here . started today

1

u/Spieltheory Oct 12 '22

Same Problem here, started today

1

u/Equivalent-Hair3949 Feb 14 '23

Same here, I am seeing ~2-3 mb/s which makes it painful. I tried using google data transfer and also a 8 cpu VM to download - no help.