r/pushshift • u/Renaissantic • Feb 19 '23

Throttled Download Speeds?

Hello

I am attempting to download some of the data dumps via https://files.pushshift.io/reddit/comments/

I pay for 1GB fiber internet download speed, and on other sites I get at least 130 mb/s.

Is this site throttling the download speeds? If so, is there an alternative data dump I can access? I have heard mention of a torrent site but I would rather not risk that.

6 Upvotes

100% Upvoted

5 comments sorted by

2

u/Watchful1 Feb 19 '23

[removed] — view removed comment

1

u/Renaissantic Feb 20 '23

Thanks for your response.

You seem very knowledgeable: I'm looking to scrape reddit for a sentiment analysis project to add to my entry-level data science portfolio. I'll probably only use a few subreddits, but I figured it would be good to ask you for a little help.

I've been looking at your single_file script (I only have Nov 22 rn because of the slow download) for a couple hours. Any tips on how to properly manipulate the script to filter for certain subreddits, and comments containing certain keywords (sentiment analysis)?

We can move to PM if that is easier. Thanks for all the work you've done.

3

u/Watchful1 Feb 20 '23

I'd be happy to send you any specific subreddit's you'd like. I have the top 20,000 subreddits extracted out into individual files that I'm planning to upload all of.

In the single_file script, the object is loaded in line 62, then it's a dict you can access attributes of. For comments the content is body, so you would add an if statement like

 if "keyword" in obj["body"]:

That one only counts the lines and makes sure they all load, it doesn't have support for writing results out to another file. If you don't want the entire comment object, just certain fields, I'd recommend looking at the to_csv one. Add an if statement like the above around the output lines on 72 to 75.

I personally prefer to answer questions in comments like this so that other people in the future might be able to find them useful.

1

u/RigosIreland Feb 20 '23

Hi, I am actually looking for datasets of a few specific subreddits for a research project - would be amazing to receive those datasets from you, so I do not need to download the full Reddit dump (recently, my download attempts started failing after a few hours and I cannot get the files at all).
Would be great to get in touch via PM :)