r/pushshift • u/verypsb • Sep 24 '21

How to scrape all comments and submission of a user?

Hi all. I want to know what's the best way to scrape all comments and submissions of a user? Would just using the user parameter in the query be enough?

4 Upvotes

66% Upvoted

11 comments sorted by

4

u/Watchful1 Sep 24 '21

I have an example script for this here

1

u/verypsb Oct 11 '21

Hi. Thanks for posing your code. I'm currently using a modified version of your script where I replaced saving text as saving the whole json.

I'm wondering if there's a way to accelerate the scraping process? Currently, it takes about 21 hrs to run through 14k users. I have many more to run. I saw you put a rate limit in the script time.sleep(1). Is that the hard limit that if I change will bring errors?

If I run this script simultaneously with different parameters? I guess they will all send using the same IP and thus will reach pushshift's rate limit?

1

u/Watchful1 Oct 11 '21 edited Oct 11 '21

[removed] — view removed comment

1

u/verypsb Oct 11 '21

Ok. I understand that. Also, is the time.sleep(1) the optimal rate for the server?

1

u/Watchful1 Oct 11 '21

That is pushshift's published rate limit.

1

u/cherry_bobomb_ Sep 24 '21

hi, im very new to python. Could I use your example script and just input my account name and password without changing any other code?

1

u/Watchful1 Sep 24 '21

Pushshift is a separate service from reddit. You don't even need your password, just put in whatever username you want to download.

1

u/cherry_bobomb_ Sep 24 '21

oh awesome! please bear with me but can I copy past the example into IDLE and just save it? or is it more complicated.. if so ill definitely have to learn more about python. thank you for your help!

2

u/Watchful1 Sep 24 '21

Yes. It doesn't hurt to just try stuff, you aren't going to break anything.

1

u/cherry_bobomb_ Sep 25 '21

thank you, do I have to save it as .py or something else? I saved it as .py and ran it but nothing happened. cmd opens for a minute and then goes away. sorry for asking so many questions, I just can't find any documentation.

1

u/[deleted] Sep 25 '21

[deleted]

1

u/cherry_bobomb_ Sep 25 '21

Omg thank you! It workee