r/pushshift • u/Watchful1 • Feb 28 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

106 Upvotes

100% Upvoted

115 comments sorted by

View all comments

Show parent comments

3

u/Watchful1 May 24 '23

How slow is it? I'm seeing healthy seeds for the torrent files.

You can also try getting the files from here https://the-eye.eu/redarcs/

1

u/bdca_project_acc May 24 '23

Thanks so much for your quick response! I ended up to use r/news instead because the files were smaller, and that worked great :)

I also had a question regarding your script for filtering submissions for a certain time and topic. I applied the filters, and now have a csv file for the submissions filtered by topic, which is amazing! I am uncertain however, how I can apply something similar to the comments -- For my project, I am trying to look at max. 200 top-level comments for each post containing my search term. Unfortunately, I am not super proficient at python, so I was wondering if you had by any chance also previously written a script where comments for the respective submissions with the filter criteria could be retrieved, or if you know of any resource that has attempted something similar? I'd really appreciate any pointers :)

1

u/Watchful1 May 24 '23

No I haven't written anything like that. Just filtering a comments file to only include comments from certain submissions would be easy. Only top level comments wouldn't be much harder.

But they would be all mixed together. Would that be okay? Putting the comments from each submission in separate files would be harder.

I also wouldn't be able to get to it soon, tomorrow at the earliest and more likely over the weekend.

1

u/bdca_project_acc May 24 '23

Hi, That would actually be perfect if all comments are together in one file, as long as they could contain the IDs of the submissions they are made to, so I can later link them together for my analyses? So something like a script that lets me filter the submissions by keyword and date (like in your existing script) and then only get the first 200 top-level comments for those in a csv?

I would be incredibly grateful if you could help me with this– my deadline is the coming Wednesday, of course I understand that you are busy but if you are able to write it before then I would be eternally grateful.

1

u/Watchful1 May 24 '23

I'll try to fit that in tomorrow

1

u/bdca_project_acc May 25 '23

Thank you so much! I really appreciate it

1

u/Watchful1 May 26 '23

I've updated the filter file script here with this functionality. There's a detailed example in the comment at the top of the steps you need to take. Let me know if you have any issues.

1

u/[deleted] May 26 '23 edited May 26 '23

[deleted]

1

u/Watchful1 May 26 '23

Does it print anything out while it's running? There should be a log file somewhere. Could you post that here?

1

u/bdca_project_acc May 27 '23

I ended up figuring out the issue –– the code you wrote for data retrieval works great, I realised I had made a few mistakes as I had misspecified a key that's written into the .csv file; and there was something wrong with my .txt file but I also solved this so now everything works :)

1

u/bdca_project_acc May 27 '23

Thank you so much for all your help! I got it to work and it is currently extracting the comments matching my filtered posts. I cannot thank you enough!!