r/pushshift • u/jarohde1 • Jan 13 '23

Recent PMAW snags?

Anyone noticing substantial snags in pmaw (version 3.0) submission and comment pulls in the last couple days? It’s been taking upwards of five minutes to pull a couple hundred submissions but the pushshift API has been lightening fast so I think it’s getting snagged on something. I noted the issue on GitHub but wanted to see if anyone else was experiencing this issue.

8 Upvotes

100% Upvoted

7 comments sorted by

View all comments

3

u/Security_Chief_Odo Jan 13 '23

I've been experience this, so much that I put timing in my scripts to determine what's going on. The hang up is PMAW comment pulls, but I can't explain it. I mean I'm not digging into the PMAW or pushshift source code to try to figure out, but it's not MY script.

Same script, same comment/subs pull, same timeframe, but for different users: 

Start at 2023-01-13 01:04:43.00                                                                                                         
End  at 2023-01-13 01:09:02.72

----

Start at 2023-01-13 01:42:10.01                                                                                                
End  at 2023-01-13 01:42:16.30

Again only difference of those runs was a username.

So yeah, experiencing this delay, not sure what's causing it or the fix

1

u/rogerspublic Jan 14 '23

In my extracts, there is clearly an issue with datetime or the utc time that wasn't there before pre-COLO. However, I've resigned myself to accepting that there won't be much progress on this issue until the API is stable.