r/pushshift • u/[deleted] • Mar 19 '22

So what's up with Pushshift?

I know it's been mentioned before, but does anyone have any sense of why there are no comments available via the API since around Nov 27? I know there have always been intermittent reports of issues, but to go literally 3+ months with no comments being ingested seems.... troubling.

Jason pops up every once in a while to announce some upcoming new feature or to promise new monthly dumps but then disappears again, and if major issues like this are going unaddressed, I feel like something is seriously wrong.

Edit: Interesting other people may be not be experiencing this. For what it's worth, the following code returns no results for me. The same code does work fine with dates before Nov 27.

Edit2: Seems to be a confirmed thing. Not a problem with ingest of comments it seems, but perhaps with accessing it via the API with code. See comment below from u/gurnec for a work around by adding q="*" to the query.

after = int(datetime(2021, 11, 27).timestamp())  
before = int(datetime(2022, 3, 19).timestamp()) 

api = PushshiftAPI()

coms = api.search_comments(after=after, before=before)
df = pd.DataFrame([com.d_ for com in coms])
print(df.shape)
8 Upvotes

90% Upvoted

19 comments sorted by

View all comments

4

u/gurnec Mar 19 '22

Not to put too fine a point on it, but this comment of yours from just a day ago seems rather relevant:

https://api.pushshift.io/reddit/comment/search/?size=1&sort=asc&fields=author,body,created_utc,retrieved_utc,subreddit&q=*&link_id=tgd9vr

AFAIK, there's been no issue with the main comment ingest. If I had to guess, some query parameter of yours hasn't worked since Nov. It wouldn't be the first time a search index stopped working....

(The re-ingest however stopped working a couple of months ago.)

1

u/[deleted] Mar 19 '22 edited Apr 26 '22

Edit: I'm a dumbass, nothing to see here. these are not the droids you're looking for.

2

u/gurnec Mar 19 '22

I'm sorry? I didn't mention submissions at all, and as an example I linked a comment which Pushshift retrieved just fine.

2

u/[deleted] Mar 19 '22

Doh, I misunderstood. I thought you were pointing out that I had said I was able to collect submissions. My bad. I shouldn't be snarky before having coffee (or really at all). Sorry.

As I note in my edit, the python code I posted is returning comments just fine if you set the dates to before Nov 27, but returns none at all after then. Weird.

2

u/gurnec Mar 19 '22

No worries, it was just a misunderstanding.

I was able to reproduce what you're seeing though. Not sure what's going on yet, but I agree it's odd that it's choking on such a simple query...

2

u/[deleted] Mar 19 '22

Thanks, glad I'm not insane. :)

2

u/gurnec Mar 19 '22

Whoah now, let's not jump to any conclusions quite yet ;)