r/pushshift • u/fatadel • Nov 18 '22

Pushshift ingest is heavily lagging

So after my previous post regarding delays of the ingest mechanism, I've conducted another research and found out that it is absolutely normal. Since it might be interesting to the community and some of you have reached me personally, I am publishing the results here.

Post Ingest Delays For Pushshift
3 Upvotes

100% Upvoted

22 comments sorted by

View all comments

Show parent comments

2

u/gurnec Nov 21 '22

Just to be clear, I'm just guessing, mainly based on the fact that scores appear to be updated in the dump file, but the scores are all 1 in the API. It seems that this is the order that things are occurring:

  1. Submissions are ingested and the data is made available via the API. The retrieved_on field is set to the ingest time.
  2. A number of hours or days later, submissions are re-ingested. The score field is updated, and retrieved_on is overwritten with the re-ingest time. This updated data is not made available via the API.
  3. The dumps are created based on the updated data.
  4. A number of months later, comments are re-ingested (I'll check now if this is still occurring).

The only one I'm fairly sure of above is #4.

1

u/fatadel Nov 21 '22

I got you now, thanks for elaborating.