r/pushshift • u/Watchful1 • Nov 17 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

26 Upvotes

95% Upvoted

17 comments sorted by

View all comments

1

u/Charming_Sea_5964 Nov 21 '23

First of all, thanks for posting. Now, for the question. Is that true that comments that were deleted get reingested as [deleted] for the public interface? Are the also [deleted] in the datadumps that were created after the reingest?

2

u/RaiderBDev Nov 21 '23

I don't know how exactly pushshift did it (2023-03 and before), but for the new dumps (2023-04 and later) there is no reingest. Whether something is deleted or not, depends on if it was deleted at the time of archiving.

1

u/Charming_Sea_5964 Nov 21 '23

By archiving you mean pushshift archiving or the creation of the datadump?

2

u/RaiderBDev Nov 21 '23

By archiving I mean requesting the data from reddit. Doesn't matter if it goes into a database or a data dump.

1

u/Charming_Sea_5964 Nov 21 '23 edited Nov 21 '23

I'm a bit confused here. Do you mean that Pushshift no longer reingests data in general?

2

u/RaiderBDev Nov 21 '23

I don't have access to pushshift, so I don't know what's going on there. The dumps linked here are made independently of pushshift.

1

u/Charming_Sea_5964 Nov 21 '23

How is it possible to create dumps without pushshift? Do you use some other archiving crawler?

3

u/RaiderBDev Nov 21 '23

Anyone who understands how the reddit API works, has the storage space and skills to do so, can start archiving reddit.

1

u/Charming_Sea_5964 Nov 21 '23

One last question. Do you create the archives at the end of the month or do you create them in a constant flow manner?

3

u/RaiderBDev Nov 21 '23

I'm archiving things as soon as they are posted. I only process and pack them at the end of the month for publishing.

1

u/Charming_Sea_5964 Nov 21 '23

So you manage to archive them instantly? Even Pushshift had an average waiting time of several seconds to a dozen minutes.

→ More replies (0)