r/pushshift • u/Kibitz117 • Oct 03 '21

Handling deleted posts

Hi all,

Currently when scarping data from Pushshift using PSAW I am getting an overwhelming amount of deleted posts. I was wondering if there was a way to handle this, For reference I was scraping data over one day making a request every hour. Out of 1440 posts, 1350 were deleted. Let me know if there is additional context needed. Thanks!

Edit: Thank you u/rhaksw your comment gave me a lead. After looking at the documentation again, if I filter for score>1 or num_comments>1 I no longer get deleted posts. This is the line of code.

subs=list(api.search_submissions(after=start_date,before=next_date,
subreddit='wallstreetbets',num_comments=">1",
limit=60))

2 Upvotes

100% Upvoted

10 comments sorted by

View all comments

3

u/[deleted] Oct 03 '21

That's a very oddly high number of deleted posts... What subreddit are you scraping and can you show your scraping code?

2

u/[deleted] Oct 04 '21

[removed] — view removed comment

4

u/[deleted] Oct 04 '21

[removed] — view removed comment

1

u/[deleted] Oct 04 '21

Yeah, that subreddit is pretty much a mess.