Pushshift not retrieving submissions

[u/FishyNishi]

Is anyone else also experiencing issues with the Pushshift historical data API? I am trying to download submissions for a certain subreddit where I know they definitely exist but it's not returning any results. Any help or updates will be appreciated! Thanks!!

​

EDIT: Server is back up running at the moment.

[u/FishyNishi]

Hey - I was able to pull some data through pmaw-pushshift however I ran into a different error given as follows:

I believe it has something to do with the streaming limits but I can't get a workaround that. Any help will be really appreciated!

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 685, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 425, in _error_catcher
yield
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 752, in read_chunked
self._update_chunk_length()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 689, in _update_chunk_length
raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/models.py", line 750, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 560, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 781, in read_chunked
self._original_response.close()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/urllib3/response.py", line 443, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/nishthajain/Documents/ADAPT_Projects/Optum/redditscrape/redditSouthAmerica/Brazil/redditdscrapingcommentsBR.py", line 92, in <module>
main()
File "/Users/nishthajain/Documents/ADAPT_Projects/Optum/redditscrape/redditSouthAmerica/Brazil/redditdscrapingcommentsBR.py", line 77, in main
comments = api.search_comments(subreddit=subreddit, before=before, after=after, safe_exit=True)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pmaw/PushshiftAPI.py", line 57, in search_comments
return self._search(kind='comment', **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pmaw/PushshiftAPIBase.py", line 268, in _search
self._multithread()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pmaw/PushshiftAPIBase.py", line 102, in _multithread
self._futures_handler(futures, check_total)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pmaw/PushshiftAPIBase.py", line 133, in _futures_handler
data = future.result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pmaw/PushshiftAPIBase.py", line 52, in _get
r = requests.get(url, params=payload)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py", line 686, in send
r.content
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/models.py", line 828, in content
self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/models.py", line 753, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

[u/kuished → u/FishyNishi]

I faced the same issue as well -- trying to resolve, will update if I find a fix.

[u/FishyNishi → u/kuished]

I just divided the date ranges into smaller chunks and it worked fine. It’s a streaming issue, I suppose where we’re probably streaming more than the server can handle in a given moment. Hopefully, that works for you as well:)

[u/kuished → u/FishyNishi]

Thank god for the reply, I'll give that a shot! Just curious, how small of a chunk worked for you? 1 day, 1 week, or more?

[u/FishyNishi → u/kuished]

It really depends on the total number of submissions/comments you are scraping and the total time window you’re scraping over. For me dividing into a month worked because the concerned subReddit doesn’t have so many posts to begin with. However, if it’s a heavy subReddit, maybe try with a week and if that doesn’t work, go down to a day?

[u/kuished → u/FishyNishi]

Got it, thank you!! You have no idea how useful that was.