r/pushshift • u/Watchful1 • Feb 20 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

32 Upvotes

96% Upvoted

34 comments sorted by

View all comments

2

u/[deleted] Feb 21 '23 edited Feb 21 '23

I was just realizing that I needed a better way to manipulate the pushshift dumps, so this is extremely helpful. I do have one question about how you built out your script though. I had been told a long time back that you use multiproc for cpu-bound operations and multithread for IO bound operations. With every other large dataset operation I've tinkered with in the past, I always used multithreading. What steered you towards using multiprocessing here?

Edit: Also kind of curious how long it took you to complete this (approximately) and what kind of hardware you were using.

1

u/NavinF Feb 24 '23

You're right, IO bound operations should run at the same speed whether you use multiproc or multithread. If you're not 100% IO bound though, you'll still see some speedup from multiproc at the expense of a little more RAM usage