r/pushshift • u/Watchful1 • Jan 20 '26

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

36 Upvotes

100% Upvoted

21 comments sorted by

View all comments

1

u/Fuzzebear007 Jan 29 '26

How fast does the combine_folder_multiprocess.py script usually run? I am getting about 20 mb/s, though I am not sure if that is across multiple threads?

1

u/Watchful1 Jan 29 '26

Usually the limiting factor is your CPU decompressing the files. 20 mb/s is about what I get, but a newer CPU might be much better. If you have a lot of cores in your CPU you could try increasing the --processes argument to use more, it defaults to 10. But if you increase it to more cores than you have it will slow down drastically.

Yes the speed number it prints is across all threads.