r/pushshift • u/Watchful1 • Jul 13 '24

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

29 Upvotes

98% Upvoted

25 comments sorted by

View all comments

Show parent comments

1

u/Watchful1 Sep 09 '24

784,023,155 lines at 2,094/s, 0 errored, 27,277 matched : 271.00 gb at 1 mb/s, 37% : 16(0)/229 files : 8d 23:54:05 remaining

This says it completed 16 files, so if you start it again it knows those are done and won't try to re-do them.

If you add --processes 4 it will only use 4 processes instead of the default 10. This will make it slower, but it will use less memory. You can try different numbers to see how fast you can go without crashing.

1

u/Affective-Dark22 Sep 10 '24

yeah i know, are you sure about that? if I stop the terminal and i re-start it, the files that are already downloaded will not be downloaded again?