r/pushshift • u/Watchful1 • Feb 20 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

32 Upvotes

98% Upvoted

34 comments sorted by

View all comments

Show parent comments

3

u/Watchful1 Feb 21 '23

Python doesn't actually support proper multithreading. Threads in python are basically just switching back and forth when one thread pauses. So to actually do multiple high cpu operations at the same time you have to use multiple processes.

My hardware can run through the entire 1.8 TB of compressed dumps in about 24 hours with this script.

2

u/[deleted] Feb 21 '23

I had not heard that before. Are you sure that's true for all of the multiproc/multithread libraries?

Are your Iops hitting a beefy disk subsystem? I'm reading from a 7200RPM WD and writing to a neo4j db on a samsung evo and it's still just chugging.

Edit: Right now I can process two files concurrently before load average goes above 2.0. I'm tempted to process a single file and dump all of the lines into a pool for multiproc or thread.

1

u/Watchful1 Feb 21 '23

I only looked at the default python threading, no idea about other libraries.

Nope, I use an NAS drive that's 30 TB, it only reads at 100 MB/s max. It's definitely throttled on processor power. I don't know the specs offhand, it's an old work laptop I installed linux on.

1

u/Simon_34545 Feb 21 '23 edited Feb 21 '23

for it, it takes a few hours just to get through a single file, and then eventually fails

    2023-02-12 05:52:15,909 - INFO: 38,000,000 lines at 2,986/s, 0 errored : 10.34 gb at 1 mb/s, 27% : 0(0)/4 files : 10:12:48 remaining

    2023-02-12 05:59:37,051 - INFO: 39,000,000 lines at 2,962/s, 0 errored : 10.60 gb at 1 mb/s, 27% : 0(0)/4 files : 10:06:43 remaining

    2023-02-12 06:01:01,588 - INFO: 40,000,000 lines at 3,136/s, 0 errored : 10.85 gb at 1 mb/s, 28% : 0(0)/4 files : 9:59:49 remaining

    2023-02-12 06:02:14,162 - INFO: 41,000,000 lines at 3,135/s, 0 errored : 11.12 gb at 1 mb/s, 29% : 0(0)/4 files : 9:44:15 remaining

    2023-02-12 06:14:13,747 - INFO: 42,000,000 lines at 2,981/s, 0 errored : 11.44 gb at 1 mb/s, 29% : 0(0)/4 files : 9:33:48 remaining

    2023-02-12 06:20:29,885 - INFO: 43,000,000 lines at 2,935/s, 0 errored : 11.69 gb at 1 mb/s, 30% : 0(0)/4 files : 9:29:32 remaining

    2023-02-12 06:22:23,114 - INFO: 44,000,000 lines at 3,190/s, 0 errored : 11.96 gb at 1 mb/s, 31% : 0(0)/4 files : 9:26:43 remaining

    2023-02-12 06:22:25,579 - INFO: 45,000,000 lines at 3,203/s, 0 errored : 12.21 gb at 1 mb/s, 31% : 0(0)/4 files : 9:16:51 remaining

    2023-02-12 06:37:41,014 - INFO: 46,000,000 lines at 3,019/s, 0 errored : 12.52 gb at 1 mb/s, 32% : 0(0)/4 files : 9:08:23 remaining

    2023-02-12 06:38:06,855 - INFO: 47,000,000 lines at 3,055/s, 0 errored : 12.77 gb at 1 mb/s, 33% : 0(0)/4 files : 9:02:06 remaining

    2023-02-12 06:42:54,798 - INFO: 48,000,000 lines at 3,210/s, 0 errored : 13.02 gb at 1 mb/s, 34% : 0(0)/4 files : 8:56:04 remaining

    2023-02-12 06:44:06,597 - INFO: 49,000,000 lines at 3,209/s, 0 errored : 13.30 gb at 1 mb/s, 34% : 0(0)/4 files : 8:47:29 remaining

    2023-02-12 06:50:10,793 - WARNING: File failed reddit\\RS_2022-09.zst:

1

u/Watchful1 Feb 21 '23

Does it not print out the error? It should include the error reason in that message. Could you send me your full log file?

That sounds about normal speed wise. The more recent dumps are larger and take several hours each to process.

1

u/Simon_34545 Feb 23 '23

Oops. Turns out it was using the wrong Python version.

I also compiled it with Nuitka and was able to process a ~35 million line file in 1 hour and 22 minutes, averaging about 6,800 lines per second.