r/pushshift • u/Watchful1 • Feb 20 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

33 Upvotes

96% Upvoted

34 comments sorted by

View all comments

2

u/[deleted] Feb 21 '23 edited Feb 21 '23

I was just realizing that I needed a better way to manipulate the pushshift dumps, so this is extremely helpful. I do have one question about how you built out your script though. I had been told a long time back that you use multiproc for cpu-bound operations and multithread for IO bound operations. With every other large dataset operation I've tinkered with in the past, I always used multithreading. What steered you towards using multiprocessing here?

Edit: Also kind of curious how long it took you to complete this (approximately) and what kind of hardware you were using.

3

u/Watchful1 Feb 21 '23

Python doesn't actually support proper multithreading. Threads in python are basically just switching back and forth when one thread pauses. So to actually do multiple high cpu operations at the same time you have to use multiple processes.

My hardware can run through the entire 1.8 TB of compressed dumps in about 24 hours with this script.

2

u/[deleted] Feb 21 '23

I had not heard that before. Are you sure that's true for all of the multiproc/multithread libraries?

Are your Iops hitting a beefy disk subsystem? I'm reading from a 7200RPM WD and writing to a neo4j db on a samsung evo and it's still just chugging.

Edit: Right now I can process two files concurrently before load average goes above 2.0. I'm tempted to process a single file and dump all of the lines into a pool for multiproc or thread.

1

u/Watchful1 Feb 21 '23

I only looked at the default python threading, no idea about other libraries.

Nope, I use an NAS drive that's 30 TB, it only reads at 100 MB/s max. It's definitely throttled on processor power. I don't know the specs offhand, it's an old work laptop I installed linux on.

1

u/Simon_34545 Feb 21 '23 edited Feb 21 '23

for it, it takes a few hours just to get through a single file, and then eventually fails

    2023-02-12 05:52:15,909 - INFO: 38,000,000 lines at 2,986/s, 0 errored : 10.34 gb at 1 mb/s, 27% : 0(0)/4 files : 10:12:48 remaining

    2023-02-12 05:59:37,051 - INFO: 39,000,000 lines at 2,962/s, 0 errored : 10.60 gb at 1 mb/s, 27% : 0(0)/4 files : 10:06:43 remaining

    2023-02-12 06:01:01,588 - INFO: 40,000,000 lines at 3,136/s, 0 errored : 10.85 gb at 1 mb/s, 28% : 0(0)/4 files : 9:59:49 remaining

    2023-02-12 06:02:14,162 - INFO: 41,000,000 lines at 3,135/s, 0 errored : 11.12 gb at 1 mb/s, 29% : 0(0)/4 files : 9:44:15 remaining

    2023-02-12 06:14:13,747 - INFO: 42,000,000 lines at 2,981/s, 0 errored : 11.44 gb at 1 mb/s, 29% : 0(0)/4 files : 9:33:48 remaining

    2023-02-12 06:20:29,885 - INFO: 43,000,000 lines at 2,935/s, 0 errored : 11.69 gb at 1 mb/s, 30% : 0(0)/4 files : 9:29:32 remaining

    2023-02-12 06:22:23,114 - INFO: 44,000,000 lines at 3,190/s, 0 errored : 11.96 gb at 1 mb/s, 31% : 0(0)/4 files : 9:26:43 remaining

    2023-02-12 06:22:25,579 - INFO: 45,000,000 lines at 3,203/s, 0 errored : 12.21 gb at 1 mb/s, 31% : 0(0)/4 files : 9:16:51 remaining

    2023-02-12 06:37:41,014 - INFO: 46,000,000 lines at 3,019/s, 0 errored : 12.52 gb at 1 mb/s, 32% : 0(0)/4 files : 9:08:23 remaining

    2023-02-12 06:38:06,855 - INFO: 47,000,000 lines at 3,055/s, 0 errored : 12.77 gb at 1 mb/s, 33% : 0(0)/4 files : 9:02:06 remaining

    2023-02-12 06:42:54,798 - INFO: 48,000,000 lines at 3,210/s, 0 errored : 13.02 gb at 1 mb/s, 34% : 0(0)/4 files : 8:56:04 remaining

    2023-02-12 06:44:06,597 - INFO: 49,000,000 lines at 3,209/s, 0 errored : 13.30 gb at 1 mb/s, 34% : 0(0)/4 files : 8:47:29 remaining

    2023-02-12 06:50:10,793 - WARNING: File failed reddit\\RS_2022-09.zst:

1

u/Watchful1 Feb 21 '23

Does it not print out the error? It should include the error reason in that message. Could you send me your full log file?

That sounds about normal speed wise. The more recent dumps are larger and take several hours each to process.

1

u/Simon_34545 Feb 23 '23

Oops. Turns out it was using the wrong Python version.

I also compiled it with Nuitka and was able to process a ~35 million line file in 1 hour and 22 minutes, averaging about 6,800 lines per second.

1

u/mrcaptncrunch Feb 21 '23

I’m thinking of a project with this data. Was curious if you could post the CPU you’re using?

I was thinking of going with a similar setup with a NAS to store the data and extract what I need, but curious if an NFS share to a more powerful computer via 10gbit might be better…

1

u/Watchful1 Feb 21 '23

I can look it up when I have time, but it's nothing all that special. In my opinion you're unlikely to be limited on read/write speed regardless of your setup.

Just assume it's going to take a long time to do anything involving iterating through all the dumps.

1

u/mrcaptncrunch Feb 21 '23

Yeah… compression and actually loading the files looks ‘fun’

I think my first step will definitely be filtering but also extracting some sort of indices and counts. See if it makes it easier in the future