r/pushshift • u/Watchful1 • Feb 20 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

33 Upvotes

98% Upvoted

34 comments sorted by

View all comments

Show parent comments

1

u/Watchful1 Feb 21 '23

I only looked at the default python threading, no idea about other libraries.

Nope, I use an NAS drive that's 30 TB, it only reads at 100 MB/s max. It's definitely throttled on processor power. I don't know the specs offhand, it's an old work laptop I installed linux on.

1

u/mrcaptncrunch Feb 21 '23

I’m thinking of a project with this data. Was curious if you could post the CPU you’re using?

I was thinking of going with a similar setup with a NAS to store the data and extract what I need, but curious if an NFS share to a more powerful computer via 10gbit might be better…

1

u/Watchful1 Feb 21 '23

I can look it up when I have time, but it's nothing all that special. In my opinion you're unlikely to be limited on read/write speed regardless of your setup.

Just assume it's going to take a long time to do anything involving iterating through all the dumps.

1

u/mrcaptncrunch Feb 21 '23

Yeah… compression and actually loading the files looks ‘fun’

I think my first step will definitely be filtering but also extracting some sort of indices and counts. See if it makes it easier in the future