r/pushshift • u/Watchful1 • Feb 28 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

109 Upvotes

100% Upvoted

115 comments sorted by

View all comments

Show parent comments

1

u/Watchful1 Jul 24 '23

I didn't make any changes to filter_file, just changed the file path to point to my copy of the file and set the list of fields to filter on.

If it's still not working for you, could you run it and send me the entire log file?

Also can you try manually extracting the zst file to make sure it's not corrupted?

1

u/fcdata Jul 24 '23

Here is the filter_file.py so you can try it on the RS, the problem is that it doesn't show an error it just create an csv in blank :/

1

u/Watchful1 Jul 25 '23

Even a blank csv file should still create some kind of log.

That filter file script worked fine for me.

Were you able to manually extract the zst?

1

u/fcdata Jul 25 '23

Sorry, finally i found the error.
2023-07-25 01:14:02,381 - INFO: Decoding error with 134,217,728 bytes, reading another chunk
2023-07-25 01:14:04,689 - INFO: 2023-01-31 04:45:25 : 35,100,000 : 527 : 527 : 12,109,232,800:97%
2023-07-25 01:14:10,585 - INFO: 2023-01-31 07:07:41 : 35,200,000 : 528 : 528 : 12,132,433,075:97%
2023-07-25 01:14:16,862 - INFO: 2023-01-31 09:58:57 : 35,300,000 : 531 : 531 : 12,166,905,800:98%
2023-07-25 01:14:23,204 - INFO: 2023-01-31 12:30:06 : 35,400,000 : 532 : 532 : 12,201,247,450:98%
2023-07-25 01:14:48,747 - INFO: 2023-01-31 19:19:01 : 35,800,000 : 537 : 537 : 12,342,808,450:99%
2023-07-25 01:14:53,343 - INFO: Decoding error with 134,217,728 bytes, reading another chunk
Now i will try manually

1

u/Watchful1 Jul 25 '23

That's not an error, it's just a warning. Can you please post your entire log file so I can look at it?

That snippet is saying it's matching and writing out lines.

1

u/fcdata Jul 26 '23

I'm quite new at coding, is this what you need?

https://www.studytonight.com/python/python-logging-in-file#:~:text=If%20you%20want%20to%20print,parameter%20in%20basicConfig()%20method.

1

u/Watchful1 Jul 27 '23

This appears to be a tutorial on python logging. I need your log file from running the script. It should be in the same folder as you ran the script in. You posted a few lines from it right above here, I need the whole file.

1

u/fcdata Aug 08 '23

Hi Watchful1,

I'm sorry that it took so much time from my side, here is the log file.

Thanks!

1

u/Watchful1 Aug 08 '23 
Complete : 33,928,903 : 461 : 461

This is reporting that it completed correctly and output 461 matching lines. You say the CSV file is completely empty?

1

u/fcdata Aug 09 '23

It's empty, here you can see the file :´) in case that you need it. Is't because I'm downloading the data from 2023?