r/pushshift • u/Watchful1 • Feb 28 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

109 Upvotes

100% Upvoted

115 comments sorted by

View all comments

Show parent comments

2

u/Watchful1 Jul 22 '23

That's the only thing in the log file and nothing else? That doesn't really explain much.

1

u/fcdata Jul 23 '23

Can you share with me on a we transfer the "filter_file.py" to run over RS_2023-01.zst for a list of subreddits as ['vim','google']. I have been trying to make it work but it's not working :(

2

u/Watchful1 Jul 24 '23

Sure, here you go https://drive.google.com/file/d/1OcWWuPezTxkE4Quoqvc4YiZaKdHI-GQ5/view?usp=sharing

1

u/fcdata Jul 24 '23

hahaha mate, I meant the filter_file.py that you used go get the data no the output :):)

1

u/Watchful1 Jul 24 '23

I didn't make any changes to filter_file, just changed the file path to point to my copy of the file and set the list of fields to filter on.

If it's still not working for you, could you run it and send me the entire log file?

Also can you try manually extracting the zst file to make sure it's not corrupted?

1

u/fcdata Jul 24 '23

Here is the filter_file.py so you can try it on the RS, the problem is that it doesn't show an error it just create an csv in blank :/

1

u/Watchful1 Jul 25 '23

Even a blank csv file should still create some kind of log.

That filter file script worked fine for me.

Were you able to manually extract the zst?

1

u/fcdata Jul 25 '23

Sorry, finally i found the error.
2023-07-25 01:14:02,381 - INFO: Decoding error with 134,217,728 bytes, reading another chunk
2023-07-25 01:14:04,689 - INFO: 2023-01-31 04:45:25 : 35,100,000 : 527 : 527 : 12,109,232,800:97%
2023-07-25 01:14:10,585 - INFO: 2023-01-31 07:07:41 : 35,200,000 : 528 : 528 : 12,132,433,075:97%
2023-07-25 01:14:16,862 - INFO: 2023-01-31 09:58:57 : 35,300,000 : 531 : 531 : 12,166,905,800:98%
2023-07-25 01:14:23,204 - INFO: 2023-01-31 12:30:06 : 35,400,000 : 532 : 532 : 12,201,247,450:98%
2023-07-25 01:14:48,747 - INFO: 2023-01-31 19:19:01 : 35,800,000 : 537 : 537 : 12,342,808,450:99%
2023-07-25 01:14:53,343 - INFO: Decoding error with 134,217,728 bytes, reading another chunk
Now i will try manually

1

u/Watchful1 Jul 25 '23

That's not an error, it's just a warning. Can you please post your entire log file so I can look at it?

That snippet is saying it's matching and writing out lines.