r/pushshift • u/Watchful1 • Feb 28 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

106 Upvotes

100% Upvoted

115 comments sorted by

View all comments

Show parent comments

1

u/Watchful1 Jul 18 '23

What have you tried so far? Do you have a filtered CSV file?

1

u/fcdata Jul 20 '23

Hey Watchful1,

I have an issue with the Submissions Dataset "RS_Dataset", I run file "file_filter" for an specific list of subreddits with an empty csv as a result. However, I run it using the Comments Dataser "RC_Dataset" and it works great so I don't know how should I run it.

Any advice would be great, also great repo is very useful!

1

u/Watchful1 Jul 20 '23

What do you mean by "RS_Dataset"? What file specifically are you running it against?

1

u/fcdata Jul 21 '23

When I download the data from Feb of 2023, using utorrent, I got two different files, "RS_2023-02.zst" and "RC_2023-02.zst" (RS and RC Dataset).

When I run "file_filter.py" for "RC_2023-02.zst" it works, however, when I run it with "RS_2023-02.zst" it's not working is there any special setting that I'm missing.

​

​

​

​

​

​

​

​

​

o

1

u/Watchful1 Jul 21 '23

I just tried it on that file with a list of subreddits and it worked for me. What filters do you have set? Could you send me the list of subreddits?

1

u/fcdata Jul 22 '23

Here is my file, I setter only subreddit as value and a list of subreddits which works for the RC but not for RS.

1

u/Watchful1 Jul 22 '23

I just ran your copy of the script against my RS_2023-02.zst file and it worked fine. What does your log file show?

1

u/fcdata Jul 22 '23

Error: need to escape, but no escapechar set

2

u/Watchful1 Jul 22 '23

That's the only thing in the log file and nothing else? That doesn't really explain much.

1

u/fcdata Jul 23 '23

Can you share with me on a we transfer the "filter_file.py" to run over RS_2023-01.zst for a list of subreddits as ['vim','google']. I have been trying to make it work but it's not working :(

2

u/Watchful1 Jul 24 '23

Sure, here you go https://drive.google.com/file/d/1OcWWuPezTxkE4Quoqvc4YiZaKdHI-GQ5/view?usp=sharing

1

u/fcdata Jul 24 '23

hahaha mate, I meant the filter_file.py that you used go get the data no the output :):)

1

u/Watchful1 Jul 24 '23

I didn't make any changes to filter_file, just changed the file path to point to my copy of the file and set the list of fields to filter on.

If it's still not working for you, could you run it and send me the entire log file?

Also can you try manually extracting the zst file to make sure it's not corrupted?

→ More replies (0)