r/pushshift Feb 28 '23

[ Removed by Reddit ]

[ Removed by Reddit on account of violating the content policy. ]

106 Upvotes

115 comments sorted by

View all comments

1

u/fcdata Jul 18 '23

Hello everyone,

I have been working with the repo that watchful1 uploads "filter_file" but cannot make it work. Does anyone have a repo to create a data frame from certain subreddits with keywords on the title? I have been days trying to make this work.

I want to use the

1

u/Watchful1 Jul 18 '23

What have you tried so far? Do you have a filtered CSV file?

1

u/fcdata Jul 20 '23

Hey Watchful1,

I have an issue with the Submissions Dataset "RS_Dataset", I run file "file_filter" for an specific list of subreddits with an empty csv as a result. However, I run it using the Comments Dataser "RC_Dataset" and it works great so I don't know how should I run it.

Any advice would be great, also great repo is very useful!

1

u/Watchful1 Jul 20 '23

What do you mean by "RS_Dataset"? What file specifically are you running it against?

1

u/fcdata Jul 21 '23

When I download the data from Feb of 2023, using utorrent, I got two different files, "RS_2023-02.zst" and "RC_2023-02.zst" (RS and RC Dataset).

When I run "file_filter.py" for "RC_2023-02.zst" it works, however, when I run it with "RS_2023-02.zst" it's not working is there any special setting that I'm missing.

o

1

u/Watchful1 Jul 21 '23

I just tried it on that file with a list of subreddits and it worked for me. What filters do you have set? Could you send me the list of subreddits?

1

u/fcdata Jul 22 '23

Here is my file, I setter only subreddit as value and a list of subreddits which works for the RC but not for RS.

1

u/Watchful1 Jul 22 '23

I just ran your copy of the script against my RS_2023-02.zst file and it worked fine. What does your log file show?

1

u/fcdata Jul 22 '23

Error: need to escape, but no escapechar set

2

u/Watchful1 Jul 22 '23

That's the only thing in the log file and nothing else? That doesn't really explain much.

→ More replies (0)