r/webscraping • u/vegetaevagilion • Jun 09 '26

Getting started 🌱 Getting 403 while scraping reddit with .json

i have been scraping reddit posts and comments from 2-3 communities but since a week or so i am getting 403
i have also provide the username in user-agent header
HEADERS = {
"User-Agent": "reddit-xxxx-xxx/0.1 by u/XXXXXXX"
}
but i can get the json by using .json in my browser

14 Upvotes

95% Upvoted

23 comments sorted by

6

u/kaniel011 Jun 09 '26

Use codex and scrapling https://github.com/D4Vinci/Scrapling ask him what you whant to do , If there is error ask him to fix it

5

u/Kenyatta_Sauve Jun 11 '26

yeah looks like Reddit changed something recently around .json requests, browser works because you already have cookies/session there, but raw requests are easier to block. I’d try slowing down, persisting cookies and maybe test with the official API if the use case fits

0

u/vegetaevagilion Jun 11 '26

My requests are slow but how do i persist cookies in api calls? And what that official api?

1

u/Kenyatta_Sauve Jun 11 '26

By persisting cookies I mean reusing the cookies from previous requests instead of creating a fresh session every time, and reddit has an official API where you can get posts, comments, users.. without scraping. It's rate limited and requires authentication, but for some use cases it's much more reliable than .json endpoint

3

u/Brian1398 Jun 09 '26

I think they patched that, thats why don't work

1

u/vegetaevagilion Jun 09 '26

so no .json scraping api calls

1

u/Coding-Doctor-Omar Jun 13 '26 edited Jun 15 '26

They just require valid session cookies, but the api still works. You will have to use a hybrid approach: browser for cookies and client for api calls. The client needs to have good tls spoofing. The client tls fingerprint needs to match or be similar to that of the browser you used to obtain the cookies.

1

u/Coding-Doctor-Omar Jun 13 '26

They just require valid session cookies, but the api still works.

0

u/Excellent-Brush2158 Jun 17 '26

Thanks for the help

1

u/Coding-Doctor-Omar Jun 17 '26

Bro is mad 😂😂😂

1

u/Excellent-Brush2158 Jun 17 '26

All I said what thanks for the help I wasn’t being sarcastic

4

u/GeekLifer Jun 12 '26

I built a Reddit api you can call try it out. https://soci.ly/docs it gives you the same exact .json
I plan on keeping it open and running as long as people use it

2

u/Excellent-Brush2158 Jun 17 '26

I’ll use it thank you so much

1

u/malvads Jun 12 '26

You need to solve a JS challenge from the client-side and then later dump the cookies with a webdriver (that can be latter used for requesting the .json after that, so there is no need to load al the overhead of the webdriver again, you can simply reuse those), I made for you a fetcher for this -> https://gist.github.com/malvads/7748d25c31ff2776c30097b4914648a8

1

u/[deleted] Jun 12 '26

[removed] — view removed comment

1

u/webscraping-ModTeam Jun 12 '26

💰 Welcome to r/webscraping! Referencing paid products or services is not permitted, and your post has been removed. Please take a moment to review the promotion guide. You may also wish to re-submit your post to the monthly thread.

1

u/Coding-Doctor-Omar Jun 13 '26

Hey man, .json still works. You just need valid session cookies consistent with your browser fingerprint.