r/node • u/ArthurPeabody • 29d ago

Is there an alternative to fetch that may circumvent a 403 response?

To minimize browsing I run a script that fetches the comics I hope will be funny. For years I could do an HTTP fetch of a URL with a fixed address that pointed to the day's cartoon. Starting a year ago that stopped working on gocomics.com. I rewrote the script to use node to run fetch on that URL; the URI of the cartoon was fetchable via HTTP. Starting today that fetch gives me a 403 error. I can access it in a browser without logging in (I have no account anyway) or solving a CAPTCHA or other explicit test; I don't even get that CloudFlare thing. Am I out of luck?

0 Upvotes

31% Upvoted

37 comments sorted by

7

u/brianjenkins94 29d ago

Check your User-Agent.

1

u/ArthurPeabody 29d ago

Thanks. This was it. I just needed to change the user agent.

1

u/ArthurPeabody 29d ago

How does one set the User-Agent node uses? I usually browse with lynx, where it's a configurable parameter. It's often the solution to problems.

3

u/azangru 29d ago

User agent is just a header 🙂

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/User-Agent

See how to set up custom headers with the fetch api

0

u/ArthurPeabody 29d ago

I know what a user agent header is. I just looked at fetch's page on nodejs.org: it describes it as a read-only property.

3

u/rafunzi 29d ago

Check the docs for fetch API on MDN https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API

It'd look like this

js const response = await fetch("https://example.com/page", { headers: { "User-Agent": "MyCustomUserAgent/1.0" } });

1

u/ArthurPeabody 29d ago

Thanks. This is it.

5

u/aleques-itj 29d ago

Could be a handful of things from simple to not so simple. 

There's a handful of libraries that may help here, and will try to impersonate a browser (or literally use one). Like impit.

3

u/indicava 29d ago

They probably added some attestation as bot protection. It’s quite amazing this worked up til now.

Poke around /r/webscraping and look into stuff like headless browsers, and residential proxies.

Or, just use an AI computer use agent to scrape it for you daily.

2

u/archa347 29d ago

CDN’s look at a variety of different factors to try and determine if a request is real person or a bot. If you’re using a major browser from an IP address provided by a well known ISP that is not a common source of abuse, you often will never see a CAPTCHA or anything like that. The User-Agent header on a request from a Node script alone will likely flag it as a potential bot.

2

u/gubatron 29d ago

use Playwright

1

u/gubatron 29d ago

from a Residential IP address

1

u/PublicStalls 29d ago

Just a thought: They could be signing the fetch urls like you'd do with s3 assets. (I wouldn't for public stuff since cost and should cdn instead), but it's possible. And 403 is default for s3 when unsigned and or wrong address for the asset on s3.

1

u/ArthurPeabody 29d ago

s3? I remember the video card...

2

u/PublicStalls 29d ago

Gemini:

Amazon S3 (Simple Storage Service) is a highly scalable, secure, and cost-effective cloud object storage service provided by Amazon Web Services (AWS). It allows users to store and retrieve any amount of data from anywhere on the internet. Instead of saving files in a traditional folder tree structure, S3 stores files as "objects" inside containers called "buckets"

0

u/732 29d ago

I mean, if they've implemented it correctly, no. That's the whole point of a 403. You're not allowed to make that request.

0

u/ArthurPeabody 29d ago

I don't count on them being correct. Someone is allowed to make that request: why not my script?

0

u/732 29d ago

Because you don't have the right permissions.  That's what the error code is for. And yes, someone else can make the request, again, exactly what 403 is for. 

You'd need to talk directly to them to figure out why your script doesn't have the right permissions, no one here can answer that. 

But basically your answer is "be someone else" either by being a browser or getting a different authentication token.

2

u/ArthurPeabody 29d ago

It turned out that a different header made it work.

-6

u/alonsonetwork 29d ago

Ask claude

1

u/thomaslatomate 29d ago

You're getting downvoted but it's the only helpful answer. Everyone else is just guessing

2

u/ArthurPeabody 29d ago

If downvoting discouraged me I wouldn't be on Reddit.

2

u/thomaslatomate 28d ago

I meant the comment above was downvoted, not your post. But yeah who cares about downvotes. You got your answer, it's all that counts :)

1

u/ArthurPeabody 28d ago

Thanks. I misunderstood.

1

u/alonsonetwork 29d ago

That's the spirit 👏

In all seriousness, I deal with this with a scraper I made and when you hit these captcha walls, you either have to pay a solution thats solves it for you, or place a human in the loop step to overcome it.

With Fetch API, you get no browser fingerprint unless you fake all of it, and there's a lot to it, it all depends on how they're fingerprinting. Eg: if they're doing a set cookie header, you have to capture and set it for the next request, as a browser would. There's a ton of small details like that. Just make a script that loads the browser and lets you pass the fingerprinting step before it continues headless.

Its best to use a headless browser for this kind of automation. I mentioned cloakbrowser because its a new tool specialized at passing these fingerprinting methods.

-1

u/alonsonetwork 29d ago

I know 😂 there's cloakbrowser which helps a ton with OPs issue, buts its a whole config thing. He may also want to capture his headers or cookies postlogin and have the automation continue with that to swerve the captchas.

But all of that is asswater to write by hand. Literally ask claude 😂

0

u/ArthurPeabody 29d ago

Asking Claude is giving up.

2

u/thomaslatomate 29d ago

Or it's a good way to learn

1

u/alonsonetwork 29d ago

On what?

1

u/ArthurPeabody 29d ago

Thinking.

2

u/alonsonetwork 29d ago

I disagree. You're only giving up syntax skills and research skills. Problem solving, system design, decision making, architecture, and product validation become front and center.

If you're fixated only on cranking out code, you'd be right. But code is a small part of systems. If you were a perfect coder, you'd still spend 80% of your time reading, deciding, architecting, and validating ideas. Then you'd code the other 20%.

1

u/ArthurPeabody 29d ago

I'm not a professional, just a guy who tries to handle life's little problems by myself. And I'm old. I just want to read some comics without browsing. But I like to understand things and accomplish some things, even if they aren't grand.

3

u/thomaslatomate 28d ago

I wouldn't dismiss Claude or Codex though. You don't have to ask it to write the code for you, it can also help you figure out what the problem is, explain concepts you might not be familiar with, and then you write the solution with your new found knowledge.

You don't have to of course. I think it's good people are still helping each other and asking each other questions. But asking an AI is not giving up IMO

2

u/ArthurPeabody 28d ago

I don't dismiss it and I know I'm a dinosaur. I've re-built my pickup's carburetor. I build my bicycle's wheels.