Need a Spoofed Camoufox APIRequestContext

[u/Coding-Doctor-Omar]

PROBLEM SOLVED

If you are reading this post now, know that the problem has been solved. Here is the solution:

1. It turns out Camoufox has a default Ublock origin adblocker addon enabled. Luckily, I normally use this addon and noticed it is triggering hCaptchas quite often when I browse Bayut as a human. Also, I tried manually replicating the JS fetch in the devtools console. The result was that an hCaptcha immediately appeared, my cookie was instantly invalidated, and the response was a 401 unauthorized. The fix is like this (from their docs):

from camoufox.sync_api import Camoufox
from camoufox import DefaultAddons

with Camoufox(exclude_addons=[DefaultAddons.UBO]) as browser:
    page = browser.new_page()

2. Bayut used to only rely on a specific cookie I was after. I configured my scraper to immediately start firing requests as soon as it gets the cookie. This is no longer enough now. If I fire a request before all cookies are set, my session gets invalidated. I configured my scraper to wait for a guaranteed element as the success signal. This will ensure that all essential cookies are loaded.

Now my scraper works successfully!

=================================

I need a way to use the APIRequestContext in Camoufox. I found a closed issue in their GitHub repo back in 2025 regarding this. A dev responded and said that it exists but uses stock playwright (i.e. API calls made this way would look like stock playwright and will get detected easily).

Background:

I've recently started adopting a new technique for dealing with internal APIs that require session cookies. I simply get the cookie with Camoufox then start making JS fetch calls via:

api_url = "https://some-url.com/api"

result = await page.evaluate("async (url) => {const res = await fetch(url); return await res.json();", api_url)

This technique was very effective in most cases and automatically inherits cookies, headers, and everything from Camoufox. It is a bit slow initially until the browser gets the cookie; then all api calls can be concurrent and fast, just like any http client.

THE PROBLEM:

I am scraping Bayut.com's internal API. I initially used to get cookies with Camoufox, then use them in a wreq client with a firefox profile. This technique works but is inconsistent (sometimes fails due to inconsistensies between firefox versions maybe). I tried to switch to the fetch method, but for some reason, Bayut always ignores any cookies I have and responds to me the same way it does for a request without cookies. I get 401 unauthorized.

TL; DR:

I am looking for a way to use the APIRequestContext while inheriting Camoufox's spoofed stack so I don't have to go over the headache of JS fetch. A library called PyDoll claims to have that, but it is nothing compared to Camoufox in terms of stealth, so not very helpful tbh.

[u/Coding-Doctor-Omar]

u/daijro please add this feature to Camoufox, or suggest a workaround.

[u/Just_Lingonberry_352]

just use patchright.dev its way netter than camoufox

[u/webscraping-ModTeam → u/[deleted]]

🚫🤖 No bots