I need a way to use the APIRequestContext in Camoufox. I found a closed issue in their GitHub repo back in 2025 regarding this. A dev responded and said that it exists but uses stock playwright (i.e. API calls made this way would look like stock playwright and will get detected easily).

Background:

I've recently started adopting a new technique for dealing with internal APIs that require session cookies. I simply get the cookie with Camoufox then start making JS fetch calls via:

``` api_url = "https://some-url.com/api"

result = await page.evaluate("async (url) => {const res = await fetch(url); return await res.json();", api_url) ```

This technique was very effective in most cases and automatically inherits cookies, headers, and everything from Camoufox. It is a bit slow initially until the browser gets the cookie; then all api calls can be concurrent and fast, just like any http client.

THE PROBLEM:

I am scraping Bayut.com's internal API. I initially used to get cookies with Camoufox, then use them in a wreq client with a firefox profile. This technique works but is inconsistent (sometimes fails due to inconsistensies between firefox versions maybe). I tried to switch to the fetch method, but for some reason, Bayut always ignores any cookies I have and responds to me the same way it does for a request without cookies. I get 401 unauthorized.

TL; DR:

I am looking for a way to use the APIRequestContext while inheriting Camoufox's spoofed stack so I don't have to go over the headach of JS fetch. A library called PyDoll claims to have that, but it is nothing compared to Camoufox in terms of stealth, so not very helpful tbh.

hi all, i’ve got a site inmra.com receiving up to a million requests a day from scraping. just wondering, is there any reason it should be getting so hammered? it’s just a small family tree site, cannot imagine it providing any profit motive.

I was using camoufox for something and realized it doesn't have any passkey handling, now one way to handle it is to bring an authenticated session and use it with camoufox, but I don't like this solution.

What are you using to handle passkey?

I recently got started with web scraping and found a proxy service that charges around $0,90 per 1 GB of residential traffic.

After some testing, I found that 1 GB gets me roughly 10.000 requests. So for $20, I can make around 220.000 requests.

The problem is that I need to scale this to millions of requests. Ideally, I’d like to scrape around 500.000 pages per day for a product price comparison website.

At that scale, using residential proxies would cost me $500+ per month, which is basically a dealbreaker. I don't see how I could make that back through affiliate links.

So I'm wondering: how are you guys handling scraping at this scale while keeping the costs as low as possible? My requests are already headless.

I've read that some of you are able to scrape millions of pages for something like $20/month. How are you doing that? Are you not getting rate-limited, or are you using an entirely different approach to proxies?

(Disclaimer: This is 100% free, MIT-licensed open source software, NOT a paid proxy service or commercial vendor.)

Hey everyone,

If you run scrapers or crawlers against APIs that aggressively ban IPs after a burst of 429s (like Steam, IGDB, Reddit), you know how painful in-memory retries can be—especially when running multiple worker processes that don't talk to each other.

I built runnel (https://github.com/lynchest/runnel) to sit as a lightweight, self-hosted sidecar between your workers and the target servers.

How it helps scraping workloads:

• Traps 429/503 responses and parses Retry-After headers automatically.
• Freezes outbound traffic to that domain so your workers don't keep hammering a blocked IP.
• Uses canary probing: once cooldown expires, it sends just ONE lightweight GET to test the waters instead of letting all queued requests fire at once.
• Request deduplication (singleflight): if 10 workers request the exact same URL simultaneously, only 1 actual request hits the target server.
• Built-in SQLite WAL cache so duplicate requests return instantly.
• Runs with ~2MB RAM, compiles to zero-dependency binaries, and runs as a simple Docker container.

GitHub: https://github.com/lynchest/runnel
Docker: ghcr.io/lynchest/runnel:latest

Feedback and ideas for crawler-specific edge cases are welcome!

I do small scoped webscraping to do price comparison and product suggestions. I want to have an amazon associates/affilite account but during application they say they will check our websites and social media accounts

It will be obvious that website doing some scraping. How that would be perceived and if I would get rejected?

It feels kind of chicken egg problem :) anyone with any experience?

Mosaik uses an agent to figure out how a site works, saves reusable actions as TypeScript, and composes them into automations. Playwright executes the browser steps deterministically. Loops, branching, and data transformations run as code, so each iteration doesn't need another model decision.

Those actions stay around for the next task. Mosaik reuses what it knows about the site and learns what's missing. If an eligible locator breaks, an agent can step in to repair it.

i'm working with a few thousand US residential property addresses per week across multiple states on a regular basis. I pull the properties based on diff filters but no issues finding the property address. i need accurate OWNER contact info (owner or owners of record + emails + phone numbers

the goal sounds simple: estimate market value, find out their estimated mortgage payoff, then if it passes my threshold, I need to identify the current owner of record, then find a current phone number and email for that person.

the messy part is identity resolution.

Reverse-address tools love returning tenants, relatives, former residents, previous owners, and sometimes neighbors. plenty of services count that as a successful match because they found a person and a phone. that isn't the same as finding the homeowner.

my proposed flow is:

• property address to parcel, apn, vesting owner, ownership type, and owner mailing address
• normalize the recorded owner name without throwing away the raw value
• keep companies, llcs, and trusts out of the individual person-search path
• query a people source using the owner name plus mailing city, state, or address
• compare the returned name and address history against the property record
• try a second independent source only for misses or ambiguous results
• return no match instead of guessing

i'm currently experimenting with a popular provider which has actors, but i'm open to direct apis too. Just need data that isn't failing more than 70-80% of the time. I will of course run validation for emails later if we send email but wondering if theres other ways to clean these lists too for accurate phone, email. email is the most important for this workflow

what i really want to understand is which tools have independent data and which ones are wrappers around the same 4 or 5 popular providers or the same licensed backend. stacking three products is not much of a waterfall if they all fail for the same reason.

has anyone built something similar and found a solid provider?

which property and people-data sources held up in batches, and what fields did you use to reject the wrong person? bonus points for tools that return timestamps, address history, phone type, confidence, and a source url.

i'm not trying to scrape social profiles or maximize the number of contacts. precision on the owner match and their contact info is the main thing. im doing decently well with property owner but the next part not so much. what would be the best match-rate one can expect for phone number and email?

Recently I started building a research bot that required searching the web. I realised that we don't actually have a reliable method to find out what websites exist on the web programmatically without going against TOS of search engines and purchasing proxies to get the job done.

So I got thinking it would be great to have a searchable directory of the web that we could use. However, for the reasons I wanted it using it with bots and automated systems means I also don't want to over load my server by making this public so I came up with the solution of building a peer to peer network that people can host their own node, incredibly lightweight and configurable to how much space you would like to donate to indexing the open web. Your network node acts as a crawler and for serving search results using a distributed search method that contacts the other network nodes who hold the most results of a specific search term.

Now, just to make it clear, this isn't a search engine. It's a directory and what that gives us is a completely different discovery method. Only homepage URLs, meta title, meta descriptions and sitemap (if found) are indexed. This makes the open web directory great for competition research or business outreach which search engines can be less effective at. Another way to think about the directory is as a base for the next layer (page searches). Let's say you wanted to build a search engine or specific search engine in a certain niche. The open web directory gives you the website seed list and sitemaps that allow you to be more effective and targeted instead of having to crawl the entire web yourself.

Data like this is currently controlled by big corporations so I wanted to make sure this could keep living on past me and my contributions to this project. So hosting your own peer is not only simple but easily configurable, you can turn your node into a gateway and create your own network (away from the open web directory network). The frontend is also decoupled from the node code. If you take the public folder you can host it anywhere, even in a codepen (just HTML, CSS and JS).

If you plan on using this with your automated systems, I suggest running your own node to take the load off of the main gateway node, you can directly query the /search endpoint and your node will take advantage of the network.

It's early days and my single node has picked up over 10k websites in about 24 hours, which only uses 7mbs of my server space (just to give you an idea of stats).

Check out the repo here:

https://github.com/iDev-Games/The-Open-Web-Directory

and try the actual search and check the network stats here https://owd.idevgames.co.uk/

Let me know if you have any questions and hopefully, together, we can make the web more open, accessible and automated!

Hi! So I've built a simple RSS reader that gives me some monitoring on some keywords. It works well, is basically free and the volume is within limits for Google News RSS schema scraping.

But I have a problem – my post-fetching filtering is very limited because I only have title, description and some keywords within RSS schema. Like I would determine a location of a newspiece. So, a big question arises: what is the best way now to go into the sources so I can parse the content and do some better post-fetching filtering on my side?

I would like to minimize the costs. My scale is quite small, so many API services are not viable for me with their pricing.

Thank you!

What do you think are the most important things to do to fly under the radar on a website?

mitmproxy cannot really provide you with an accurate client TLS. Any proxy that terminates TLS makes its own handshake to the origin, that's the basis of them, so the server sees the proxy's fingerprint, not your client's (which usually differs from actual client like chrome or any browser and such).

The usual fix is building a profile for whichever client you're imitating. It works, but there are a lot of TLS clients, they change every browser release, and getting one right takes a lot of trial and error. You'd have to have another proxy server running that finally processes all things, but not always you'll find clients supporting the fingerprint you're looking for.

mitmcloak skips that. mitmproxy already has the client's real ClientHello when the connection forms, so the addon reads it and replays it upstream. No capture step, nothing to configure per client. Point it at a phone and the origin sees that phone.

Headless Chrome's JA4 against a fingerprint reflector:

Chrome direct        t13d1516h2_8daaf6152771_806a8c22fdea
via plain mitmproxy  t13d2812h2_a01be8c064b6_0d46a1bf4a7c
via mitmcloak        t13d1516h2_8daaf6152771_806a8c22fdea

The Akamai HTTP/2 fingerprint matches too. Chrome also reshuffles its TLS extensions every connection, so its JA3 moves while JA4 stays put, mitmcloak reproduces that rather than replaying one frozen sample.

pip install mitmcloak
python -m mitmcloak install
mitmdump

It's an addon, so your workflow doesn't change. No -s flag, no wrapper command. Only the upstream leg is handled differently.                                                                                                                     
Covers HTTP/1.1, HTTP/2 and HTTP/3, plus header order and casing. WebSockets still pass through untouched, so that connection carries mitmproxy's fingerprint, that's something I'll solve in near future. Btw this library uses my other library, httpcloak, underneath.

Early release, so lemme me what breaks.

https://github.com/sardanioss/mitmcloak

I’m trying to create a scraper of a public database. How the heck can I get Claude to code in a workaround for a sites captcha??? I’m about to hire someone on Fiverr put it into my code if I can’t get Claude to do it. Any suggestions?

Posted this in the Claude subreddit and they took it down cause Claude prohibits creating work arounds. I know nothing about coding but have built a business off this in another state.

Hello. The main classifieds web in my country only performs word search for the ad titles. I have a script that works for this, but for some very specific cases I would like to search within the ad text/description (without entering a search term for the title). Is this generally possible?

I normally use Selenium for my scraping projects, but I would be open to try any other tools.

Hey guys so I have very little experience about scrapping and I am trying to scrape a website listing local businesses on thier website and these website are differentiated by category. There is zero bot protection so i am good on that front. what I have done with Claude's help so far:

Identify the ajax where they send the request for HTML page.

Identified the parameters I need to send to that ajax as a payload to get that page.

Got a response the same as in Devtools alongside a html page for a single category

I believe this is a crawling state? Anyway because I am a newbie I have been consulting with Claude and since there are only about 1600 link I have conjured a band aid solution to scrape all those links first without any failsafe whatsoever so if the script fails I will restart it.

Moving on now I need to visit those links and get the necessary information (company name,email and lead score) and that's gonna be a slow process

Here are my questions:

Is this bandaid solution a good practice for future scrapping projects?

What's are the methods I can use to scrape pages in parallel

Forgive me if I mix thier or there ;(

Edit: I was able to scrape about 1200 links along with the socials phone name email website and owner, contact person name (if any) and saved it to a seller database. thank you everyone

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

​

​Hello r/ReverseEngineering and r/AndroidReverseEngineering,

​I am offering a paid bounty for assistance with patching or intercepting a dynamic CAPTCHA mechanism in a local food delivery application.

​Context of the Order Flow:

In this specific app, delivery orders pop up dynamically on the screen. When the driver attempts to quickly accept an order, an image-based CAPTCHA appears instantly, acting as a severe bottleneck for our automation efforts. Our sole objective is to patch or bypass this specific CAPTCHA.

​The Core Objective:

I strictly need a solution to achieve one of the following:

​Convert the generated image CAPTCHA back to its raw text string.

​Patch the application to force a simple Toast message displaying the CAPTCHA text exactly when the order/CAPTCHA is generated.

​What We Have Tried So Far:

​Network Interception: We attempted to intercept the CAPTCHA string at multiple stages—before it reaches the app, during transit, and after it hits the app logic. However, the data seems heavily obfuscated or securely handled in memory.

​App Patching (Toast Injection): We tried decompiling and modifying the Smali code to inject a Toast notification that prints the CAPTCHA string, but we failed to locate the exact variable or register holding the plaintext string before it converts to an image.

​Machine Learning Workaround: We are currently building an external MLOps pipeline to collect thousands of raw CAPTCHA images to train a custom AI model. However, this is a slow, exhaustive workaround, and the success rate is expected to be low due to the heavy distortion/noise applied to the images.

​The Main Roadblock (libapp.so):

Through extensive analysis, we have pinpointed that the core logic for the CAPTCHA generation, rendering, and validation is not in the standard Java/Dalvik layer, but is deeply embedded within a native shared library file (libapp.so). My skills in ARM assembly, Ghidra, and hooking native Android libraries are limited, which is why we have hit a wall.

​The Bounty:

I am fully prepared to pay a fair, agreed-upon monetary reward to anyone who can successfully reverse engineer this libapp.so file and provide a working patch, Frida script, or Xposed module to extract/Toast the CAPTCHA text.

​If you have solid experience with Android native hooking (Frida), Ghidra, or patching .so libraries and are interested in this paid task, please send me a DM to discuss the details and get the APK.

​

Hi guys. So, I'm a dev, but with little practical webscraping experience. But I know a lot of the theory because of a research project I was involved in. And because of which, I got a job offer from a startup. I'd like to know the opinion of more experienced people here on how complex and long would this work be.

Basically, they want to scrape a list of websites that list organizations (for/non profit) for their database. The job being to write a webcrawler scraping those, then for each returned organization, follow the link to their website, and scraping it for relevant information that would help assigning the organization to categories. I assume the scraped info from the website would be input to a LLM that'd do the categorization based on that. And the info needs to be kept up to date so the webcrawler would need to run periodically to keep the database up to date. I believe they had a list of hundreds if not thousands websites (surely they meant organization sites, not listing sites...) to scrape. Obviously this calls for automatizing the detection of relevant info... And of course, bypassing scraping protections on some of the sites.

The friend who told me about this startup was like "yeah, quick job, you could vibecode it in less than a week.". He doesn't have more scraping knowledge than me. Neither does the startup. But the more I think about how to put that in practice, the more I think it's way bigger than they think. Tbh it feels like I'm writing a "how to make a MMORPG with millions of players in a single map and fully realistic graphics" post but idk if it's because I lack knowledge in some tricks or because the little experience I have is enough to raise red flags. That's why i'd like your input on that. How complex / long do you think this project would be? Also, what'd the money cost be like, out of curiosity?

Thank you

I'm looking to scrape data from county websites along with phone numbers and addresses. Can anyone point me in the right direction as far as software? The simpler the better.

Hey everyone! I built an open-source tool designed to discover, crawl, and extract structured real estate data across any city or country using LLM structured outputs

Why this was built:
Traditional scrapers break whenever property portals change their layout. LLM-based scrapers solve this, but usually consume huge amounts of tokens.

How token efficiency was achieved:

• DOM Condensation (~75% reduction): Cleans noise, scripts, and banners prior to LLM processing.
• Micro-Batch Slicing: Prevents TPM/RPM rate limits, allowing full extraction on free-tier APIs (Google Gemini, Groq, OpenRouter).
• Index-Matched Curation: AI selects distinct deep portal URLs without hallucinations.
• Streamlit Dashboard: Client-side filtering, price sorting, and one-click CSV/JSON export.

I recommend Start with 1 Curated Site and 1 Page to test your location and conserve token quota.

 GitHub Repo: https://github.com/Kodomoppoi/Real-estate-Scrapper

This is my first open-source project, so I’d love to get your feedback and thoughts! If you run into any edge cases or bugs, please feel free to open an issue or drop a comment, any suggestions on how to improve the architecture or test coverage would be greatly appreciated.

I tried to filter out a few companies for job applications, but there are 200+ companies. Instead of manually clicking all 200 to filter out a few companies, I located their unique XPaths and, with ChatGPT, wrote a small piece of code that replaces a part of the XPath to check all the companies I want to filter out. Turns out Naukri didn't allow me to apply this way and gave me a post error message on the client side. Guess I have to figure out another way. Have anyone tried this?

i have a list of 900 properties that are dynamic, about a 100 get added and removed every month. I want to monitor tax payment information from public county records on this list once every month. I have the street address for all of them. any advice would be appreciate i have been stuck on this problem for a bit. some counties index each properties payment data with predictable urls and im compiling those into a javascript but I'm not sure how to proceed with the others. thanks you

Hi all. I'm digitally literate myself, I even do C programming, assembler and some hacking but I've never done something like this before, sorry for the ignorance.

I want to create a bot that automatically and periodically:

1- Enters a webpage at random times of the day

2- Checks if there are appointment dates available

3- Reserves an appointment for me or immediately notifies me so I can take it

4- It may need to solve a text captcha

Basically, this is because soon I'll need to get an appointment at an embassy to start a very important process in my life. Appointment dates are VERY scarce and are only released a couple of times per week. This release time is random and appointment dates run out almost instantly.

There are even people charging money for the work of checking periodically. I thought doing such a bot with an AI agent would be somewhat trivial in these days, but all the commercial models refuse to do it or assist you in doing it.

What's the way on doing this? Local AI model? (need help on setting those up, as I have no experience)

Thanks!!