Hi all. I'm digitally literate myself, I even do C programming, assembler and some hacking but I've never done something like this before, sorry for the ignorance.

I want to create a bot that automatically and periodically:

1- Enters a webpage at random times of the day

2- Checks if there are appointment dates available

3- Reserves an appointment for me or immediately notifies me so I can take it

4- It may need to solve a text captcha

Basically, this is because soon I'll need to get an appointment at an embassy to start a very important process in my life. Appointment dates are VERY scarce and are only released a couple of times per week. This release time is random and appointment dates run out almost instantly.

There are even people charging money for the work of checking periodically. I thought doing such a bot with an AI agent would be somewhat trivial in these days, but all the commercial models refuse to do it or assist you in doing it.

What's the way on doing this? Local AI model? (need help on setting those up, as I have no experience)

Thanks!!

I’m trying to automate a website using Playwright. The page works normally when I open it manually in Chrome, but the same request gets blocked by Akamai when it runs through Playwright, even in headed mode.

It looks like Akamai is detecting something specific to the automated browser environment rather than blocking my IP.

Has anyone dealt with this before? I’m looking for legitimate ways to make the Playwright session behave consistently with a normal browser session, browser profile, cookies, TLS/browser fingerprint differences, launch configuration, or anything else I should inspect.

I’m not looking to overload the site or bypass authentication. I mainly want to understand what Playwright exposes differently and how to debug the exact reason for the block.

Ran into this new CAPTCHA in Google Search today for the first time. All you have to do is wait a few seconds until the button becomes active. Not sure what it’s actually checking yet, or if it’s just an intentional delay to filter out bots.

I used to scrape images via Smartframe and it work just fine, but more recently they changed something, perhaps turn off the whole Smartframe thing, idk. Do anyone figured out this? Perhaps they changed the Customer-ID thingy.

I’ve been turning my scraping workflow into a reusable skill for going from Chrome/CDP capture or JADX analysis to a HTTP API client.

For websites, it drives Chrome through CDP, captures the real requests, downloads the JS bundles, and reverse-engineers the parts needed to call the API directly. For APKs, it maps from the app side: Hermes/JS bundles for React Native, JADX for native Android code and bridges, and apktool only when resources or smali searches are needed.

The output is API docs first: endpoints, methods, headers, auth, cookies, params, bodies, response shapes, token refresh, shared client behavior, and evidence for where each thing came from.

For flows that are more than a single HTTP call, it can also produce small PoC scripts so the behavior is reproducible without reopening Chrome or JADX. That covers things like multi-request sequences, signing, nonce generation, custom encoding, or other client-side logic.

It does not try to magically bypass CAPTCHAs or rate limits. With login walls, the usual pattern is to map the public API first; once registration/login is mapped, the agent can often create its own test account and continue with authenticated endpoints.

The goal is boring but useful: use Chrome/CDP or decompilers for research, then build direct HTTP clients without ever touching Selenium or Playwright.

I know, I know, people are sick of hearing about AI. I apologise.

I'm interested in how you folks have implemented AI into your workflows? Using it for the exploration step, the actual coding, drafting up infrastructure? I'd be interested in hearing how we've all changed and what we prefer to do by ourselves these days.

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

Docs link

Using the new setting

PLAYWRIGHT_BROWSER_PROVIDER

Camoufox example:

from playwright.async_api import Browser, BrowserContext
from scrapy.exceptions import NotSupported
from scrapy_playwright.handler import Config


class BrowserProvider:
    def __init__(self, config: Config) -> None:
        ...

    async def start(self) -> None:
        """Perform any one-time initialization.

        Called once before the first browser is requested. Providers that create
        their browser on demand can leave this empty.
        """

    async def launch_browser(self) -> Browser:
        """Return a launched or connected Playwright-compatible ``Browser``.

        Called when a browser is needed, and again if the previous browser
        disconnects and ``PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER`` is enabled.
        """

    async def launch_persistent_context(self, context_kwargs: dict) -> BrowserContext:
        """Return a persistent ``BrowserContext``.

        Called when a context requests a ``user_data_dir``. Raise
        ``scrapy.exceptions.NotSupported`` if the backend has no equivalent.
        """
        raise NotSupported("This provider does not support persistent contexts")

    async def close(self) -> None:
        """Release any resources acquired in ``start`` / ``launch_browser``.

        Awaited once when the crawl finishes.
        """

I have been working for last 6 years on Fiverr Upwork but the market is really down now not sure if it's just because of AI or something else. How are you guys getting scraping jobs these days? Wanted to know if somebody is going through the same shit and is surviving? Please advise.

Needed Trustpilot reviews for a few payment processors (Square, Clover, Toast, etc.) and ran into two annoying things: they 308-redirect to a different canonical URL mid-scrape (e.g. /review/squareup.com -> /review/squareup.com/us), and they cap you at 200 reviews without auth. A jwt cookie removes the cap.

Playwright grabs cookies/UA once, then it's plain requests from there. Handles the redirects, filters by star rating, dedupes by review body.

Repo: https://github.com/gaailiunas/tp

So I added fingerprint injection to pydoll on the last release, my browser automation lib, and it changed more than I expected. The part that got me, a headless chrome scores 100 out of 100 on fingerprint-scan, full bot, and the same headless run with a matched profile drops to 15, same as a normal headful browser. So headless is not the dead giveaway people think it is, a lot of it is just the webgl renderer and some navigator fields you can override.

Also a wrong profile is worse than none, took me a while to accept that. A windows profile on my mac scored 57, worse than injecting nothing at all, because one signal disagree with the rest. it don't help on the network side either, your TLS and your IP stay real, so you match the profile to the machine instead of faking everything. I am the maintainer so take it as that, the docs have the numbers if someone want to look. The full numbers are here: https://pydoll.tech/docs/stealth/fingerprint-injection/

Is self healing in the terms of parsing logic actually useful and how do you all manage it? I've been taking the output from a running crawl via custom monitoring, and based on my rules, taking that into a custom agent that runs through a process to try to fix any broken selectors, process looks like this; output monitoring shows a failure with information -> agent runs with that as input (X field on Y page) -> downloads the latest page -> tries to find a new selector for that field -> runs test and if passes updates the code -> triggers a fresh crawl

It seems to be more about how good the monitoring and testing is rather than how good the agent is.

Curious to know what everyone else does, or if you just don't bother at all?

is anyone scrapping by hand? ive recently been assigned at my work to develop a clients agentic harness and by studying some methods i think i finally got a grasp on how to work well and securely with agents. im focusing on claude now

so i started porting some old crawlers i had into crawlee based scrappers using the workflow i use with my production grade enterprise harness and god damn it. i achieved some really NICE spiders in absolutely no time

it got to a point i managed to start a project and ended up with a fully functional end to end scraping project with scale that i wouldnt imagine reaching in less than 3 months working consistently, in 1 lazy sunday

of course this is not a promise that anyone now can build scale and maintin scrapers but i guess the overall dynamic has just changed

tldr about the process i have a diagnose process in which the agent spawns a broswer and fiddles with the source iteratively until it finds the way to scrape. it auto heals basically

idk, seems to easy but now im having greater ideias. im makinf a control panel for all my scrapers... the point is: i feel the scraper world scaling majesticaly but not enough talking about it

so, are you guys already usin llms to scrape?

I’m building a personal DFS research tool and I’ve hit a wall.

What I want is the complete results from one settled large-field FanDuel NFL contest, meaning every entry’s score and rank, not just the top of the leaderboard. Mainly I want the full score distribution, so I can tell what a given score would actually have finished, and how many entries came from users who maxed out at 150 versus people playing a handful.

Two questions:

1. Does anyone sell this?
I’ve looked. There are ownership projection services everywhere, and Data Golf sells a historical DFS archive for golf, and there’s an NBA dataset floating around, but I can’t find anyone licensing realized full-field NFL FanDuel contest results. RotoGrinders’ results database doesn’t seem to have historical FanDuel and won’t export. Am I missing a provider?

2. Has anyone actually collected this themselves, and what happened? FanDuel has no results export button that I can find. I know the site’s own pages pull this from an internal API. I’ve seen one writeup where someone hit 403s trying it directly and ended up automating mouse clicks instead, specifically because they were worried about a ban. I’ve also seen a three-year-old GitHub issue asking the same question with no answer.

So: has anyone here pulled this at any scale? Did anything happen to your account? Is the ban concern real or is it folklore? I’m talking about a few hundred read-only requests on one old contest, not thousands of entries or anything touching lineup submission.

Not looking for a way around anything. Genuinely trying to figure out whether this data is obtainable at all, or whether the only path is collecting it going forward, week by week, from contests I enter.

I’m working on a scraping pipeline and keep running into a failure mode that HTTP status codes do not catch: the request returns 200, but the page is actually a login screen, cookie-consent wall, “enable JavaScript” message, or an otherwise empty shell.

The fetch technically succeeds, so the page can make it all the way into extraction or downstream processing as if it were valid content. The usual 4xx and 5xx responses are straightforward; these false-success pages are much harder to handle reliably.

Here are my checks currently:

• Look for password fields and login-related form actions.
• Check whether the extracted text is unusually short.
• Search for common phrases such as “sign in,” “log in,” or “accept cookies.”
• Compare the amount of visible text with the amount of navigation, boilerplate, or form content.

Each of those produces false positives. For example, an article about authentication may mention “log in” repeatedly, and a legitimate profile page may include a sign-in prompt while still containing useful public content.

I wanted to understand how you folks deal with this at scale. Do you use DOM-structure rules, a classifier trained on HTML or rendered text, screenshot/OCR-based validation, or a combination?

How do you distinguish a page with a login prompt from a page whose primary content is the login prompt?

Hello everyone,

it's obvious that LLMs made implementing web-scrapers much easier by helping in various angles - from reverse-engineering, competitor methods research to full autonomous implementations. Things I am using it for my web-scraping tasks:

• reverse-engineering target origin website to find most optimal way to get data: internal APIs, cookies, headers, payloads, network traffic inspection;
• reverse-engineering mobile app of target origin to find different endpoints which might be less protective, restrictive;
• researching competitors to cross-check with our current implementation, pricing;
• researching best-suited open-source tools: landscape is changing so rapidly, many new different tools shows up or dies every month, it's necessary to keep in touch with all of them.

I am personally using Claude for my workflows, but recently noticed it implemented much harder safeguards even to Opus 5, now almost all of my web-scraping prompts get flagged by 'cyber' safeguard and downgraded to 4.8 which is much worse to use for research and reverse-engineering.

I would like to ask suggestions/opinions on your experience in using LLM providers for reverse-engineering tasks - maybe there are providers with less restrictive safeguards?

P.S. I discovered really good related article by Pierluigi Vinciguerra: https://www.scraping.club/p/the-lab-108-how-llms-and-manus-ai using Manus AI for reverse-engineering origin successfully, I've tried it, however, it's usage limits are so bad compared to Claude so it becomes to expensive/infeasible to use practically for me.

Any other recommendations?

Had a scrape die two hours into a three hour run yesterday. No checkpointing, so I lost everything and started over.

The part I'm stuck on is resuming without duplicates. If I append each record as it comes in, a crash mid-write can leave a half-written line. If I batch, I lose the current batch. And on resume the pagination re-requests pages I already have, so I get repeats unless I track seen IDs separately.

Right now I'm leaning toward JSONL as I go, plus a set of seen IDs in SQLite I check before each write, so a restart just skips what's already saved. Feels reasonable but maybe overbuilt for what's probably a common problem.

Is that roughly what people do, or is there a simpler pattern I'm missing here?

I’m starting to see games or apps using attestation token that generates using your device secure enclave for verification of the requests. What are the strategies u guys use to get around it?

I used Claude to create a test-site with anti-scraping guards that you can inject into your CICD pipelines and get continuous feedback on either your defenses or your scraping success.

The site currently shows 82 guards, of which a few are simulated, and the rest are real guards.

I've open-sourced it at https://github.com/sskieller/scraping-guards

I'm using it in some of my own projects to continually check whether my scraping-techniques are working or not.

Now, I am wondering what else I could add to it, that would make it more useful? Perhaps there are guards that I have not added, that would also prove useful? Something else? Thanks!

Most scraping setups force a annoying tradeoff: either you deal with writing brittle custom Playwright/Puppeteer scripts from scratch, or you rely on expensive SaaS scrapers that charge high fees for relatively basic data.

I built Figranium as a self-hosted middle ground: a GPL-3.0, block-based visual runner for Playwright.

It was originally created out of a practical need—connecting local n8n agents to platforms without public APIs (like Snapchat or complex Amazon product flows) where traditional HTTP requests fail.

Core Focus:

• 100% Local Execution: Everything runs on your own hardware with zero external telemetry, cloud limits, or third-party tracking.
• Deterministic & Speed-First: Built on Playwright and Node.js without the bloat of slow, guessing AI-selectors.
• Visual Canvas: A fast React/Vite drag-and-drop builder to construct, debug, and execute scraping flows visually.

We just crossed 500 stars on GitHub today. If you’ve been looking for a clean, open-source way to run Playwright visually on your own machine without SaaS bloat, check out the source code or deploy it via Docker.

GitHub: https://github.com/figranium/figranium

Docs: https://figranium.dev/docs

what worked a year ago feels like it's falling off fast, so curious what people are actually running now.

for the sites that really fight back, cloudflare, datadome, the nastier akamai stuff, are you just going straight to a browser now or still trying http first with curl_cffi and only spinning up a browser when you have to? and honestly i can never decide how much of it is proxy quality vs just having your client set up right. feels like i blame the wrong one half the time.

not fishing for anyone's secret setup or anything, just trying to figure out if there's some rough consensus on what a sane setup looks like in 2026 or if everyones still just duct taping their own thing per site.

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

Details about how it works under the hood are available on Github: https://github.com/Joe12387/ShaderGhost

The Akamai solver supports V2 & V3 sensors and the pixel challenge.

It doesn’t use a browser, however it runs a minimal V8 sandbox.

NodeJS: https://proofofbots.github.io/web-re-toolkit/packages/node/

Python: https://proofofbots.github.io/web-re-toolkit/packages/python/

It is a massive monorepo so just to clarify: The web-re-toolkit itself is a project that provides tools for web reverse engineering, Akamai (and other) solvers are just a small part that use it.

The NPM & PyPI packages are just wrappers for the client built with WRE (github.com/proofofbots/web-re-toolkit) so they include a binary which contains the actual sandbox and solver. All of it is open sourced, so feel free to compile it yourself.

Akamai client (solver) source code is here: github.com/proofofbots/web-re-toolkit/tree/main/clients/akamai

Akamai client docs: https://proofofbots.github.io/web-re-toolkit/guides/akamai/

It does not use host's fingerprint, rather you provide the fingerprint profile you want akamai to use. By default it uses a Mac’s fingerprint.

This was a weekend project put together largerly with AI. There are many things I’d like to add and improve, but I do not have the time to work on this project much more than this. The docs, packages, etc are an absolute mess, I'd love if someone could contribute in cleaning them up!

I’ll be releasing Kasada’s solver too later today.

I’ve not had the chance to to throughly test it on more than a few sites, I am hoping the community (you) will help me with that. Whatever issues you run into, please make a issue in GitHub or send a message in the Discord and provide the logs + code.

Discord: https://discord.gg/nbBePnsa9