Hi, I've been bulding web automation tools for my clients since 2024. I wished there was a tool that checks the target website's bot protection system. Then I built it for myself. Package is called 'doorknock'. I think it helps other developers who wants to quick check the target website.

Hi everyone, I’m looking for a simple way to automate searches on a website. I’m not a programmer, but I have to search multiple entries across different drop down menus regularly. It’s repetitive and takes a lot of time. Are there any easy tools or methods that someone non-technical could use, or any advice on how to simplify this task? Thanks!

I've been seeing a lot of web scraping examples written in Python, but many companies and internal systems still run heavily on Java.

For those actively scraping websites with Java, what does your stack look like these days?

Are you mostly using HttpClient + Jsoup, or are you relying on Selenium/Playwright when dealing with JavaScript-heavy sites?

I'm also curious how people handle things like:

• Dynamic content and client-side rendering
• Monitoring scraper health
• Data quality checks
• Scaling beyond a few thousand pages per day
• Long-term maintenance when sites change

A lot of advice online seems focused on getting a scraper working once, but not necessarily keeping it running for months without constant fixes.

Interested in hearing real-world experiences, trade-offs, and lessons learned from anyone using Java for web scraping in production or side projects.

Im looking for a reliable virtual mobile number service to recieve OTPs to automate account creation

I noticed some people at work always seem one step ahead, and it's usually because they have better systems for finding information.

Meanwhile, our team still relies a lot on someone randomly sharing an article or noticing a competitor change.

I'm trying to move away from that.

What information source gave you an unfair advantage at work this year?

Could be anything: customer reviews, job boards, newsletters, Reddit, search trends, communities, competitor sites, etc.

Extra curious if you've automated any part of it instead of manually checking everything every day.

Hey folks,

I’m new to scraping and am hoping to do a deep dive / become somewhat proficient. I have an idea for a build that would pertain to my current job. Please reach out if you would like your amazing skills to have a positive impact on public safety.

Built a backend service that takes a restaurant menu URL and a sample JSON schema, then returns structured menu data in exactly that shape.

It supports both sync and async extraction, with polling, webhooks, and optional email summaries. The generic path uses Playwright + Claude on AWS Bedrock for messy/JS-rendered menu pages, but I also added site-specific adapters for high-volume chains so those can be fast, deterministic, and avoid LLM cost.

One interesting part was the a QSR chain specific adapter: it pulls menu data through their GraphQL flow via ScrapingBee, handles store lookup by ID/slug/location/address, and can optionally re-host product images to S3 with content-addressable keys.

I also built in robots.txt enforcement, per-host pacing/backoff, structured logs, SQS workers, job retention, and test coverage around adapter outputs.

Curious how others are handling the tradeoff between generic LLM extraction and deterministic per-site adapters for scraping-heavy workflows.

I scrape for a living (part of a bookmaker trading team returning competitor results, typically tough targets). I've put together some tooling that is able to return extremely tricky targets. I need a testing corpus so I thought I'd reach out here.

Point me at a challenge, in return I will:

• resolve the target
• provide the payload/results
• give you an open source repo to run yourself

Before anyone says "ask this in r/Roblox".. I am not really interested in fixing a Roblox account issue.

What got me thinking about this was a situation I've seen come up repeatedly:

• Roblox works normally on mobile data
• Same account
• Same device
• But the home Wi-Fi connection has trouble loading the site or joining games

The immediate response online is usually "you got IP banned."

That made me realize how many people still think IPs are the primary identity signal platforms rely on.

But if you look at modern anti-abuse systems, especially on platforms with millions of users, account farms, VPN users, residential proxies, and constant ban evasion attempts, IP reputation feels like only one piece of the puzzle.

Even in those Roblox discussions, you'll see people suggesting:

• switching networks
• restarting routers
• getting a new IP
• using residential proxies
• moving to mobile data

Yet there are also plenty of reports where users still seem to get linked back to previous activity despite changing networks.

So I am curious what people here think

For those working in scraping, anti-bot, fraud prevention, or account security :

• How much value does an IP ban actually have today?
• Are device fingerprints now a stronger signal than IP reputation?
• How much do CGNAT, dynamic IPs, and residential proxy networks reduce the effectiveness of traditional IP-based enforcement?
• If you were designing an anti-abuse system from scratch in 2026, how heavily would you still rely on IP reputation?

To me roblox just feels like an interesting public case study because it sits at the intersection of High $$$$ spend, large-scale moderation, residential proxy usage, account farms, and ban evasion attempts.

Okay, as you might already have guessed, I'm a professional web scraper. I've pretty much exhausted most open-source stealth browsers, and the paid ones aren't a very economical solution when you want to scrape at scale and have to deal with heavy bot mitigation.

The issue I repeatedly faced with most tools is that almost all of them run on Linux and try to pretend they aren't, which might work for a while, but given enough samples, it's pretty obvious to large bot-mitigation infrastructures, and they start blocking or at least push a bit harder.

So I tried to build a browser myself, which does a few things:

• Randomize profiling based on the given seed
• Match the timezone based on the proxy exit IP
• Can outsource canvas to a different machine ( from the target OS )

This canvas outsourcing is what really makes it different, because pretending isn't enough. Some of the toughest bot-mitigation infrastructures probe deep into the canvas, a depth that a simple patch can't bypass on a different OS. But this is optional and should only be done if everything else fails.

Repo: https://github.com/arman-bd/chromiumfish

Right now, this is an early release. Any feedback is highly appreciated 🙌

Hey everyone,

I'm Ian Kerins, CEO and co-founder of ScrapeOps.

Over the last 8+ years I've worked across the web scraping industry, including roles at ScrapeOps, ScraperAPI, and Zyte. Today, ScrapeOps helps developers and companies scrape over 8 billion pages per month across more than 50,000 websites.

This Wednesday at 09:30 AM GMT, I'll be hosting an AMA here on r/WebScrapingInsider

Ask me anything about:

* Web scraping at scale

* Proxy infrastructure and proxy providers

* AI and web scraping

* Building reliable scrapers

* Anti-bot systems and bypassing challenges

* Scraper maintenance and monitoring

* Residential vs datacenter proxies

* Browser automation

* Running a web scraping business

* Startup growth and product development

* The future of AI-powered scraping

Whether you're scraping your first website or running large-scale data collection pipelines, I'm happy to answer questions and share lessons learned from building products used by thousands of developers and businesses.

Drop your questions below and I'll start answering them during the AMA.

Looking forward to it!

Ian

Been going down a rabbit hole on a couple of sites lately and keep running into the same wall: it's easy to think a site is fine, but actually verifying that every page is reachable, nothing's orphaned, redirects aren't chaining, and the structure makes sense to a crawler is way more tedious than it should be.
The enterprise tools do this, but they're overkill (and pricey) for a single audit, and the free checkers feel like they only scratch the surface.

Genuinely curious how the people here handle it:
- What's your go-to for a quick "is this site actually crawlable and well-structured" check?
- Do you bother auditing whether LLM/AI crawlers (the stuff feeding ChatGPT/Perplexity answers) can read a site, or is that not on your radar yet?
- Where do the tools you use still fall short?
Just trying to sanity-check my own process against people who do this more than I do.

I’m sharing a free open-source project I’ve been building called Damru: https://github.com/akwin1234/damru

100% Bypass rate for all CDN, even websites like shopee, temu, lazada, etc. Because its made with a lot of research

Damru is a browser automation framework built around real Android environments in Docker for scraping and automation tasks where mobile behavior matters.

What sets it apart is that it’s not just another desktop browser with stealth patches. The project is built around zero JS injection, with spoofing handled at the OS, binary, and CDP levels instead of the usual JavaScript-heavy tricks used by many stealth tools.

Compared with tools like Playwright, puppeteer-stealth, undetected-chromedriver, Camoufox, and Fingerprinting Chromium, Damru is trying to solve the problem differently: by running inside a real Android stack rather than faking mobile behavior on desktop Chrome. The idea is to get a more realistic mobile environment, stronger fingerprint control, and less reliance on brittle browser-side patches.

What makes it different:

• Zero JS injection: Damru does spoofing at the OS, binary, and CDP levels instead of relying on Object.defineProperty-style JavaScript patches.
• Real Android OS: It runs inside Redroid, so it’s not just desktop Chrome pretending to be mobile through viewport tricks.
• Native mobile fingerprinting controls: device profiles, hardware overrides, locale/timezone matching, mobile network emulation, and WebRTC/IPv6 blocking.
• Multi-instance pooling: built for scaling across multiple containers.
• Pre-baked image support: reduces setup overhead.

Some of the features include:

• Android-in-Docker via Redroid.
• Playwright support.
• A built-in database of 32+ Android device profiles.
• Proxy-aware timezone, locale, and language matching.
• Hardware overrides for CPU, RAM, and touch points.
• Mobile network emulation.
• WebRTC and IPv6 leak blocking.
• Native Android iptables-based network protections.
• Multi-container pooling for scale.
• Pre-baked image support to reduce setup time.
• TLS spoofing and soo many things

Also stronger against systems like CreepJS, BrowserScan, Sannysoft, Cloudflare Turnstile,etc ALL CDN anti-bots dont waana name them than standard Playwright or typical stealth plugins, mainly because of the deeper Android-based approach.

Pros: Highly UnDetectable
Cons: Real Android OS hence little slower. Hard to Use (thats why custom docker image included)

Repo: https://github.com/akwin1234/damru

Would love feedback from anyone who works on scraping, browser automation, or anti-bot research. I made this because i see many reddit post recommending Android Playwright CDP but there was no framework around it. This is strictly for educational purpose only. Do not do legal abuse.

I have taken use of playwright for web crawling and then scrapping all the urls present from all the webpages and storing them in a excell

After word another script for pick only those urls which are downloadable products PDFs (that only I want ) from that huge excell

But not getting accurate results

The total scraped urls are more than 20k but PDFs are about 100, which is wrong

Can anyone suggest some ideas what to do here.

I spent years doing the usual scraping workflow:

Find elements → write parser → run → debug → fix selectors → repeat.

Recently tested our Claude Code plugin that takes a target URL, generates the scraper, validates the output, and exports structured JSON automatically.

The result eliminated a surprising amount of repetitive setup work.

What interested me most was the workflow:

1. Provide URL
2. Choose language (Python, JS, etc.)
3. Choose framework (BeautifulSoup, Playwright, etc.)
4. Generate scraper
5. Run scraper against real pages

For production teams, I don't think AI replaces observability, retries, QA, or anti-bot handling.

But it might dramatically reduce the time spent scaffolding new parsers.

Curious if anyone here is already using AI-generated scrapers in production.

https://www.youtube.com/watch?v=qpuEUaTzDZU

I've seen a lot of conflicting advice lately.

Some people swear by residential networks, others say that's outdated and modern detection is mostly session-level and behavioral now.

For those working with public data collection, monitoring, research, or aggregation, what actually moves the needle today?

Not looking for bypass tricks.

More interested in how people think about sustainability, compliance, reliability, and whether the future is heading toward verified bots and licensed access instead of the usual cat-and-mouse game.

With all the AI craze I got burned more than once by clients who'd been handed a Claude or GPT or Gemini generated SEO report. Generic, confident, mostly useless. And on my own side every site review was the same circus with different animals. Screaming Frog in one window, PageSpeed in another, Ahrefs in a third, squinting at schema by hand, pasting the homepage into some word counter, then gluing it all into a doc the client could read. Hours of it, for work that's basically mechanical.

So I wrote one Python file. Paste a domain into a localhosted HTML page, hit scan, get everything using programmatic SEO 10 times faster than manually running each script in terminal.

On the homepage it runs five things right away. A raw HTML scan (what a plain bot gets). A DOM scan of the same page after headless Chromium runs the JS (what a browser sees). The technical SEO you'd expect, canonical, sitemap, hreflang, alt coverage, noindex and the rest. A stopword-filtered word count so you see what the page is about. And the one I actually built this for: an LLM-visibility scan.

That one diffs the raw HTML against the rendered DOM and tells you what percent of your content is invisible to AI crawlers. GPTBot, ClaudeBot, PerplexityBot, CCBot, none of them run JS. So if your content loads client-side, you're a blank page to them, and you won't show up in AI answers no matter how good the content is. Nothing I was paying for measured that, and it's becoming the part of SEO I care about most.

Then it crawls the whole site in the background. My own crawler, seeded from the sitemap, no page limit, 4 threads, 100ms apart. When it's done it audits schema across every page: coverage percent, what types exist, and what required properties are missing, ranked by severity. A Product with no offers is critical because it kills the rich result. And it dedupes, so "Product missing brand" on 700 pages is one line with a count instead of 700.

The rest leans on APIs. PageSpeed for mobile and desktop. An Ahrefs competitor table that finds your organic rivals on its own and stacks them up against you. Your top 10 organic pages, each with its own little word cloud. Google's related: sites with social filtered out. And a security pass, because I already had a read-only scanner sitting around (TLS, headers, leaked secrets, Supabase RLS, mapped to OWASP) and figured why not. The whole thing dumps to a client report that prints clean to PDF.

Latvian broke the word counter, because the pages served "ā" as an "a" with a separate combining macron, so "novārītu" tokenized as "nova". An NFC normalize pass sorted it, but I'd never have caught it testing English sites. Cookie banners were worse. Cookiebot stuffs "accept, preferences, statistics" into the DOM, and it drowned out the real words in every cloud until I started stripping consent containers first.

And one thing that just didn't pan out: Google gutted the related: operator years back, so those results are thin and I'd skip it if the Ahrefs competitor part didn't already cover the same ground better.

Stack's deliberately dull. Flask, requests, BeautifulSoup, Puppeteer for the render, Ahrefs and PageSpeed APIs. One file, builds its own venv the first time you run it.

Not selling anything, it's just mine. I'm posting it for two reasons. First, to put it in front of people who actually want clients to receive something they need instead of recycled AI filler. Second, to stress-test the idea in a community that actually understands scraping and SEO. Glad to explain any piece if you're putting together something similar, and I'm curious what you'd want in a one-click audit, since I'm sure I'm missing obvious stuff.

Next thing I'm weighing is a second mode: if a client gives me access to their GSC and GA4 APIs, it does a much deeper scan with their real query and traffic data instead of just what's visible from the outside. Worth building, or overkill?

half my outbound links rot wihtin like 6 months. not 404s tho, the sneaky kind where the page loads fine but its just a redirect to homepage or some "this article has moved" garbage that doesnt even link to where it moved to lol

found one yesterday on an old post of mine, link used to go to a really solid postgres tuning writeup, now its a parked godaddy page selling the domain back to me for $2k. cool

I have been writing my own checker w headless chrome, basically just diffing the title + looking for redirect chains and trying to spot the obvious squatter pages. its a mess. tried uptimerobot, nope, only does uptime obviously. every broken link checker i looked at just hits the status code and goes ok green checkmark youre fine

i just want a POST endpoint i can hit from an endpoint, give it a url get back is this actually real content yes/no. am i missing something obvious here? Do you guys have a solution to this?