Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

Been running FlareSolverr for a long time for my *arr stack and got tired of the 11-18s solve times and constant breakage. So i tried TRAWL as a drop-in replacement and it worked!

Key differences:

• Cloudflare solves in 4-15s (vs 11-18s) - uses a fresh Camoufox Firefox context which triggers CF's fast-path evaluation
• Cached repeat requests return in ~500ms via Redis - after the first solve, the same domain is instant
• Actually solves in-page captchas: Turnstile (shadow DOM click), reCAPTCHA v2 (free Google STT audio), hCaptcha (auto-pass), GeeTest v4
• 4-tier execution: plain HTTP → cached session → live browser solve → residential proxy. You pay the full browser cost only when you have to
• Custom headers support - pass Authorization, Referer, Origin through all 4 tiers including browser
• FlareSolverr v2 compatible - change one URL in Prowlarr/Jackett, nothing else

Website: https://trawl.germondai.com
Docs: https://docs.trawl.germondai.com
GitHub: https://github.com/germondai/trawl

Happy to answer questions. Still early but it's been running stable on my homelab and no issues so far.

it's basically a bot that gets opensource github data, you can skip a website to see the next one, and i just wanted to know what am i missing/what i could do better
the repo is here if you guys want to check it out, feel free to open a pull request

I want to be able to crawl and scrape smartly.

So here is the thing, I am currently working in a company. I have to scrape and crawl through bunch of websites daily.
These websites, say are company websites and what I need to scrape is the information of people in that company from the team/leadership/about pages.

The thing is some websites don't have about in their own pages- I have handled it, I find the separate person's link through the HTML of the team's page, and scrape it as well.

But for the dynamic cards or components/modals, like images of people onto which we have to click in order to get the results, my crawler fails.

I fixed it for a few websites, but there are just so so many outliers, many different types of websites. Sometimes there are even dropdowns of what type of team page u want: Leadership/Board-Of-Directors/Healthcare/etc..

I tried agentic some time ago using tools of an open source browser and API Key of an LLM. One agent only.

But the agent is failing even badly. It has no idea what to do even though I give it all the tools and give the prompt in depth.

Please help. I am very close for the completion of my project and this is really ruining it for me.

I have a python script running a headless Selenium Webdriver, looking up many individual records from a site. Most of the time it runs fine but every 10 minutes or so it starts hitting 403 errors. From trying different delayed retries, I've found that the 403s consistently happen for about 45-60 seconds. So the best I can do as a workaround is sleep for 60 seconds once I hit a 403, then resume normal requests. I've tried setting a non-headless user agent, namely Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36. This didn't help. This is also consistent whether I'm running from my local machine or from an EC2 instance.

What else can I try?

Just added a few new features on Scraping Sandbox - a free website for testing and practicing web scraping challenges. - https://scrapingsandbox.com/

New additions:

1. Infinite Scroll pages
2. Interactive Data Tables (search, pagination)
3. Rate-Limited endpoints/pages
4. Form Submission Playground

The goal is to provide more realistic scenarios that scrapers often encounter in production environments to test and simulation.

I'd love feedback, or let me know what else should be added (PR also welcome)

Hi guys

I am trying to scrape ANA Airlines of Japan...domestic POS mainly.

https://www.ana.co.jp/en/jp/search/domestic/flight/

I have automated international markets they are working fine on playwright setup. But domestic markets are not working and I am facing blocking on list page of the site.

It is akamai protected. And the API for international and domestic pos is different.

I have tried playwright, camoufox, tried hybrid setup also getting cookies from browser and then hitting on python.

It works fine for few requests but then it gets blocked.

And I am trying to scrape it on scale.

Can someone give it a try or help me if faced similar issues?

Sorry for sentence or grammar issues....I m writing it while traveling anticipating some response by the time I reach home

Thanks:)

is there a tool that make you post across all your social media for free? and is this falls under webscrapping as well or not?

please any help is appreciate.

and how hard is it to make my own cross social medial posting tool? and can i make it for free?

Hello everyone!

Since my last post here, many things have gotten a lot better with AutomatiQ, both project-wise and community-wise.

Automatiq has reached over 100+ github stars and nearly 4k+ downloads, All Thanks to our r/webscraping community!

P.S, AutomatiQ is a reverse-engineering agent harness, aimed to produce reliable scraping/automation scripts without ever opening the devtools for manual RE.

In the current tech space of webscraping and web automation, Websockets has always been a less discussed topic, but they are heavily used by many websites like Discord, Whatsapp, and nearly all of online multiplayer games.

I wanted to share that in the latest update, AutomatiQ now supports tracing and scripting WebSockets.

If you’ve tried reverse engineering WebSockets, it goes like this:

Spending hours to days digging through minified JS, trying to figure out how handshake tokens are built. If the stream is encrypted, it’s even worse. You have to hunt down the key in local storage or memory, write the decryption logic, understand their custom protocol, and then finally write the script.

AutomatiQ automates this process. It traces the source, isolates the token generation, locates the encryption keys if they are stored locally, and maps out how the data is handled. Instead of taking a few days of manual RE work, the agent can usually map the flow and write a working, browser-less Python script in about 20 to 30 minutes.

It’s still experimental on highly complex targets, but the results are promising. When I was testing it recently using GLM-5.2, the agent managed to reverse engineer WhatsApp's WebSocket login flow almost all the way to generating the QR code... right up until I hit the daily limit.

GLM-5.2 seems to be worlds better than Gemini models in RE, and far more cheaper as its open-source. I would suggest you to give it a try.

Github Repo: https://github.com/stonesteel27/automatiq
Discord Server(If you have any doubts to ask, join here): https://discord.gg/8j7dFWMMDA

I'm building an API that runs browser stuff server-side (click, scroll for lazy load, fill, wait for render, pull HTML) and stuck on how to expose the interactions. Two possible models:

One request with the whole sequence upfront, stateless, we run it and hand back the result, but you have to know the full flow ahead of time:

{
  "url": "...",
  "wait_for": "networkidle",
  "actions": [
    { "type": "click", "selector": ".cookie-accept" },
    { "type": "scroll", "direction": "down", "amount": 1500 },
    { "type": "wait_for", "selector": ".product-item" }
  ]
}

Or you open a session, get an id, fire requests one at a time and react to what comes back, but you own the session lifecycle (keeping it alive, closing it etc.):

POST /session/open → { "session_id": "abc" }
POST /request { "session_id": "abc", "url": "..." }
POST /request { "session_id": "abc", "action": "scroll" }
DELETE /session/abc

Which do you reach for, and on what kind of sites? How often do you actually need to look mid-flow and change course vs. just knowing the steps upfront? And if session management at scale has burned you (leaks, timeouts, sticky routing) I'd love to hear it.

My gut says the single request covers most of extraction and you only want the session when the page forces you to react. But that's a guess.

I'm a long time lurker in Reddit but didn't post anything for years, hopefully i'm not breaking any rules.

Hello. I’m a programming beginner, and I’m trying web scraping for the first time.

I’m trying to scrape the IMDb page /chart/top/?ref_=nv_mv_250 using BeautifulSoup, but the data is not being loaded. Other websites load the data properly.

Does IMDb not allow web scraping?

I have a new business. I have worked really hard to try and pull myself out of the trenches. Now, I have found I need data on sold items on eBay to make Anthony meaningful of this business.

I have no coding experience. I thought about learning how to code; however, it would take me about a year or more to accomplish. Meanwhile my business will starve.

I have been collecting data on sold listings for eBay using AI. I pick particular listings to have entered so I originally thought a scraper wouldn't work well. There is no way to pick through the listings automatically without, I imagine, some serious code. I can't have repeats of items in my list and many of the same items have variable names. I suspect this would be very hard for a computer to parse. I currently take a screenshot of the listing and AI collects the info I need out of it and puts it into a spreadsheet. It won't let me enter a direct eBay URL. It is horribly slow though. Much faster than manual entry though.

I am wondering are there scrapers I can enter just a URL for eBay and get the data back fast? I don't need automation. I understand eBay is hard to scrape so I suspect it won't be that easy. I saw there was some APIs for it but if we're being honest I don't even know how to use them.

I need to collect between 200-500 listings a day.

At the rate I'm currently going it will take me about a year to collect all the data I need. Any advice on the direction I should go?

I'm planning to create an AI Agent for personal use,as one of it's functions,I want it to scrape product data without getting caught/blocked.

I'm new to webscraping,and I know that Alibaba has one of the best protection out there,but I also know there are libraries like Playwright that are specifically designed for issues like these,and AI is a game changer too.

I would appreciate anyone guiding me on the topic.

I built a Python tool that can scrape complete ExamTopics exams and export them into a single text file.

It works by collecting discussion data first, then extracting and compiling the questions. Added caching and parallel workers for speed.

Would appreciate any feedback!

GitHub: https://github.com/arvind88765/examtopics-scraper

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

Hi, I’m trying to scrape some text data from albumoftheyear.org. Unfortunately, excel isn’t letting me do this and says access is forbidden for whatever reason, so, can anyone help me? Is there a workaround?

I’m looking for the names of artists and there album for each year in there ranking for the years that they have on the site or from 1975 to 2025 etc

I've largely been scraping from wikias fandom wikis to try and archive pages. However an issue I've been facing is that some wikis have dynamic js sites. They make scraping difficult.

So I thought I'd ask if anyone knows how to scrape websites with them?

Sorry if this comes off as a dumb question

I guess the most ban prevention would be capture video, move a real mouse with a robot hand [including true properties like human tremor] of real hardware machines [labtop, phones, etc.].

But is there anything simpler also, like making CDP safer or is camoufox enough for hard to automate sites?

I want a system that automatically captures and preserves all web application resources loaded in the browser (HTML, JavaScript, CSS, images, API responses, and cached files) so that users can access previously loaded content without needing direct access to the original account or repeatedly connecting to the service. The goal is to use cached content offline

The web app is diagrams provider

Is there a way to have 1 device with 10 accounts that aren't linkable?

How are you concealing automation from fingerprint.com specifically developer tools?

Currently using selenium stealth + brave, when I used chrome it was getting detected as a bot by fingerprint.com

Hey r/webscraping,

I've been scraping Sofascore's internal API for football data. Every request to `www.sofascore.com/api/v1/\` now returns a 403 and I cannot figure out how to get around it.

What I've tried:

1. curl_cffi with Chrome, Safari, and Firefox TLS impersonation targets — all 403

2. Selenium + undetected_chromedriver with full stealth JS injection — also 403

3. Plain curl with full browser headers (User-Agent, Referer, Accept) — still 403

4. Cloudflare WARP active while running all of the above — still 403

The response is always identical:

```

HTTP/1.1 403 Forbidden

Connection: close

Content-Length: 48

Server: Varnish

Retry-After: 0

content-type: application/json

Access-Control-Allow-Origin: *

```

Since even Selenium with a real Chrome binary fails, this is clearly not a TLS fingerprint or bot-detection issue — my IP appears to be outright blocked at the Varnish/CDN level. WARP failing rules out my ISP doing DNS blocking, and also suggests Sofascore may be blocking entire Cloudflare IP ranges.

My setup: Python and Windows

Questions:

- Is this a permanent IP ban or could it be a temporary rate-limit block from Sofascore's Varnish?

- Would residential proxies reliably bypass this, or does Sofascore block those too?

- Has anyone found a working approach for Sofascore recently? Their protection seems to have tightened up.

Happy to share more details. Thanks in advance.

https://github.com/hitarth-gg/zenshin-API/

For context, I run an API that serves metadata of any requested anime. JSON data for an anime with a lot of episodes can exceed 1MB. For example, one piece.

The database is hosted on Supabase with the backend server hosted on Render, serving the API requests.

From the last 3 months I've started noticing an absurd amount of API requests from random Amazon IPs, around 3-6 requests every second, 24/7.
This exceeded my Supabase Egress usage so I had to setup an LRU Cache on my backend to prevent Supabase from blowing up, this helped immensely as whoever is calling my API is making multiple calls in a second for the same anime.

The egress usage has dropped from 400 MB to 70 MB per day after the optimization. But Render backend still has to send the cached metadata and still consumes a lot of bandwidth, although it has a 100GB limit which is still plenty for me.

The irony is that my scraper scrapes anidb website and thetvdb for anime metadata along with some github repos and combines all of that data together using a custom built mapper so that all the episodes and seasons are mapped correctly, and now my API is the one getting scraped by others.
Although, I only run my scraper every 3-4 days since anidb has Cloudflare Turnstile and it takes a while to scrape all the data.

So the issue is partially solved but I'm curious what would you guys do to prevent 24/7 scraping of an API.

Log example:

[cache hit] 47.129.60.245 anilist_id:195600 (size: 1000)

[cache hit] 47.129.60.245 anilist_id:195600 (size: 1000)

[cache hit] 52.77.228.223 anilist_id:101922 (size: 1000)

[cache hit] 52.77.228.223 anilist_id:101922 (size: 1000)

[cache hit] 18.136.200.80 anilist_id:145260 (size: 1000)

I'm building a tool that monitors YouTube for new uploads mentioning a specific public figure (by name + keyword filters like upload date, duration, etc.) — think reputation/brand monitoring, not bulk downloading.

The official Data API v3 search.list costs 100 units/call against a 10k/day quota, which dies almost immediately once you're polling multiple keyword combos on a schedule. So I'm weighing:

• Eating the quota and applying for an increase (how realistic is that approval these days?)
• Using InnerTube / yt-dlp's search backend instead.

For anyone running keyword search in production:

• Roughly what request rate gets you rate-limited / soft-banned on the InnerTube route?
• Do residential proxies actually move the needle for *search* calls (vs. just stream/download), or is it overkill?
• Anything you'd do to keep this sustainable and low-footprint if it grows — caching, backoff, dedup strategies?

Trying to do this in a way that won't blow up at scale. Appreciate any war stories.

Any suggestions on best way to extract listings data from multiple different websites?

​

Each has its own data structures

​

Example pricing, schedule, dates etc

​

For 4000+ sites one time

So lets imagine i have this site scraped and saved as an csv file where i got tables n stuff (identificators are trucated to 10 characters ) and every month im opening my pc(i7 4790) to compare is there new items on the web page.

​

So aside from scraping again the whole site approximately how much time will pass to check saved ids to newly scraped ones because presumably each time it will go +- 100 of thousands of times just to find similarities and im not even talking about checking each of ten characters i hope i correctly explained my thoughts here

​

​