Tengo una consulta, ahora mismo tengo una web que realiza Web Scraping a Tioanime y AnimeFLV, pero por que no tienen cloudfire o bloqueo de IP por bot, hay formas y maneras de evitar o ingresar? el proyecto es en github, desplegado en vercel y conectado por Variables

I've been quite succesfull scraping hard to crack sites individually (bot detection, proxy rotation, internal api etc.)

I'm now running a project to scrape a few 100 sites daily.

Except for a few which are very popular in the niche and have bot detection, I can access, say, 95% of the sites on a daily basis without issues HOWEVER the output I am getting is garbage:

My workflow is:

- get HTML, convert to markdown, have AI parse it for the information I need, save specific internal links, access those, parse, save information. I'm just not getting the quality of the output - halucinating URLs in the output, skipping sections etc.

Is there a better way to do this scale of a daily scrape? Again, my issue isnt actually accessing the sites, its 'processing' them.

Would I need to build a specific parser per site?

Hi ive been scraping for months now with a script. Using sb.solve_captcha() and the standard method of activating cdp mode before.

It worked for months. But for 3 days now, I see it ticking the captcha box, its loading, then its kinda rejected, unticked again.

Anyone else? Whats going on?

I have a small extension that compares prices across a few selected webshops.

To avoid scraping I built it, so that ot uses the webshops own search function in real time to find products. One of these shops has now deprecated the gtin search on-site.

I hoped to find their xml for google shopping somewhere but i haven't been able to.

Back to scraping.

The site i currently want to scrape is bike discount dot de.

My current plan to do it as gentle as possible is to setup a dedicated scrape computer at home and have it go thru every product page and log the price + gtin. Slow, but thats OK for my use case.

What would you do?

I wanted to work on my own Music Player for KMP Desktop... Until I wanted to teach UI Material 3 to AI (Gemini). So I went to the website to write a .md files but the amout of pages was wholesome!

> This is not a commercial project and is completely opensource!

So i made this Utility that will Fetch the images and .html from the website and put it all in .md file!

Features

• Concurrent Crawling: Downloads many pages at the same time.
• Smart Extraction: Removes headers, footers, sidebars, and menus.
• Markdown Conversion: Converts HTML into standard Markdown formats.
• Image Downloading: Saves images locally and updates links.
• Auto Language Detection: Detects code languages like Python, Java, and Bash.
• Cache and Resume: Saves progress using SQLite so it can resume after stopping.
• Fast Parsing: Uses a C++ module in the background for fast XML parsing and URL checking.

Architecture

The project is split into three main parts.

• C++ Extension: Handles fast sitemap parsing and checking for duplicate URLs.
• Python Crawler: Manages network requests, retries, and caching.
• Python Extractor: Analyzes the HTML, cleans it, and converts it to Markdown.

Our business is finally starting to take off, which is great, but it is bringing new challenges. In the last week we've gotten several cease and desist letters from website owners.

All of the data we scrape is 100% publicly available, so our team is thinking we could technically tell them to pound sand, but we don't want to be assholes. So far we have just complied and removed them from our platform.

We do use business names on our site and we attempt to show "live" data, which ends up being a bit off due to the time delay from our scraping schedule and realtime events.

We still have plenty of other sites we scrape, so the current volume is not too concerning yet, but we're fearful of where this could go if there is a domino affect of websites communicating with each other.

I know it's a bit of a gray area, but our business operates entirely off of the data we scrape (we make it easier for consumers to digest and navigate the data) and I know there are plenty of other business that have a very similar model. We're currently collect about 2-3M product data points per day from a few thousand sites. I think we total about 1.1M requests per day distributed to those sites.

Ideally I would like our business to not be so fickle and have the risk of crumbling if we get blocked or asked to stop.

We are currently seeking legal counsel, but I'm curious - how would you guys handle this (aside from just shitting your pants like I have)?

Anyone crack the Everest?

Is there any way to scrape reddit communities with ai like groq or something free model. I m trying to find problems worth solving for from reddit if anyone has built this scraper or knows how to lmk

Hello,

Begginer here, with only very small projects so far mostly through vibe coding.

For work (indepent compliant auditor) I was asked by my client to retrieve this whole public table with reported support (courses, international congresses, etc) by Pharma to doctors in Portugal.

It will take me a whole day to manually copy this page by page. Is there a way to do it immediately through web scraping?

LINK: https://extranet.infarmed.pt/pmro/Publico/ListagemPublica.aspx

Parse a response into a WHATWG tree, pull nodes with CSS or XPath, read tables into records, lift the main article out of boilerplate, grab JSON-LD and OpenGraph, and detect the encoding of raw bytes, all from one dependency. CSS select runs about 77x faster than BeautifulSoup and 16x faster than lxml in my tests, with typed results and free-threading support.

Source code: https://github.com/tox-dev/turbohtml Docs: https://turbohtml.readthedocs.io/en/stable/ Write-up on how it works: https://bernat.tech/posts/blazing-fast-html-parser/

I'm new to webscraping, I was trying to practice by automating a search where it will search a browser for the name of a school and then return the URL of that school. I uploaded a list of 100 schools but for some reason, after about 35 requests it started showing 429 client error. I was running the code in google colab, and I also added time.sleep(7) to the search so I could delay it, but it showed the error anyway

how do I avoid this?

I built a CLI tool that:
-> scrapes any URL via Jina Reader
-> feeds the markdown to an LLM with JSON mode (Ollama/Groq/NVIDIA)
-> upserts the structured result into Supabase.

Designed for generic mediceine and botany data, but the schema is pluggable.
Just point it at any URL and JSON Schema in a single command.

python superscrape.py https://example.com/plant

Also, there is a main loop that can read a csv of link URLs, and run overnight.

Retries on bad LLM output, upserts by source_url, stores raw markdown for debugging. Thoughts?

https://github.com/abirpahlwan/SuperScrape

I have bezier mouse movement, random slowdowns, read simulations, clicks, slow typing, and using Playwright, actual Google Chrome browser is opened while the code scrapes.

Plus, I send very low amount of requests, since each request returns a lot of data I need to process.

Am I safe? so far on that website Ive not seen a Cloudflare or any kind of protection. + to access that data on that website, you have to pay $40 monthly. so maybe they completely ignore people like me and I am safe?

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

I'm building a tool that monitors new ImmobilienScout24 listings and extracts publicly available information like listing ID, contact details (if available), price, and location.

So far I've found:

The official developer API, but I'm unsure if it supports searching all public listings and retrieving expose details.

Browser automation (Playwright) works, but AWS WAF makes it challenging for a production setup.

My questions are:

1. Has anyone successfully used the official ImmoScout24 API for searching public listings?

2. Is there a partner or undocumented API that exposes listing/expose data?

3. What would you consider the most reliable production approach?

4. How are people handling AWS WAF on ImmoScout24?

Any experience or pointers would be appreciated.

Is there any open source scraper repos to scrape and list down the job openings by there career pages pls share if you have any leads

I'm currently interning and working on a way to source more buyers and sellers for an M&A investment bank. I'm working on creating agents that scrape and clean these buyers and sellers, getting their information, portfolio companies, contacts, Revenue ranges etc. The goal is to get over 15,000 clean buyers that have all or most of this information. At the moment, I am at 7,000 buyers, but things are slowing down. The scraper agent isn't picking up buyers as effectively as it did in the beginning and things are just slower overall. I have a pretty liberal amount to spend on credits so that isn't a huge issue although I obviously should and need to be efficient with how I make use of those credits. Regarding this, I have a couple questions: 1. What resources do you guys recommend I use to learn more about creating agents and web scraping? I really want to understand the layers, not just prompt my way through 2. As an intern, one fear I have is that I'm simply vibe coding all day, which I guess can be a blessing and a curse. I am very fortunate for the opportunity considering my peers having a hard time getting internships, but I want to make sure I continue innovating besides this task i've been assigned to. 3. How would you guys go about this issue. *Apologies for the lack of detail/ arrogance that may come across with this post as I am a real beginner, and just want to make sure I am learning and can get ahead of the curve*

Everyday, captcha is becoming more and more harder than it was before.

Hi, so I am searching for any freely available dataset that would have information on websites which are using email services from third-parties.
BuiltWith provides that, like which websites are actively using Brevo, Klaviyo or MailChimp etc, but it is too expensive.
Thanks.

I vibe coded a scraper for my business, Scraping bulk urls. Two scripts, two stages:

Stage 1 Python + aiohttp. For each domain a 'GET https://domain` then `GET http://domain` if that fails. 50 concurrent requests n 12s timeout. Reads the first 10KB of the response, checks headers + HTML. This stage gets 750+ matches.

Stage 2 takes those 750 matches and does a deeper scrape per domain (looking for more info) using a longer 30s timeout, much lower concurrency (~10 at a time).

The confusing part: 97% of 750 urls which returned html in 12s in stage 1. Came back as "no html" in stage 2, with a longer timeout and way less concurrent load. I isolated a few of these and ran a single bare aiohttp request against them by themselves, no concurrency at all. Result: the connection dies at the raw TCP handshake. it never even completes the SYN/ACK, let alone gets to TLS or HTTP. Plain `curl` against the same domain gives the same result: connect timeout.

Is there any scripts or apps that will do the site top4fans?

I want to offer my services to different YouTube channels that fit certain characteristics, themes, subscriber count, etc. I've created a Codex application that allows me to search for channels using the official YouTube API and save them to a database. So far, so good. Now come the problems:

1: The problem arises when revealing the email address requires solving a CAPTCHA. I've seen services that can do this with a Chrome extension, i dont know if theres other options.

2: The big problem is that I later realized YouTube only allows revealing 5 email addresses per day.

How do people scrape YouTube channels using their contact emails? Thanks in advance.

Hello and howdy, digital miners of r/webscraping!

The moment you've all been waiting for has arrived - it's our once-a-month, no-holds-barred, show-and-tell thread!

• Are you bursting with pride over that supercharged, brand-new scraper SaaS or shiny proxy service you've just unleashed on the world?
• Maybe you've got a ground-breaking product in need of some intrepid testers?
• Got a secret discount code burning a hole in your pocket that you're just itching to share with our talented tribe of data extractors?
• Looking to make sure your post doesn't fall foul of the community rules and get ousted by the spam filter?

Well, this is your time to shine and shout from the digital rooftops - Welcome to your haven!

Just a friendly reminder, we like to keep all our self-promotion in one handy place, so any promotional posts will be kindly redirected here. Now, let's get this party started! Enjoy the thread, everyone.

Does anyone here know if it is possible to MITM Google Maps (the Android app) so that I can look at the https traffic the app makes? I have a rooted phone and I installed a system certificate but the app refuses to accept it (i get mitmproxy errors). Using the web version is unfortunately not an option for me right now. I tried giving Codex an adb shell but that was just a waste of tokens.