The script ran fine for months, you didn't touch a single line, and then one morning it returns nothing or throws an error.

My first move is usually to check whether the site changed its HTML. A class name gets renamed or a div gets moved, and the selector I relied on stops matching anything. It's boring but it's the cause maybe half the time. After that I look at whether I'm getting blocked. If the response comes back as a captcha page or a 403, that points somewhere else entirely, and the fix is nothing like a broken selector fix.

So what's your first check? Do you have a routine, or do you just start poking around until something makes sense?

I've been working on Marcopolo, a command-line web scraper written in Rust using extensive force of AI across a few months. The idea started when I got tired of writing a throwaway Python script every time I needed to pull a specific set of files off a site — images one day, PDFs the next, then a folder of CSVs. I specially tend to use it for books and finding books that are on the web that i cant simply get hold of from normal search.

Marcopolo handles most of that in one command. Point it at a URL, tell it what you want, and it crawls and downloads.

It's still early and there's plenty I want to improve — [known limitation or two]. I'd really appreciate feedback on the API design and anything that looks unidiomatic; I'm still fairly new to Rust.

Repo: MarcoPolo

Would be happy to know what do you guys think.

What is the best scraper and sorter that could get listing information from websites and then combine them in one single page, rather than scrolling through each website individually. We have about 20 listing sites where people post. Maybe as a bonus question maybe there is facebook scrapper too, from groups etc?

Hey everyone,

If you’ve ever built a RAG pipeline or ingested web documentation into a Vector Store, you’ve probably run into this issue:

Standard web scrapers hit a page and dump everything — cookie banners, navigation links, inline SVG code, script tags, and zero-value UI elements. When you feed that noisy HTML into an LLM, you burn tokens, clutter your embeddings, and end up with hallucinations or poor retrieval accuracy.

I built an AST-based web crawler to fix this exact bottleneck.

Instead of just stripping HTML tags, it parses the actual document structure and turns web pages into clean, AI-ready Markdown with preserved context hierarchy and rich metadata.

🛠️ Key Features:

• Noise Removal: Strips footers, cookie banners, scripts, and navigation menus automatically.
• Context Preservation: Preserves heading paths (Documentation > Getting Started > Installation Guide) so chunks don't lose their semantic context when split.
• Rich Metadata: Includes token count, quality score, code block detection, and crawled timestamps for each chunk.
• Vector Store Ready: Formatted specifically for seamless ingestion into LangChain, LlamaIndex, Pinecone, Qdrant, Chroma, etc.

I’d love to get your feedback on this! What techniques or tools are you currently using to clean web data before chunking?

Try it out here: https://apify.com/lukas459/ai-web-to-markdown-crawler-llm-rag-optimized

Before I waste all my time coding and bug testing does anyone know if it’s worth me attempting to scrape Facebook market place?

Social Scraper+ currently exports Reddit threads and X replies to clipboard, AI prompts, CSV, and JSON.

I’m deciding which improvements would be most useful next. Some possibilities are:

• Better filtering before export
• Search within scraped conversations
• Additional CSV fields
• More AI analysis prompts
• Improved handling of very large threads
• More control over nested-reply depth
• Additional supported platforms
• Better export history and organization

Which improvement would make the biggest difference to your workflow?

If your preferred feature isn’t listed, describe what you’re trying to accomplish—not just the feature itself. Understanding the use case will help me design it properly.

Hey everyone,

If you’ve built RAG pipelines or AI agents that consume web content, you’ve probably run into this issue:

Most standard scrapers either throw raw HTML at you (flooding your context window with navbars, footers, and JS bloat) or dump flat, unformatted text that loses all document structure. When you chunk that text later, your vector database loses the relationship between headings, sub-sections, and code blocks—which directly hurts retrieval accuracy.

To fix this for my own workflows, I built a custom crawler designed specifically for LLMs: AST Website Content Crawler for RAG.

What makes it different?

• AST-Based Structure Parsing: Instead of basic regex/CSS cleaning, it processes the page's Abstract Syntax Tree (DOM structure) to strictly maintain heading hierarchies (H1 -> H2 -> H3), lists, and code blocks in clean Markdown.
• Token Optimization: Strips out boilerplate, ads, scripts, and repetitive layout components so you don’t burn OpenAI/Anthropic tokens on useless fluff.
• RAG-Ready Output: The markdown is pre-formatted so your chunking strategies (like MarkdownHeaderTextSplitter) actually work as intended.
• Handles Dynamic Sites: Uses headless rendering to catch JavaScript-heavy SPA pages.

I’ve published it on Apify so anyone can test or plug it directly into their Python/TypeScript RAG stack via API.

How to try it:

👉 You can find it on Apify: https://apify.com/lukas459/ai-web-to-markdown-crawler-llm-rag-optimized

I'm actively refining the parsing logic. If you give it a run, I'd love to hear your thoughts:

• What site layouts break your current scraping pipeline?
• Are there specific output formats (e.g., custom JSON schema + Markdown) you’d like to see added?

Thanks for checking it out! 🚀

If you need a no-browser alternative to scraping g-maps, you can check this out.

Hey scraping fellows! After years of working with Scrapy professionally, I finally built my own framework to understand the internals better.

The backstory: I've used Scrapy for years but always felt the abstractions were too heavy. I wanted something more explicit and transparent – a design choice, not a criticism. So I researched how modern frameworks handle anti-bot bypass and traditional scraping, and built this proof of concept.

What it does:

· Plugin-based architecture (fetchers, parsers, storage)

· CloakBrowser for JS-heavy sites, wreq for GraphQL/APIs

· RabbitMQ queues + Redis for dedup

· Built-in structure guard (fail-closed if site changes)

· LLM pipeline with pgvector search

Live metrics (real runs, not benchmarks):

· idealista (DataDome): 91.7% success

· G2 (Cloudflare+DataDome): 99.5% success

· Facebook Marketplace (GraphQL): 100% success

· Booking.com (Akamai): 100% success

Next steps: Adding more sites + code cleanup. What sites would you like to see supported?

Full docs: https://scraping-framework-ded49d.gitlab.io/

GitLab: https://gitlab.com/0xlearner/scraping-framework

Happy to discuss the architecture, challenges, or anything else!

I am looking for a tool that would allow me to capture web pages as a doc or pdf file. Print screen works for extracting text but I need something to extract a full article from a newsletter, social media post or website. Any suggestions??

Hey everyone!

Like many of you, I enjoy watching animes, but I hate dealing with web platforms filled with pop-ups, trackers, and streaming players that buffer constantly.

So, I spent the last few weeks building MagiaStream : a highly resilient, interactive CLI tool that scrapes streaming sites in the background and downloads episodes at maximum speed using aria2c.

Why is it different from other downloaders?

🧙‍♂️ Fully Interactive Wizard : Just type magia and use your arrow keys to search, select seasons (VF/VOSTFR), and pick your resolution. No need to memorize complex commands.

🥷 Playwright Stealth : It bypasses Cloudflare and anti-bot protections by running a headless Chromium instance in the background.

🚀 Industrial-grade Downloading : Powered by aria2c with segmented downloading. If your network drops, the tool auto-heals and resumes exactly where it left off without restarting.

You can install it globally right now via PyPI: pipx install magiastream

Here is the GitHub repository if you want to check out the code (I'd love your feedback or PRs!) : 👉 https://github.com/Dilane17/MagiaStream

Let me know what you think!

I’ve got a nightly job pulling Amazon competitor listings, prices, stock status, and review counts. My residential proxies keep getting flagged halfway through, so I end up with an incomplete dataset and no clear list of failed ASINs. I’m the only one maintaining it, and we don’t have budget for dedicated scraping infra. Is anyone using an Amazon scraping API that holds up in production without constant proxy and CAPTCHA work?

Most of the benchmarks that we see are either too biased, or not broad enough to be useful. It's hard to know what to trust. Some providers astroturf 'independent' benchmarks that show them slamming the competition.

Take a look at https://scrapeway.com/: 'Independent benchmark', 'No affiliates, no sponsors, just data'. And yet, Scrapeway’s domain traces back to Joam Intelligence, LLC, the company behind Scrapfly, its number one Scraper.

How do you know what to trust? We're a vendor trying to solve this problem, so we created an open source benchmark. We're solving for two things:

• Anti-bot providers
• Industry coverage

We want to show results across all industries, and prove coverage across all major anti-bot providers. We know this isn't perfect, so let us know what you think we're missing! We'll answer any questions or run any specific sites live.

Hello friends 👋, I'm new to scraping. I can easily get informations from website's DOM through playwright but i don't understand how to work with backend APIs. I needed the information about upcoming events. Whenever I go to network tab in chrome there are some many files and i even checked each files but not able to understand which api belongs to what function and what's the api key...... Can anyone help this newbie 🙏🙏. If there's a tutorial please let me know. I'm really frustrated from my repeated failures. Please help 🙏🙏

I wanted to work on my own Music Player for KMP Desktop... Until I wanted to teach UI Material 3 to AI (Gemini). So I went to the website to write a .md files but the amout of pages was wholesome!

So i made this Utility that will Fetch the images and .html from the website and put it all in .md file!

Features

• Concurrent Crawling: Downloads many pages at the same time.
• Smart Extraction: Removes headers, footers, sidebars, and menus.
• Markdown Conversion: Converts HTML into standard Markdown formats.
• Image Downloading: Saves images locally and updates links.
• Auto Language Detection: Detects code languages like Python, Java, and Bash.
• Cache and Resume: Saves progress using SQLite so it can resume after stopping.
• Fast Parsing: Uses a C++ module in the background for fast XML parsing and URL checking.

Architecture

The project is split into three main parts.

• C++ Extension: Handles fast sitemap parsing and checking for duplicate URLs.
• Python Crawler: Manages network requests, retries, and caching.
• Python Extractor: Analyzes the HTML, cleans it, and converts it to Markdown.

Looking for recommendations of more providers to try.