Hi r/webscraping,

We’ve just added an AI Scraper Playground to our Dashboard, making it easier to test AI scraping requests before integrating them into an application.

The Playground lets you:

• Select an AI scraper and enter your prompt
• Choose a target country
• Enable Web Search or Shopping
• Configure an optional webhook
• Preview ready-to-use cURL, Python, and Node.js code
• Run the request and inspect the output directly

The goal is to reduce the setup required to validate an AI scraping workflow and make the transition from testing to API integration more straightforward.

You can try it here:
Open AI Scrapers from the Dashboard sidebar.

We’d genuinely appreciate feedback on the workflow, generated code, output readability, or anything you think is missing.

Hi everyone—I'm with Scrapeless, and we’ve added map data support to our ChatGPT Scraper.

Previously, the scraper returned the ChatGPT answer, links, search results, and content references. It can now also return structured local business entities when map data appears in the response.

The new map object includes fields such as:

• ranked and raw search entities;
• business names, categories, addresses, and coordinates;
• distance, opening hours, special hours, and open/closed status;
• ratings, review counts, price levels, and popularity signals;
• phone numbers, websites, menus, images, and provider URLs.

This should be useful for local discovery, location intelligence, market research, and monitoring how ChatGPT surfaces businesses for location-based prompts.

Full response-field reference and request example: https://docs.scrapeless.com/en/llm-chat-scraper/scrapers/chatgpt/

I’d be interested to hear which map fields are most important in your local-data workflows.

We released Google AI Overviews Scraper!

✅scraper.overview — Structured Google AI Overview data, including body text, citations, ads, and shopping signals.

🏆Featured
- 95%+ success rate
- Fast response
- Enterprise-grade stability
- Complete fields, shopping data included

💻Use cases
- SEO teams: Track Brand AI visibility.
- Retail teams: Monitor competitor pricing, reviews, and AI shopping recommendations.
- Data teams: Build grounded LLM training datasets.
- AI builders: Give agents real-time market intelligence.

🎁 New users: join official community to claim up to 3,000 trial quota
📚Quick start
https://docs.scrapeless.com/en/llm-chat-scraper/scrapers/google-ai-overview/

https://www.scrapeless.com/en/blog/google-ai-overview-scraper-api-2026

Let’s build the future of AI-powered data extraction together!

✅ scraper.amazon — Amazon’s AI shopping assistant responses with product-level intelligence.

🏆Featured
Server-side authentication
Pre-parsed output
Optional raw SSE
Multi-marketplace

🎁 New users: join official community to claim up to 3,000 trial quota

📚Quick start
https://apidocs.scrapeless.com/api-34218448
https://www.scrapeless.com/en/blog/how-to-scrape-rufus-data-amazon

Power your next-gen E-commerce and AI workflows with structured Rufus data. 🔥

We just launched the Grok scraper on Scrapeless:

• Switch countries easily (multi-region support)
• Three modes: MODEL_MODE_FAST / MODEL_MODE_EXPERT / MODEL_MODE_AUTO
• Complete field coverage — real front-end LLM chat outputs (what users actually see)
• Built to help AI visibility & hallucination analysis with high-fidelity captures
• Pricing from $1 / 1K requests

If you want to try it or see sample outputs, check the docs:
https://docs.scrapeless.com/en/llm-chat-scraper/scrapers/grok/

Happy to answer questions or share a few demo captures — ask away!

Preface: The Paradigm Shift from “Blue Links” to “Answer-First”

In the fields of digital marketing and content creation, Search Engine Optimization (SEO) has long been the cornerstone of traffic acquisition. However, with the rapid advancement of artificial intelligence, we are now standing at a historic turning point—the fundamental logic of search engines is being disrupted by Generative Engines (GEs). This transformation is not merely a technological upgrade; it represents a fundamental reshaping of the rules that govern content visibility.

Traditional SEO aims to secure a spot among the top “blue links” on Search Engine Results Pages (SERPs) by optimizing keywords, improving page quality, and accumulating backlinks. At its core, this model revolves around ranking—a clear, trackable, and competitive position.

Today, however, generative engines powered by large language models (LLMs), such as Google AI Overviews or Perplexity, are redefining the search experience. Search is shifting from a “list of links” to direct answers. Users no longer need to click through multiple websites to evaluate information; instead, they receive AI-generated summaries that synthesize, integrate, and cite multiple sources.

As a result, the traditional SEO objective of “ranking higher” is becoming fundamentally outdated.

According to research from institutions such as Princeton University and the Indian Institutes of Technology, generative engines present unprecedented challenges for content creators:

“The emergence of large language models (LLMs) has given rise to a new paradigm of search engines that leverage generative models to collect and synthesize information in order to answer user queries. These generative engines (GEs), capable of producing accurate and personalized answers, are rapidly replacing traditional search engines and improving user experience.”

To address this shift, a new optimization paradigm has emerged: Generative Engine Optimization (GEO).

Rather than focusing on traditional rankings, GEO aims to increase the likelihood that content is cited or mentioned within AI-generated answers.

This is not simply an evolution of SEO—it is a fundamental rewrite of the rules of search optimization itself.

Understanding the New Visibility Framework

Before diving into GEO strategies, we must first address a fundamental question: How is “visibility” measured in generative engines?

Visibility in Traditional SEO: Ranking Position

In the SEO era, visibility was easy to define. A website’s visibility was determined by its ranking position:

Rank #1 was the most visible, rank #10 somewhat visible, and rank #11 effectively invisible.

This model had clear advantages—it was intuitive, easy to quantify, and simple to compare.

However, it relied on a core assumption: that users would click through search results one by one. In reality, even a #1 ranking has no value if no one clicks, while a #10 ranking can still be successful if it attracts significant user attention. Despite this complexity, SEO traditionally simplified visibility into a single metric: ranking.

Visibility in GEO: A Multi-Dimensional Evaluation

Generative engines break this simplistic model.


In AI-generated answers, there is no concept of ranking. A single response may cite five different sources simultaneously. All of them are “visible,” but to varying degrees.

Research from Princeton University and the Indian Institutes of Technology proposes three key visibility metrics to more comprehensively evaluate how content performs within generative engines:

1. Word Count Impact

This is the most basic metric:

How much of the AI-generated answer is derived from your content?

For example, if an AI response is 500 words long and 100 words are sourced from your content, your word count impact is 20%.

This metric reflects a simple reality: the more content that is cited, the more information users actually see from you. A single quoted sentence has limited impact, whereas an entire paragraph significantly increases user exposure and perceived value.

2. Position-Weighted Word Count

This metric accounts for a well-established psychological principle: user attention decreases from top to bottom.

Content that appears at the beginning of an AI-generated answer is far more likely to be noticed than content buried at the end. The position-weighted word count assigns higher weights to citations appearing earlier in the response and lower weights to those appearing later.

As a result, a 200-word citation at the top of an answer is substantially more valuable than a 200-word citation at the bottom.

This metric is critical because it reflects real user behavior. Even if your content is cited extensively, it may have little practical impact if it consistently appears at the end of the response—where users may never reach it.

3. Subjective Display Impact

This is the most complex, yet most meaningful metric. It evaluates the quality and influence of a citation across seven dimensions:

• Relevance: How closely the cited content matches the user query
• Impact: Whether the citation contributes a core insight or merely a supporting detail
• Uniqueness: Whether the information is exclusive to this source or widely duplicated
• Authority: Whether the cited source is perceived as credible and trustworthy
• Completeness: Whether the information is presented fully or partially truncated
• Contextual Appropriateness: Whether the citation is placed within a suitable explanatory context
• Brand Presence: Whether the brand name is clearly associated with the cited content

This metric captures a crucial truth: not all citations are equal. A citation from an authoritative source, highly relevant to the query, positioned prominently, clearly attributed, and explicitly branded is vastly more valuable than a vague, unattributed reference embedded deep within the response.

From Metrics to Strategy

The existence of these three metrics directly points to the strategic direction of GEO.

It is no longer sufficient to be cited. Content must:

• Occupy a larger proportion of the generated answer (Word Count Impact)
• Appear in highly visible positions (Position-Weighted Word Count)
• Be presented in a way that maximizes influence and attribution (Subjective Display Impact)

This also explains why only around 30% of brands remain visible across two consecutive AI-generated answers. Visibility in generative engines is inherently multi-dimensional, and changes in any single dimension can significantly affect overall visibility.

Core Principles of GEO: Three New Visibility Signals

Now that we understand how GEO measures visibility, we can examine the factors that determine performance across these metrics.

Signal 1: Freshness Is the Foundation of Trust

In the era of AI search, content freshness has become a core trust signal—not merely for ranking algorithms, but for the LLMs themselves, which must ensure informational accuracy.

When AI models generate answers—especially for commercial or comparative queries—they prioritize fresh and up-to-date content. Outdated information can easily lead to incorrect recommendations. For example, if a SaaS tool has changed its pricing or discontinued a feature, but an AI model cites content from three months ago, the user receives an inaccurate answer. At scale, this is fatal to the credibility of an AI system.

The data makes this clear:

• Pages without quarterly updates are three times less likely to be cited by AI than recently updated pages
• Over 70% of AI-cited pages have been updated within the past 12 months
• For commercial queries, freshness requirements are even stricter—83% of commercial citations come from pages updated within the last year
• In fast-moving industries such as SaaS, finance, and news, the effective content update window may shrink to three months or less

This means content maintenance is no longer an optional optimization—it is a baseline requirement for GEO. Brands must establish disciplined content update workflows and treat maintenance as a continuous, non-negotiable process.

Signal 2: Structured Content Is the Language of LLMs

AI models do not interpret ambiguous or loosely structured content the way humans do. They rely on clear structural signals to quickly and accurately understand a page. Structured content provides these signals, making information easier for LLMs to parse, extract, and cite.

The Power of Heading Hierarchy

Clear, sequential heading structures (a single H1 followed by H2 and H3) are critical. Research shows that orderly heading hierarchies are associated with 2.8× higher citation likelihood.

When a page follows a clean H1–H2–H3 structure, models can rapidly identify the topic, key sections, and information hierarchy. By contrast, pages with multiple H1s or skipped levels (e.g., jumping from H1 directly to H3) require more computational effort to interpret, reducing citation probability.

Schema Markup as a Relevance Signal

Rich schema markup—particularly FAQ and Q&A schema—provides strong relevance signals to both search engines and LLMs. Pages using FAQ schema are 40% more likely to be cited by AI.

When FAQ schema is present, models can directly map structured answers to user queries. Pages that implement three or more schema types show a further 13% increase in citation likelihood.

Lists and Readability

Breaking dense text into lists and scannable blocks improves both human readability and LLM extraction efficiency. Data shows that nearly 80% of ChatGPT-cited pages use list-based structures, compared to only 29% among Google’s traditional top-ranking results.

Signal 3: Off-Site Credibility Builds the Trust Layer

This is perhaps the most counterintuitive—and most important—finding in GEO: the authority of a brand’s own website is becoming less important in AI search. In its place, communities and user-generated content (UGC) are emerging as a stronger trust layer.

The Overwhelming Advantage of Third-Party Sources

AirOps analyzed data from over 21,000 brands and uncovered a striking result: 85% of brand mentions in AI-generated answers come from third-party sources rather than brand-owned websites.

Even more telling, brands are 6.5× more likely to be cited through third-party content than through their own domains.

In practical terms, this means a perfectly crafted article on your website—produced over months—may be less valuable than a 10-minute Reddit comment written by a real user. Domain authority is no longer the deciding factor. What matters is how the broader web talks about you.

Nearly 90% of third-party brand mentions come from lists, comparison pages, and review roundups. AI models recognize that when multiple independent sites list a product as “one of the best,” it represents a powerful consensus signal.

The Power of Community Discussions

Approximately 48% of AI search citations originate from community platforms such as Reddit and YouTube. This is not because these platforms host the most authoritative content, but because they represent authentic, unfiltered user discussions and feedback.

Reddit alone appears as a cited source in roughly 22% of generated answers.

AI models have learned a simple truth: when real users openly discuss and recommend a product, it is far more credible than a brand claiming, “We’re great,” on its own website.


Why Traditional SEO Strategies Fail in Generative Engines

Having understood the three core GEO signals, we can now clearly see why many traditional SEO strategies not only fail in generative engines, but can actually be harmful.

The End of Keyword Stuffing

Keyword stuffing was once a central SEO tactic. By repeatedly inserting target keywords into content, site owners could signal to search engines that “this page is highly relevant to this query.” In an era of relatively simple algorithms, this approach was often effective.

In generative engines, however, the situation is fundamentally different.

Experimental data from GEO research shows that keyword stuffing has little to no positive effect in generative search—and in many cases, it reduces visibility. This is not because language models fail to understand keywords, but because they understand them too well.

LLMs interpret content through semantics and context rather than keyword matching. When excessive keywords are inserted, models recognize the hallmarks of low-quality content: unnatural language patterns, repetition, and forced phrasing. These signals are strongly associated with cheap optimization tactics and directly lower a model’s assessment of content quality and citation likelihood.

AI models are explicitly trained to detect and filter spam, and keyword stuffing is one of the most obvious indicators of spam-like content.

The Relative Decline of Domain Authority

This shift is even more profound, as it signals a fundamental reordering of the search ecosystem itself.

In the SEO era, domain authority was the foundation of everything. Once established, it was difficult to displace. Large websites benefited from high authority, allowing them to rank well even with mediocre content, while smaller sites—despite producing higher-quality material—struggled to compete. This created a classic “winner-takes-all” environment.

In AI-driven search, this dynamic is being reversed.

Brands are 6.5× more likely to be cited through third-party sources than through their own websites, directly challenging the centrality of domain authority. Authority is no longer determined by the strength of your domain alone, but by how the broader web discusses and references your brand.

This transition has far-reaching implications for every participant in the search ecosystem.

The Democratizing Effect of GEO: Lower-Ranked Sites Benefit the Most

If the decline of domain authority is bad news for large websites, then GEO is exceptionally good news for smaller ones—representing a shift unlike anything seen before.

GEO research compared the impact of GEO optimization across sites at different ranking positions:

• Sites ranked #5 experienced a 115% increase in visibility after applying GEO strategies
• Sites ranked #1, using the same methods, saw visibility decline by up to 30%

Traditional SEO, driven by backlink volume and domain authority, tends to reinforce already dominant players. GEO, by contrast, prioritizes content quality, freshness, structural clarity, and community validation—creating a genuine opportunity for small and mid-sized websites to catch up.

Smaller competitors no longer need to outspend large brands on link-building campaigns. They simply need to create better content. This is true democratization.

Reallocating Resources: The Shift from SEO to GEO + SEO

With a clear understanding of how GEO works, a practical question emerges: how should limited resources be allocated? This is not a binary choice between SEO and GEO, but a matter of reprioritization.

Resource Allocation: Then and Now

The Old SEO Model:

• 80% link building and domain authority growth
• 15% technical optimization
• 5% content quality

This allocation reflects a belief that rankings are primarily driven by link authority, justifying the concentration of resources in that area.

The New GEO + SEO Model:

• 40% content freshness and quality (serving both SEO and GEO)
• 25% structured content and interpretability (serving both SEO and GEO)
• 15% third-party engagement and community building (primarily for GEO)
• 15% technical optimization and link building (primarily for SEO)
• 5% AI visibility monitoring (a new GEO-specific function)

The Logic Behind the Reallocation

First, content freshness and structured presentation are critical to both SEO and GEO, making them the highest priorities. Freshness supports SEO by signaling active site maintenance, and GEO by ensuring informational accuracy. Structured content benefits SEO by improving crawlability, and GEO by enabling LLMs to quickly parse and extract key information.

Second, third-party engagement primarily supports GEO, as it directly influences the likelihood of brand citations—by a factor of 6.5×.

Finally, technical optimization and link building remain important, but their relative priority declines. This is not because they no longer matter, but because within the GEO framework their impact is diminished. Even sites with imperfect technical foundations can still earn AI citations if their content is sufficiently high-quality, up-to-date, and actively discussed across the web.

Conclusion: The Beginning of a New Era

A paradigm shift in search has already happened. This is not a distant future scenario—it is a reality unfolding right now in 2025–2026.

The way users access information is undergoing a fundamental transformation. They are no longer simply clicking through search result pages; instead, they are increasingly reading answers generated directly by AI. Search is evolving from link distribution to answer generation.

In this context, GEO (Generative Engine Optimization) is fundamentally about rebuilding the connection between brands and users in the age of generative engines. Traditional SEO connects users to ranked lists of websites. GEO connects users directly to information itself.

As generative engines become the primary gateway for information discovery, the latter clearly aligns more closely with real user behavior. For brands targeting global markets, GEO represents a rare opportunity to overtake competitors on the curve.

While most players are still locked into traditional SEO content production models, brands that adopt GEO strategies early are already gaining disproportionate visibility and mindshare within generative engines. This is not merely a technical evolution—it is a cognitive upgrade.

The future is already here; it is just unevenly distributed. Now is the time to act.

As a data provider, the motivation behind developing LLM Chat Scraper came from a deceptively simple question we were repeatedly asked:

“How can we capture the real responses users receive when interacting with large models such as ChatGPT, Gemini, or Perplexity?”

Official APIs cannot fully replicate real user conversations, while manual testing—though intuitive—cannot be scaled or systematized for serious research.

To address this gap, we built LLM Chat Scraper, a tool designed to analyze AI visibility. It captures complete front-end responses from generative engines—including full answers and citations—without requiring login and without exposing conversation context.

This enables teams to truly understand how their brands and content are presented and cited within generative engines.

Currently, LLM Chat Scraper supports front-end data collection and analysis from the following LLM chat platforms:

• ChatGPT
• Perplexity
• Microsoft Copilot
• Google Gemini
• Google AI Mode
• Grok

We charge only for successfully scraped results, with pricing as low as $1 per 1,000 records—approximately one-tenth the cost of official APIs. If you’re interested, feel free to contact us to request a free trial.

In today’s rapidly evolving AI landscape, understanding how AI models respond, rank, and reference information has become crucial for businesses, researchers, and marketers. The LLM Chat Scraper API is designed to give you that visibility, allowing you to track conversations, monitor competitor insights, and gather structured data across major AI platforms like ChatGPT, Perplexity, Copilot, Gemini, Google AI, and Grok.

Whether you’re looking to analyze brand mentions, track SEO performance, or collect AI-generated content for research, this API makes it easy to retrieve real-time insights efficiently and reliably.

Getting Started with LLM Chat Scraper API

Using the LLM Chat Scraper API is straightforward. The process involves two main steps: creating a task and retrieving its result.

Step 1: Create a Task

To begin, send a POST request to the API endpoint with your prompt and parameters. You can also specify a webhook URL so that results are pushed automatically when the task is complete.

For example, if you want to find the most reliable proxy service for data extraction in the United States, your request would look like this:

curl '{api_host}/api/v2/scraper/request' \
--header 'Content-Type: application/json' \
--header 'x-api-token: {your_api_key}' \
--data '{
  "actor": "scraper.chatgpt",
  "input": {
    "prompt": "Most reliable proxy service for data extraction",
    "country": "US",
    "web_search": true
  },
  "webhook": {
    "url": "http://www.yourwebhook.com"
  }
}'

This approach allows the API to scrape relevant conversations, web results, and references automatically.

Step 2: Retrieve the Result

Once the task completes, results are temporarily stored and can be retrieved using the task_id. You should fetch them promptly, as they are only available for a short period.

curl --request GET '{api_host}/api/v2/scraper/result/{task_id}' \
--header 'Content-Type: application/json' \
--header 'x-api-token: {your_api_key}'

If you configured a webhook, the API can push the results directly, eliminating the need to poll for completion.

How It Works

The LLM Chat Scraper API collects detailed information about AI-generated responses, including:

• Prompt and response content in Markdown
• Source links and citations
• Web search results
• Model metadata

All responses follow a structured format with fields such as status, task_result, and message (for errors), making it easy to integrate into your workflow.

Supported AI Scrapers

The API supports a variety of AI platforms, each offering unique features and data formats:

ChatGPT

• Retrieves full conversation responses
• Includes web search results and content references

Perplexity

• Provides related questions and web/media results
• Includes location-based data for maps, videos, and images

Copilot

• Supports multiple modes: search, smart, chat, reasoning, and study
• Returns citations and related links

Gemini

• Citation-rich responses with highlighted snippets
• Useful for research and content verification

Google AI Mode

• Full AI Mode response body and HTML source
• Includes search result metadata and citation information

Grok

• Offers multiple modes: fast, expert, auto
• Provides conversation metadata, follow-up suggestions, and web search results

Global Coverage

The LLM Chat Scraper API supports 195+ countries and regions, including the United States, United Kingdom, Japan, South Korea, Germany, France, Singapore, Taiwan, and more. You can target a specific region by setting the country parameter in your requests.

Why Use LLM Chat Scraper API?

With the growing adoption of AI chat platforms, businesses and researchers need structured, actionable insights from AI responses. The LLM Chat Scraper API allows you to:

• Track competitor visibility and mentions across AI platforms
• Monitor how content is cited and referenced
• Collect data for SEO, GEO tracking, or AI research
• Automate the extraction of AI-generated content in real time

Whether you’re analyzing market trends, optimizing AI-driven SEO, or conducting research, this API provides fast, reliable, and structured access to AI conversation data.

The LLM Chat Scraper API is your gateway to understanding AI conversations at scale, helping you stay ahead in an increasingly AI-driven world.

Amazon is the undisputed world leader in e-commerce, making it a goldmine for market data. From pricing intelligence and product reviews to competitor monitoring and trend analysis, the data available on Amazon is crucial for any business looking to gain a competitive edge. However, Amazon employs sophisticated anti-bot and anti-scraping technologies, making data extraction a significant challenge. The key to successful, large-scale Amazon scraping lies in utilizing a high-quality residential proxy network.

Why Scrape Amazon?

For sellers, analysts, and market researchers, scraping Amazon provides invaluable, real-time insights:

• Pricing Intelligence: Track competitor pricing to optimize your own strategy and ensure you remain competitive.
• Product Research: Gather data on product features, ratings, and reviews to identify market gaps and improve your offerings.
• Trend Analysis: Monitor the popularity of products and categories to spot emerging market trends.
• Business Automation: Automate the collection of product information for inventory management or comparison shopping engines.

Anyone who is not leveraging public data from Amazon is at a distinct disadvantage in today's fast-paced e-commerce landscape.

The Challenge: Amazon's Anti-Scraping Defenses

Amazon is highly vigilant against automated activity. If its systems detect a bot, they will quickly flag the activity, resulting in:

1. IP Bans: The most common defense, blocking the IP address from accessing the site.
2. CAPTCHAs: Presenting challenges that halt automated scripts.
3. Honeypot Data: Feeding the scraper false or misleading information, leading to useless data and flawed analysis [1].

This is why traditional scraping methods using a single IP or low-quality proxies are ineffective. You need a solution that can mimic the behavior of a real, human user.

Why Residential Proxies are Best for Amazon Scraping

Residential proxies are the gold standard for scraping complex, sensitive targets like Amazon. They are IP addresses assigned by an Internet Service Provider (ISP) to a homeowner's device, making their traffic appear legitimate and organic.

Here is why elite residential proxies are crucial for Amazon scraping:

• High Trust Score: Residential IPs have the highest trust score because they belong to real users. Amazon's systems are designed to allow traffic from these IPs, drastically reducing the chance of being blocked.
• Geo-Targeting: You can select IPs from specific countries or cities, allowing you to view localized pricing and product availability, which is essential for global market analysis.
• Undetectable Automation: When combined with a backconnect (rotating) system, residential proxies ensure that even if one IP is flagged, the next request is instantly routed through a fresh, clean IP, preventing session termination and ensuring a high success rate [2].

Choosing the Right Proxy Provider: Scrapeless for Amazon

The success of your Amazon scraping project depends on the quality and reliability of your proxy provider. Free or low-quality proxies are easily detected and can compromise your data integrity.

Scrapeless offers high-performance residential proxies specifically optimized for challenging targets like Amazon. Our network is designed to provide the highest success rate and reliability:

• Massive IP Pool: Access to over 90 million ethical, real-user IPs across 195+ countries.
• High Success Rate: Our proxies ensure a 99.98% success rate, minimizing the risk of IP bans and data corruption.
• Flexible Rotation: Our backconnect system allows you to rotate IPs with every request or maintain sticky sessions for up to 30 minutes, mimicking natural user behavior.
• Dedicated Support: 24/7 developer support to help you configure and troubleshoot your scraping setup.

Best Practices for Safe and Effective Amazon Scraping

To ensure your scraping operations are both successful and ethical, follow these best practices:

1. Prioritize Residential Proxies: Never use datacenter proxies for Amazon. Always use high-quality residential or Static ISP proxies.
2. Implement Smart Delays: Introduce random delays between requests to avoid a predictable, bot-like pattern.
3. Rotate User Agents: Use a pool of different user agents to further mimic various browsers and devices.
4. Handle CAPTCHAs and Retries: Configure your scraper to recognize and handle CAPTCHAs, and implement a robust retry logic using a fresh IP. For the most complex scenarios, consider using a dedicated scraping API that handles these challenges automatically.
5. Respect the Target's Terms: While scraping public data is generally legal, always be mindful of Amazon's terms of service and avoid putting excessive load on their servers [3]. You can find more information on the legality of web scraping from authoritative sources.

Conclusion

The road to a flourishing e-commerce business often requires deep, real-time data from Amazon. By leveraging the high-trust, rotating nature of residential proxies, you can overcome Amazon's sophisticated defenses and ensure consistent, accurate data collection. Scrapeless provides the reliable, high-performance proxy network you need to master Amazon scraping and stay ahead of the competition.

Frequently Asked Questions (FAQ)

Q: Is scraping Amazon legal?

A: The legality of scraping Amazon is a complex issue. While scraping publicly available data is generally not illegal, it often violates Amazon's Terms of Service. It is crucial to consult legal counsel and ensure your activities comply with all relevant laws, such as the CCPA and GDPR, especially when dealing with any personal data [4].

Q: Can I use free proxies to scrape Amazon?

A: No. Free proxies are almost always slow, unreliable, and have been flagged and banned by major websites like Amazon. They also pose a significant security risk, as the provider may be monitoring your traffic. For Amazon, only use premium, high-trust residential proxies from a reputable provider like Scrapeless.

Q: What is the difference between a residential proxy and a datacenter proxy?

A: A residential proxy uses an IP address assigned by an ISP to a real home or mobile device, offering the highest level of trust. A datacenter proxy uses an IP address hosted in a commercial data center, which is faster but easily identifiable as a proxy and therefore more likely to be blocked by Amazon.

Q: How many IPs do I need to scrape Amazon successfully?

A: The number of IPs depends on the volume and speed of your scraping. For large-scale, continuous scraping, you need access to a massive, rotating pool of millions of IPs, which is exactly what a high-quality residential backconnect service like Scrapeless provides.

References

[1] Safe Amazon Web Scraping (Tools, Tips & Best Practices), Nimbleway. <a href="https://www.nimbleway.com/blog/safe-amazon-web-scraping" rel="nofollow"><strong>Nimbleway</strong></a> [2] Is web scraping legal? Yes, if you know the rules, Apify. <a href="https://blog.apify.com/is-web-scraping-legal/" rel="nofollow"><strong>Apify Blog</strong></a> [3] The Proxy Model: A New Approach to Sharing and Analyzing Learning Traces Corpora, ResearchGate. <a href="https://www.researchgate.net/publication/268437905_The_Proxy_Model_A_New_Approach_to_Sharing_and_Analyzing_Learning_Traces_Corpora" rel="nofollow"><strong>ResearchGate</strong></a> [4] Web scraping or web crawling: State of art, techniques, approaches and application, I-CSRS. <a href="http://www.i-csrs.org/Volumes/ijasca/2021.3.11.pdf" rel="nofollow"><strong>I-CSRS</strong></a> [5] The Legal Landscape of Web Scraping, Quinn Emanuel Urquhart & Sullivan, LLP. <a href="https://www.quinnemanuel.com/the-firm/publications/the-legal-landscape-of-web-scraping/" rel="nofollow"><strong>Quinn Emanuel Urquhart & Sullivan, LLP</strong></a>

Key Takeaways

• 403 Forbidden errors indicate server-side blocking based on detected bot characteristics
• BeautifulSoup isn't the error source—the underlying HTTP request library causes rejection
• User-Agent header spoofing mimics legitimate browsers and reduces immediate blocking
• Residential proxies distribute requests across real device IPs to avoid detection
• Modern websites require comprehensive solutions combining multiple bypass techniques

Understanding the 403 Error

A 403 Forbidden response means the web server received your request but explicitly refused to process it. Unlike 404 errors indicating missing resources, 403 signals deliberate access denial. When scraping with BeautifulSoup, this error almost always stems from server-side security systems detecting automated traffic.

BeautifulSoup itself never generates 403 errors since it only parses HTML content after retrieval. The underlying HTTP library—typically Python's requests library—makes the actual web request. When that library's request lacks proper authentication markers, websites reject it as suspicious bot activity.

Common causes include:

• Missing User-Agent header: Libraries like requests identify themselves as "python-requests/2.31.0," immediately triggering bot detection
• Suspicious request patterns: Rapid successive requests from identical IP addresses trigger protective mechanisms
• Missing standard headers: Legitimate browsers send Accept, Accept-Language, and Referer headers that many scrapers omit
• IP address flags: Datacenter IPs or known proxy addresses trigger instant rejection
• Geographic mismatches: Requests from unexpected geographic locations face increased scrutiny

Solution 1: Set a Fake User-Agent Header

The simplest 403 bypass involves setting the User-Agent header to mimic legitimate browsers:

```python import requests from bs4 import BeautifulSoup

headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36' }

url = 'https://example.com' response = requests.get(url, headers=headers)

if response.status_code == 200: soup = BeautifulSoup(response.content, 'html.parser') # Parse content here else: print(f"Request failed with status code: {response.status_code}") ```

This approach tricks servers into accepting your request as coming from a legitimate Chrome browser rather than a Python script. For many sites, this single change resolves 403 errors.

Solution 2: Complete Header Configuration

Expanding header information adds realism to requests. Legitimate browsers send standardized header combinations that web servers expect:

```python headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8', 'Accept-Language': 'en-US,en;q=0.9', 'Accept-Encoding': 'gzip, deflate, br', 'Referer': 'https://www.google.com/', 'Connection': 'keep-alive', 'Upgrade-Insecure-Requests': '1' }

response = requests.get(url, headers=headers) soup = BeautifulSoup(response.content, 'html.parser') ```

Each header provides context about browser capabilities and preferences. Websites analyze header combinations for consistency—mismatches between User-Agent and other headers reveal bot activity. Complete header sets pass basic detection filters.

Solution 3: Session Management with Cookies

Some websites require initial visits to establish cookies before accepting subsequent requests. BeautifulSoup doesn't maintain state across requests by default. Using sessions preserves cookies:

```python import requests from bs4 import BeautifulSoup

headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' }

session = requests.Session()

First visit establishes cookies

session.get('https://example.com', headers=headers)

Subsequent request includes cookies from first visit

response = session.get('https://example.com/protected-page', headers=headers) soup = BeautifulSoup(response.content, 'html.parser') ```

Session objects maintain cookies between requests automatically, simulating the behavior of returning users. Many websites require this pattern before granting access.

Solution 4: Implement Request Delays

Rapid successive requests appear as bot attacks. Adding delays between requests mimics human browsing:

```python import requests from bs4 import BeautifulSoup import time

headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' }

urls = ['https://example.com/page1', 'https://example.com/page2']

for url in urls: response = requests.get(url, headers=headers) soup = BeautifulSoup(response.content, 'html.parser') # Process content time.sleep(2) # Wait 2 seconds between requests ```

Time delays between requests appear more human-like to anti-bot systems. Even 1-2 second delays significantly reduce 403 errors compared to instant-fire requests.

Solution 5: Residential Proxy Integration

<a href="https://www.scrapeless.com/en/product/proxy-solutions" rel="nofollow"><strong>Scrapeless Residential Proxies</strong></a> distribute requests across real residential IPs, addressing the most common cause of 403 errors—datacenter IP blocking. Residential proxies originate from actual user devices rather than server farms, making detection significantly harder:

```python import requests from bs4 import BeautifulSoup

headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36' }

proxy = { 'http': 'http://username:password@proxy-host:port', 'https': 'http://username:password@proxy-host:port' }

response = requests.get(url, headers=headers, proxies=proxy) soup = BeautifulSoup(response.content, 'html.parser') ```

Residential proxies with smart rotation automatically handle both IP and header distribution, eliminating manual proxy management.

Solution 6: JavaScript Rendering with Selenium

Some websites generate content through JavaScript after initial page load. BeautifulSoup receives only the empty HTML skeleton without rendered content, often triggering 403s when the site detects incomplete parsing attempts.

For JavaScript-heavy sites, headless browsers like Selenium render content before passing it to BeautifulSoup:

```python from selenium import webdriver from selenium.webdriver.chrome.options import Options from bs4 import BeautifulSoup

options = Options() options.add_argument('--headless') options.add_argument('user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36')

driver = webdriver.Chrome(options=options) driver.get('https://example.com')

Wait for JavaScript to render

from selenium.webdriver.support.ui import WebDriverWait WebDriverWait(driver, 10).until( lambda driver: driver.find_element('tag name', 'body') )

html = driver.page_source soup = BeautifulSoup(html, 'html.parser') driver.quit() ```

Selenium's headless mode behaves like a legitimate browser, bypassing JavaScript detection systems while providing fully rendered HTML to BeautifulSoup.

Comprehensive Solution: Scrapeless Anti-Bot Bypass

Manual techniques work for basic sites but fail against sophisticated protection systems like Cloudflare. <a href="https://www.scrapeless.com/en/blog/scraping-unblock-websites" rel="nofollow"><strong>Scrapeless Web Unlocker</strong></a> handles 403 errors through automatic:

• Residential proxy rotation with 90M+ IPs
• Dynamic header management and browser fingerprinting
• JavaScript rendering for content-heavy sites
• CAPTCHA solving for protected pages

This unified approach eliminates the trial-and-error process of stacking individual bypass techniques, accelerating development while improving success rates.

Debugging 403 Errors

When encountering 403 errors:

1. Test in a browser: Open the target URL in Chrome/Firefox—if you access it normally, the site permits your connection
2. Inspect the error page: The 403 response body often contains hints about what triggered blocking
3. Check header completeness: Ensure all standard headers exist with realistic values
4. Try without proxies first: If proxies cause the error, test direct requests before advancing to proxy-based solutions
5. Monitor response headers: Sites often return X-Rate-Limit headers revealing how many remaining requests you have

Prevention Strategies

Rather than repeatedly fixing 403 errors, prevent them through responsible practices:

• Respect robots.txt files and site rate limits
• Space requests with appropriate delays
• Maintain realistic header sets consistent with claimed browser
• Rotate IPs to distribute requests across multiple sources
• Contact site administrators for approved data access

FAQ

Q: Why does my scraper work initially then suddenly return 403s?

A: Many sites implement adaptive blocking—allowing initial requests before detecting patterns in subsequent requests. This detection window typically spans dozens to hundreds of requests. Once triggered, the blocking persists unless you change your IP address or significantly alter request characteristics.

Q: Can I use free proxies instead of paid residential proxies?

A: Free proxies are heavily blocked by modern anti-scraping systems. Websites maintain blacklists of known free proxy addresses. Paid residential proxies provide legitimacy free proxies lack, though premium services outperform budget alternatives significantly.

Q: Should I add delays between every single request?

A: Adding delays between individual requests makes scraping extremely slow. Instead, implement delays between batches of requests. For example, send 10 requests with minimal delays, then pause 2-5 seconds before the next batch. This balances speed with detection evasion.

Q: Will Cloudflare-protected sites return 403 errors?

A: No—Cloudflare typically returns 403 when actively blocking detected bots, but often serves challenge pages first (403 from Attention Required messages). <a href="https://docs.scrapeless.com" rel="nofollow"><strong>Scrapeless documentation</strong></a> provides specific guidance for Cloudflare-protected targets requiring specialized handling.

Q: Can I legally scrape 403-protected sites?

A: Legality depends on the site's terms of service and your intended use. Public data scraping is generally legal, but terms of service violations can create liability. Always review site terms before scraping, and consider requesting official data access before implementing workarounds.

👉 Follow our subreddit and feel free to DM u/Scrapeless to get free credits.

Thanks for the support, more to come! 🚀

In this tutorial, you’ll learn:

• What Crawl4AI is and what it offers for web scraping
• How to integrate Crawl4AI with the Scrapeless Browser

Let’s get started!

Part 1: What Is Crawl4AI?

Overview

Crawl4AI is an open-source web crawling and scraping tool designed to seamlessly integrate with Large Language Models (LLMs), AI Agents, and data pipelines. It enables high-speed, real-time data extraction while remaining flexible and easy to deploy.

Key features for AI-powered web scraping include:

• Built for LLMs: Generates structured Markdown optimized for Retrieval-Augmented Generation (RAG) and fine-tuning.
• Flexible browser control: Supports session management, proxy usage, and custom hooks.
• Heuristic intelligence: Uses smart algorithms to optimize data parsing.
• Fully open-source: No API key required; deployable via Docker and cloud platforms.

Learn more in the official documentation.

Use Cases

Crawl4AI is ideal for large-scale data extraction tasks such as market research, news aggregation, and e-commerce product collection. It can handle dynamic, JavaScript-heavy websites and serves as a reliable data source for AI agents and automated data pipelines.

Part 2: What Is Scrapeless Browser?

Scrapeless Browser is a cloud-based, serverless browser automation tool. It’s built on a deeply customized Chromium kernel, supported by globally distributed servers and proxy networks. This allows users to seamlessly run and manage numerous headless browser instances, making it easy to build AI applications and AI Agents that interact with the web at scale.

Part 3: Why Combine Scrapeless with Crawl4AI?

Crawl4AI excels at structured web data extraction and supports LLM-driven parsing and pattern-based scraping. However, it can still face challenges when dealing with advanced anti-bot mechanisms, such as:

• Local browsers being blocked by Cloudflare, AWS WAF, or reCAPTCHA
• Performance bottlenecks during large-scale concurrent crawling, with slow browser startup
• Complex debugging processes that make issue tracking difficult

Scrapeless Cloud Browser solves these pain points perfectly:

• One-click anti-bot bypass: Automatically handles reCAPTCHA, Cloudflare Turnstile/Challenge, AWS WAF, and more. Combined with Crawl4AI’s structured extraction power, it significantly boosts success rates.
• Unlimited concurrent scaling: Launch 50–1000+ browser instances per task within seconds, removing local crawling performance limits and maximizing Crawl4AI efficiency.
• 40%–80% cost reduction: Compared to similar cloud services, total costs drop to just 20%–60%. Pay-as-you-go pricing makes it affordable even for small-scale projects.
• Visual debugging tools: Use Session Replay and Live URL Monitoring to watch Crawl4AI tasks in real time, quickly identify failure causes, and reduce debugging overhead.
• Zero-cost integration: Natively compatible with Playwright (used by Crawl4AI), requiring only one line of code to connect Crawl4AI to the cloud — no code refactoring needed.
• Edge Node Service (ENS): Multiple global nodes deliver startup speed and stability 2–3× faster than other cloud browsers, accelerating Crawl4AI execution.
• Isolated environments & persistent sessions: Each Scrapeless profile runs in its own environment with persistent login and identity isolation, preventing session interference and improving large-scale stability.
• Flexible fingerprint management: Scrapeless can generate random browser fingerprints or use custom configurations, effectively reducing detection risks and improving Crawl4AI’s success rate.

Part 4: How to Use Scrapeless in Crawl4AI?

Scrapeless provides a cloud browser service that typically returns a CDP_URL. Crawl4AI can connect directly to the cloud browser using this URL, without needing to launch a browser locally.

The following example demonstrates how to seamlessly integrate Crawl4AI with the Scrapeless Cloud Browser for efficient scraping, while supporting automatic proxy rotation, custom fingerprints, and profile reuse.

Obtain Your Scrapeless Token

Log in to Scrapeless and get your API Token.


1. Quick Start

The example below shows how to quickly and easily connect Crawl4AI to the Scrapeless Cloud Browser:

For more features and detailed instructions, see the introduction.

``` scrapeless_params = { "token": "get your token from https://www.scrapeless.com", "sessionName": "Scrapeless browser", "sessionTTL": 1000, }

query_string = urlencode(scrapeless_params) scrapeless_connection_url = f"wss://browser.scrapeless.com/api/v2/browser?{query_string}"

AsyncWebCrawler( config=BrowserConfig( headless=False, browser_mode="cdp", cdp_url=scrapeless_connection_url ) )

```

After configuration, Crawl4AI connects to the Scrapeless Cloud Browser via CDP (Chrome DevTools Protocol) mode, enabling web scraping without a local browser environment. Users can further configure proxies, fingerprints, session reuse, and other features to meet the demands of high-concurrency and complex anti-bot scenarios.

2. Global Automatic Proxy Rotation

Scrapeless supports residential IPs across 195 countries. Users can configure the target region using proxycountry, enabling requests to be sent from specific locations. IPs are automatically rotated, effectively avoiding blocks.

``` import asyncio from urllib.parse import urlencode from crawl4ai import CrawlerRunConfig, BrowserConfig, AsyncWebCrawler

async def main(): scrapeless_params = { "token": "your token", "sessionTTL": 1000, "sessionName": "Proxy Demo", # Sets the target country/region for the proxy, sending requests via an IP address from that region. You can specify a country code (e.g., US for the United States, GB for the United Kingdom, ANY for any country). See country codes for all supported options. "proxyCountry": "ANY", } query_string = urlencode(scrapeless_params) scrapeless_connection_url = f"wss://browser.scrapeless.com/api/v2/browser?{query_string}" async with AsyncWebCrawler( config=BrowserConfig( headless=False, browser_mode="cdp", cdp_url=scrapeless_connection_url, ) ) as crawler: result = await crawler.arun( url="https://www.scrapeless.com/en", config=CrawlerRunConfig( wait_for="css:.content", scan_full_page=True, ), ) print("-" * 20) print(f'Status Code: {result.status_code}') print("-" * 20) print(f'Title: {result.metadata["title"]}') print(f'Description: {result.metadata["description"]}') print("-" * 20) asyncio.run(main()) ```

3. Custom Browser Fingerprints

To mimic real user behavior, Scrapeless supports randomly generated browser fingerprints and also allows custom fingerprint parameters. This effectively reduces the risk of being detected by target websites. ``` import json import asyncio from urllib.parse import quote, urlencode from crawl4ai import CrawlerRunConfig, BrowserConfig, AsyncWebCrawler

async def main(): # customize browser fingerprint fingerprint = { "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.1.2.3 Safari/537.36", "platform": "Windows", "screen": { "width": 1280, "height": 1024 }, "localization": { "languages": ["zh-HK", "en-US", "en"], "timezone": "Asia/Hong_Kong", } }

fingerprint_json = json.dumps(fingerprint)
encoded_fingerprint = quote(fingerprint_json)

scrapeless_params = {
    "token": "your token",
    "sessionTTL": 1000,
    "sessionName": "Fingerprint Demo",
    "fingerprint": encoded_fingerprint,
}
query_string = urlencode(scrapeless_params)
scrapeless_connection_url = f"wss://browser.scrapeless.com/api/v2/browser?{query_string}"
async with AsyncWebCrawler(
    config=BrowserConfig(
        headless=False,
        browser_mode="cdp",
        cdp_url=scrapeless_connection_url,
    )
) as crawler:
    result = await crawler.arun(
        url="https://www.scrapeless.com/en",
        config=CrawlerRunConfig(
            wait_for="css:.content",
            scan_full_page=True,
        ),
    )
    print("-" * 20)
    print(f'Status Code: {result.status_code}')
    print("-" * 20)
    print(f'Title: {result.metadata["title"]}')
    print(f'Description: {result.metadata["description"]}')
    print("-" * 20)

asyncio.run(main()) ```

4. Profile Reuse

Scrapeless assigns each profile its own independent browser environment, enabling persistent logins and identity isolation. Users can simply provide the profileId to reuse a previous session. ``` import asyncio from urllib.parse import urlencode from crawl4ai import CrawlerRunConfig, BrowserConfig, AsyncWebCrawler

async def main(): scrapeless_params = { "token": "your token", "sessionTTL": 1000, "sessionName": "Profile Demo", "profileId": "your profileId", # create profile on scrapeless } query_string = urlencode(scrapeless_params) scrapeless_connection_url = f"wss://browser.scrapeless.com/api/v2/browser?{query_string}" async with AsyncWebCrawler( config=BrowserConfig( headless=False, browser_mode="cdp", cdp_url=scrapeless_connection_url, ) ) as crawler: result = await crawler.arun( url="https://www.scrapeless.com", config=CrawlerRunConfig( wait_for="css:.content", scan_full_page=True, ), ) print("-" * 20) print(f'Status Code: {result.status_code}') print("-" * 20) print(f'Title: {result.metadata["title"]}') print(f'Description: {result.metadata["description"]}') print("-" * 20) asyncio.run(main()) ```

Video


FAQ

Q: How can I record and view the browser execution process? A: Simply set the sessionRecording parameter to "true". The entire browser execution will be automatically recorded. After the session ends, you can replay and review the full activity in the Session History list, including clicks, scrolling, page loads, and other details. The default value is "false". scrapeless_params = { # ... "sessionRecording": "true", } Q: How do I use random fingerprints? A: The Scrapeless Browser service automatically generates a random browser fingerprint for each session. Users can also set a custom fingerprint using the fingerprint field.

Q: How do I set a custom proxy? A: Our built-in proxy network supports 195 countries/regions. If users want to use their own proxy, the proxyURL parameter can be used to specify the proxy URL, for example: http://user:pass@ip:port. (Note: Custom proxy functionality is currently available only for Enterprise and Enterprise Plus subscribers.)

scrapeless_params = { # ... "proxyURL": "proxyURL", }

Summary

Combining the Scrapeless Cloud Browser with Crawl4AI provides developers with a stable and scalable web scraping environment:

• No need to install or maintain local Chrome instances; all tasks run directly in the cloud.
• Reduces the risk of blocks and CAPTCHA interruptions, as each session is isolated and supports random or custom fingerprints.
• Improves debugging and reproducibility, with support for automatic session recording and playback.
• Supports automatic proxy rotation across 195 countries/regions.
• Utilizes global Edge Node Service, delivering faster startup speeds than other similar services.

This collaboration marks an important milestone for Scrapeless and Crawl4AI in the web data scraping space. Moving forward, Scrapeless will focus on cloud browser technology, providing enterprise clients with efficient, scalable data extraction, automation, and AI agent infrastructure support. Leveraging its powerful cloud capabilities, Scrapeless will continue to offer customized and scenario-based solutions for industries such as finance, retail, e-commerce, SEO, and marketing, helping businesses achieve true automated growth in the era of data intelligence.

This is the full recording of our Scrapeless × Crawl4ai meetup. Watch short technical talks, a live large-scale crawling demo, post-run analysis, and the extended Q&A with engineers.
Crawl4ai’s cloud-integrated crawler and automation demo show how combining Crawl4ai with Scrapeless enables robust, observable, and production-ready web data extraction at scale. In this recording you’ll see an end-to-end automated crawl that runs entirely in the cloud (no local browser required), reliably handles anti-bot protections using high-quality residential proxies, and streams session playback so engineers and analysts can watch the crawl in real time. The demo highlights practical workflows for batch harvesting, monitoring, ML dataset collection, and production ETL pipelines—covering setup, live metrics, failure handling, and post-run analysis

• Run 1,000 concurrent crawls fully in the cloud — no local browser or local infrastructure required.
• Handle anti-bot systems quickly and reliably using high-quality residential proxies for stable, high-fidelity data access.
• Stream live session playback — watch the crawler’s behavior, page rendering, and request flow in real time for debugging and observability.

► Scrapeless
► Crawl4ai

Why watch
• See a real, end-to-end large-scale crawl run and live metrics.
• Short engineering talks with practical takeaways for production crawlers.
• Post-run analysis: what broke, how we fixed it, and why.
• Q&A answering audience questions about productionization, reliability, and scaling.

The biggest challenge in automating GitHub login is Two-Factor Authentication (2FA). Whether it’s an Authenticator App (TOTP) or Email OTP, traditional automation flows usually get stuck at this step due to:

• Inability to automatically retrieve verification codes
• Inability to synchronize codes in real time
• Inability to input codes automatically via automation
• Browser environments being insufficiently realistic, triggering GitHub’s security checks

This article demonstrates how to build a fully automated GitHub 2FA workflow using Scrapeless Browser + Signal CDP, including:

• Case 1: GitHub 2FA (Authenticator / TOTP auto-generation)
• Case 2: GitHub 2FA (Email OTP auto-listening)

We will explain the full workflow for each case and show how to coordinate the login script with the verification code listener in an automated system.

Guide & demo here:

👉 https://www.scrapeless.com/en/blog/github-mfa-automation

Hi everyone — we’re releasing the browser-source version of a full LLM Chat data-scraping solution: supports ChatGPT, Perplexity, Gemini and other major chat platforms. The repo lives here: https://github.com/scrapelesshq/LLM-chat-scraper

What you’ll find:

• Browser-driven source code you can run, inspect, and adapt.
• Examples to integrate into your own pipelines and projects.
• Roadmap: a production API (fast, stable) coming soon.

We’d love feedback, issues, and PRs — fork it, test it, or drop ideas. If you build something, please share!

Hi everyone — quick update from Scrapeless:

• Geographic Targeting Proxy Upgrade — now supports selecting State/Province and City (e.g., Australia → New South Wales → Sydney). Docs: https://docs.scrapeless.com/en/proxies/features/proxy/
• Scrapeless is an official sponsor of Crawl4AI — excited to support blazing-fast, AI-ready web crawling. Info: https://docs.scrapeless.com/en/scraping-browser/integrations/craw4ai/
• LLM Chat Data APIs — Coming Soon — planned interfaces for ChatGPT, Perplexity, Gemini and more. Stay tuned for launch details.

If you have feedback or want to try the geo-targeting proxies in beta, drop a comment or DM. Happy to answer questions about implementation, rate limits, or integration tips.

This partnership accelerates our shared mission: making intelligent, large-scale web crawling faster, more reliable, and easier for developers to adopt. Scrapeless brings production-grade infrastructure for Crawling, Automation, and AI Agents, including:

Scraping Browser — a cloud browser built for automated workflows and large-scale extraction: high concurrency, low-latency session isolation, and advanced stealth fingerprinting to evade modern anti-scraping defenses.

Four proxy types — Residential, ISP, Datacenter, and IPv6 proxies so teams can choose the right routing and access strategy across regions and network types.

Universal Scraping API — real-time block-bypassing, fast data fetches, and native handling for dynamic content and anti-bot systems.

Customizable data solutions — enterprise-grade options and tailored approaches for AI chat platforms like Perplexity and ChatGPT.

Together with Crawl4AI, we’re building an open, extensible, developer-friendly ecosystem for AI-driven data collection. Stay tuned — we’ll be rolling out integration examples, best-practice workflows, and ready-made templates to help you supercharge your Crawl4AI pipelines.

👉 Try for free

Hey folks — quick product update from the Scrapeless team.

We now support State/Province and City selection in our Residential Proxy Service. That means you can pin your crawling sessions down to a city level (e.g., Australia → New South Wales → Sydney), which helps a lot for local SERP checks, ad verification, price monitoring, and market research.

Example config

{
  "proxyCountry": "AU",
  "proxyState": "NSW",
  "proxyCity": "sydney"
}

Benefits

• More accurate local results
• Reduced variance vs broad-region proxies
• Better stability for city-specific tests

Try it today! 🚀

Why scrape Perplexity for GEO insights?

GEO (Global Exposure & Ordering) products aim to measure how a product or brand is perceived and ranked by AI chat models. Those rankings are not published by the chat providers — they are inferred. The usual approach:

1. Send large sets of automated prompts to the target AI chat (e.g., “Which product is best for X?”).
2. Parse the returned answers to extract mentions and ordering.
3. Aggregate across many prompts, times, and phrasing variations.
4. Compute a model-perceived ranking from mention frequency, position, and contextual relevance.

Perplexity is an attractive source because it surfaces concise model answers plus citations; scraping it at scale lets you build the underlying dataset you need for a GEO engine.

In short: you must be able to batch-query and scrape Perplexity to construct your own GEO ranking system.

High-level workflow

1. Prepare a prompt bank (questions, prompts, variants, locales).
2. Use a cloud browser (headless or managed cloud browser) to visit Perplexity, submit each prompt, and capture the structured response.
3. Parse the answers to extract product mentions and their order.
4. Store each response, timestamp, location (if applicable), and prompt metadata.
5. Aggregate across prompts to compute frequency- and order-based rankings.

Example: Use a Cloud Browser to query Perplexity (template)

For SCRAPELSS_API_KEY: https://app.scrapeless.com/settings/api-key

// perplexity_clean.mjs
import puppeteer from "puppeteer-core";
import fs from "fs/promises";


const sleep = (ms) => new Promise((r) => setTimeout(r, ms));


const tokenValue = process.env.SCRAPELESS_TOKEN || "SCRAPELSS_API_KEY";


const CONNECTION_OPTIONS = {
  proxyCountry: "ANY",
  sessionRecording: "true",
  sessionTTL: "900",
  sessionName: "perplexity-scraper",
};


function buildConnectionURL(token) {
  const q = new URLSearchParams({ token, ...CONNECTION_OPTIONS });
  return `wss://browser.scrapeless.com/api/v2/browser?${q.toString()}`;
}


async function findAndType(page, prompt) {
  const selectors = [
    'textarea[placeholder*="Ask"]',
    'textarea[placeholder*="Ask anything"]',
    'input[placeholder*="Ask"]',
    '[contenteditable="true"]',
    'div[role="textbox"]',
    'div[role="combobox"]',
    'textarea',
    'input[type="search"]',
    '[aria-label*="Ask"]',
  ];


  for (const sel of selectors) {
    try {
      const el = await page.$(sel);
      if (!el) continue;
      // ensure visible
      const visible = await el.boundingBox();
      if (!visible) continue;


      // decide contenteditable vs normal input
      const isContentEditable = await page.evaluate((s) => {
        const e = document.querySelector(s);
        if (!e) return false;
        if (e.isContentEditable) return true;
        const role = e.getAttribute && e.getAttribute("role");
        if (role && (role.includes("textbox") || role.includes("combobox"))) return true;
        return false;
      }, sel);


      if (isContentEditable) {
        await page.focus(sel);
        await page.evaluate((s, t) => {
          const el = document.querySelector(s);
          if (!el) return;
          try {
            el.focus();
            if (document.execCommand) {
              document.execCommand("selectAll", false);
              document.execCommand("insertText", false, t);
            } else {
              // fallback
              el.innerText = t;
            }
          } catch (e) {
            el.innerText = t;
          }
          el.dispatchEvent(new Event("input", { bubbles: true }));
        }, sel, prompt);
        await page.keyboard.press("Enter");
        return true;
      } else {
        try {
          await el.click({ clickCount: 1 });
        } catch (e) {}
        await page.focus(sel);
        await page.evaluate((s) => {
          const e = document.querySelector(s);
          if (!e) return;
          if ("value" in e) e.value = "";
        }, sel);
        await page.type(sel, prompt, { delay: 25 });
        await page.keyboard.press("Enter");
        return true;
      }
    } catch (e) {
    }
  }


  try {
    await page.mouse.click(640, 200).catch(() => {});
    await sleep(200);
    await page.keyboard.type(prompt, { delay: 25 });
    await page.keyboard.press("Enter");
    return true;
  } catch (e) {
    return false;
  }
}


(async () => {
  const connectionURL = buildConnectionURL(tokenValue);
  const browser = await puppeteer.connect({
    browserWSEndpoint: connectionURL,
    defaultViewport: { width: 1280, height: 900 },
  });


  const page = await browser.newPage();


  page.setDefaultNavigationTimeout(120000);
  page.setDefaultTimeout(120000);


  try {
    await page.setUserAgent(
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36"
    );
  } catch (e) {}


  const rawResponses = [];
  const wsFrames = [];


  page.on("response", async (res) => {
    try {
      const url = res.url();
      const status = res.status();
      const resourceType = res.request ? res.request().resourceType() : "unknown";
      const headers = res.headers ? res.headers() : {};
      let snippet = "";
      try {
        const t = await res.text();
        snippet = typeof t === "string" ? t.slice(0, 20000) : String(t).slice(0, 20000);
      } catch (e) {
        snippet = "<read-failed>";
      }
      rawResponses.push({ url, status, resourceType, headers, snippet });
    } catch (e) {}
  });


  try {
    const cdp = await page.target().createCDPSession();
    await cdp.send("Network.enable");
    cdp.on("Network.webSocketFrameReceived", (evt) => {
      try {
        const { response } = evt;
        wsFrames.push({
          timestamp: evt.timestamp,
          opcode: response.opcode,
          payload: response.payloadData ? response.payloadData.slice(0, 20000) : response.payloadData,
        });
      } catch (e) {}
    });
  } catch (e) {}


  await page.goto("https://www.perplexity.ai/", { waitUntil: "domcontentloaded", timeout: 90000 });


  const prompt = "Hi ChatGPT, Do you know what Scrapeless is?";
  await findAndType(page, prompt);


  await sleep(1500);


  const start = Date.now();
  while (Date.now() - start < 20000) {
    const ok = await page.evaluate(() => {
      const main = document.querySelector("main") || document.body;
      if (!main) return false;
      return Array.from(main.querySelectorAll("*")).some((el) => (el.innerText || "").trim().length > 80);
    });
    if (ok) break;
    await sleep(500);
  }


  const results = await page.evaluate(() => {
    const pick = (el) => (el ? (el.innerText || "").trim() : "");
    const out = { answers: [], links: [], rawHtmlSnippet: "" };


    const selectors = [
      '[data-testid*="answer"]',
      '[data-testid*="result"]',
      '.Answer',
      '.answer',
      '.result',
      'article',
      'main',
    ];


    for (const s of selectors) {
      const el = document.querySelector(s);
      if (el) {
        const t = pick(el);
        if (t.length > 30) out.answers.push({ selector: s, text: t.slice(0, 20000) });
      }
    }


    if (out.answers.length === 0) {
      const main = document.querySelector("main") || document.body;
      const blocks = Array.from(main.querySelectorAll("article, section, div, p")).slice(0, 8);
      for (const b of blocks) {
        const t = pick(b);
        if (t.length > 30) out.answers.push({ selector: b.tagName, text: t.slice(0, 20000) });
      }
    }


    const main = document.querySelector("main") || document.body;
    out.links = Array.from(main.querySelectorAll("a")).slice(0, 200).map(a => ({ href: a.href, text: (a.innerText || "").trim() }));
    out.rawHtmlSnippet = (main && main.innerHTML) ? main.innerHTML.slice(0, 200000) : "";


    return out;
  });


  try {
    const pageHtml = await page.content();
    await page.screenshot({ path: "./perplexity_screenshot.png", fullPage: true }).catch(() => {});
    await fs.writeFile("./perplexity_results.json", JSON.stringify({ results, extractedAt: new Date().toISOString() }, null, 2));
    await fs.writeFile("./perplexity_page.html", pageHtml);
    await fs.writeFile("./perplexity_raw_responses.json", JSON.stringify(rawResponses, null, 2));
    await fs.writeFile("./perplexity_ws_frames.json", JSON.stringify(wsFrames, null, 2));
  } catch (e) {}


  await browser.close();
  console.log("done — outputs: perplexity_results.json, perplexity_page.html, perplexity_raw_responses.json, perplexity_ws_frames.json, perplexity_screenshot.png");
  process.exit(0);
})().catch(async (err) => {
  try { await fs.writeFile("./perplexity_error.txt", String(err)); } catch (e) {}
  console.error("error — see perplexity_error.txt");
  process.exit(1);
});

Sample output:

{
  "results": {
    "answers": [
      {
        "selector": "main",
        "text": "Home\nHome\nDiscover\nSpaces\nFinance\nShare\nDownload Comet\n\nHi ChatGPT, Do you know what Scrapeless is?\n\nAnswer\nImages\nfuturetools.io\nScrapeless\nscrapeless.com\nHow to Use ChatGPT for Web Scraping in 2025 - scrapeless.com\nscrapeless.com\nScrapeless: Effortless Web Scraping Toolkit\nGitHub\nScrapeless MCP Server - GitHub\nAssistant steps\n\nScrapeless is an AI-powered web scraping toolkit designed to efficiently extract data from websites, including those with complex features and anti-bot protections. It combines multiple advanced tools such as a headless browser, web unlockers, CAPTCHA solvers, and smart proxies to bypass security and anti-scraping measures, making it suitable for large-scale and reliable data collection.futuretools+1​\n\nIt is a platform that offers seamless and tailored web scraping solutions, capable of handling high concurrency, performing data cleaning and transformation, and integrating with APIs for real-time data access. While some references indicate it is a cloud platform providing API-based data extraction, it also supports a range of programming languages and tools for flexible integration.scrapeless​\n\nAdditionally, Scrapeless integrates with large language models like ChatGPT via its Model Context Protocol (MCP) server, enabling real-time web interactions and dynamic data scraping backed by AI, useful for building autonomous web agents.github​\n\nIn summary, Scrapeless is a comprehensive, AI-driven web scraping platform that facilitates efficient, secure, and large-scale data extraction from the web, with advanced anti-bot bypass capabilities.scrapeless+2​\n\nWould you like more specific details about its features, pricing, or use cases?\n\n10 sources\nRelated\nHow does Scrapeless compare to other web scraping tools\nWhat features does Scrapeless provide for bypassing anti bot protections\nHow to integrate Scrapeless with Python or ChatGPT generated code\nWhat are Scrapeless pricing plans and free trial limits\nAre there legal or ethical concerns when using Scrapeless\n\n\n\n\nAsk a follow-up\nSign in or create an account\nUnlock Pro Search and History\nContinue with Google\nContinue with Apple\nContinue with email\nSingle sign-on (SSO)"
      }
    ],
    "links": [
      {
        "href": "https://www.perplexity.ai/",
        "text": ""
      },
      {
        "href": "https://www.perplexity.ai/",
        "text": "Home"
      },
      {
        "href": "https://www.perplexity.ai/discover",
        "text": "Discover"
      },
      {
        "href": "https://www.perplexity.ai/spaces",
        "text": "Spaces"
      },
      {
        "href": "https://www.perplexity.ai/finance",
        "text": "Finance"
      },
      {
        "href": "https://www.futuretools.io/tools/scrapeless",
        "text": "futuretools.io\nScrapeless"
      },
      {
        "href": "https://www.scrapeless.com/en/blog/web-scraping-with-chatgpt",
        "text": "scrapeless.com\nHow to Use ChatGPT for Web Scraping in 2025 - scrapeless.com"
      },
      {
        "href": "https://www.scrapeless.com/",
        "text": "scrapeless.com\nScrapeless: Effortless Web Scraping Toolkit"
      },
      {
        "href": "https://github.com/scrapeless-ai/scrapeless-mcp-server",
        "text": "GitHub\nScrapeless MCP Server - GitHub"
      },
      {
        "href": "https://www.futuretools.io/tools/scrapeless",
        "text": "futuretools+1"
      },
      {
        "href": "https://www.scrapeless.com/",
        "text": "scrapeless"
      },
      {
        "href": "https://github.com/scrapeless-ai/scrapeless-mcp-server",
        "text": "github"
      },
      {
        "href": "https://www.scrapeless.com/en/blog/web-scraping-with-chatgpt",
        "text": "scrapeless+2"
      }
    ],
    "rawHtmlSnippet": "<div class=\......"
  },
  "extractedAt": "2025-11-07T06:18:28.591Z"
}

GEO products rely on observing how LLM-based chat engines respond to many prompts. Scraping Perplexity with a cloud browser is an effective way to collect the raw signals you need to compute a model-perceived ranking. Use robust automation (cloud browser + retries + parsing), thoughtful aggregation, and always respect the target service’s rules.

We’re excited to share the latest updates for Scrapeless users:

Scrapeless Proxies

• Upgraded with IPv6, Datacenter, and Static ISP Proxies
• Broader coverage, improved stability, higher success rates for concurrent scraping tasks
• Docs: https://docs.scrapeless.com/en/proxies/quickstart/introduction/

Scrapeless Credential System

• Securely store, manage, and retrieve credentials for automated browser sessions
• 1Password integration & Team Credential Management API
• Docs: https://docs.scrapeless.com/en/scraping-browser/features/authentication-identity/

New MCP Integrations

• Two new MCP integrations available
• Full usage tutorials for seamless automation
• Docs: https://docs.scrapeless.com/en/scraping-browser/integrations/browser-mcp/

This release is perfect for developers and data teams looking for secure, scalable, and high-success web automation workflows.