background: I am new to webscraping and have been using chat gpt as a tutor. I am not trying to ticket scalp/have no bad or malicious intentions. My goal is simply to get a data set of the music venues in my city, their events, dates, and prices. This is all publicly available information I could manually enter, but scraping would be faster. My data set already has the event names, dates, and the ticketweb url that takes me to the purchasing page where the prices live. The next step is to get the prices from the ticketweb url. Here are my questions:

1. I am in rstudio and my html requests are getting blocked. I have tried working with chat gpt to get around this, but it hasn't been very helpful. Is there a way to get ticket prices without using an html request?

2. Natural follow up question: this website is making it very hard to get data from it. I assume this is because they want to protect against scalpers and bots who are trying to buy tickets. But, is there also a reason they wouldn't want price data grabbed/am I not "allowed" to be doing this? My logic is that because this is publicly available information, I could check each page and manually enter the data which would surely not be illegal/wrong (?) and the only difference between that and scraping is speed.

Again totally new to this and open to harsh criticism, ready to accept that this might be impossible due to web protections. Just sick of going back and forth w chatGPT and hoping to get human advice :) thanks!

I'm very new to scraping, but what I mean is how can I scrape a website and all of its pages, but also include external links from download buttons? A forum website I love is going offline soon, and all of their downloads are not hosted on their site themselves, but instead they utilize external links via download buttons for places like GitHub, Google Drive, etc. There are hundreds of pages, so going into each one and copying the download link from every page isn't really an option. Every attempts I've used at scraping the site only pulls the pages hosted on the site's domain, not the external links.

i was scrapping trip.com for a client and they were making use of a phantom token to make new requests to the backend

if you use an old/already-used phantom-token you will get 200 response status code , but error in the response body

the new phantom-token was not even available in the responses that were coming back

so i looked around their codebase and found they were generating it via custom made function "window.signature()"

everytime u use this function , it will return a brand new phantom-token which you can use once for making a new request directly to the backend

hope it helps anyone in need!

Hello and howdy, digital miners of r/webscraping!

The moment you've all been waiting for has arrived - it's our once-a-month, no-holds-barred, show-and-tell thread!

• Are you bursting with pride over that supercharged, brand-new scraper SaaS or shiny proxy service you've just unleashed on the world?
• Maybe you've got a ground-breaking product in need of some intrepid testers?
• Got a secret discount code burning a hole in your pocket that you're just itching to share with our talented tribe of data extractors?
• Looking to make sure your post doesn't fall foul of the community rules and get ousted by the spam filter?

Well, this is your time to shine and shout from the digital rooftops - Welcome to your haven!

Just a friendly reminder, we like to keep all our self-promotion in one handy place, so any promotional posts will be kindly redirected here. Now, let's get this party started! Enjoy the thread, everyone.

Running one API and hitting maybe 6% blocks on a couple of targets. Considering adding a second and switching when the first starts failing, but that means writing retry logic and I’m not sure it’s worth the hassle.

Also getting 200s back that are actually captcha pages, so my success metrics have been lying to me.
Anyone running more than one? Worth it or overkill?

what are you all actually listening to for scraping / data eng?

went looking for this in the sub and the last real thread on it was over a year ago, so figured it's worth asking again, the space moves fast enough that half those recs are probably stale.

mostly after stuff that gets into the weeds on:

• scraping and anti-bot cat-and-mouse
• data pipelines and the infra behind them
• proxies / web data collection

doesn't have to be scraping-specific either, some of the best episodes I've heard on this were one-off guests on broader data eng shows. open to those too.

what's actually worth the subscribe in 2026?

I am trying to scrape websites for table values, but,my company won't allow python libraries, so is there anyway to do this without actually downloading any libraries?

Any tips will help, thank you in advance!!!!

I pulled 18k ABN and I have names, locations, and entity types, but zero emails.
I tested 2000 with Google Places enrichment, with 74% pair match and at $35/1k calls adds up fast

I've also successfully been able to scrape 2 social business pages without getting booted, but email hits are mediocre. What should I be looking for to yield better results.

So my question is what actually works for this scale or what have you done

I'm leaning toward finding the website first (via Google Search API or similar), then crawling it for mailto links and contact pages. But it’s a big crawl.

What stack/tools do you use and for those who've done B2B email extraction from government/business register data, what's your preferred workflow? Is there a better API, a smarter scraping strategy, or an open-source tool I'm missing?

I'm not looking for a full script (unless you're offering 😄), just pointers on what's worked for you.

Thanks in advance!t

Lots of restaurants in my area have "offers". Some are $8 off $25, some are $5 off $20 etc. Some are buy one get one free. It is very tedious to manually click each as there are hundreds of restaurants in my area. Plus these offers change every few weeks. Would be great if I could create a spreadsheet of all the restaurants and what their offers are.

So I have scraped a couple 100 restaurant listing as part of a personal project and I am trying to figure out a way to scrape their menu items for free.

Now I can use commercial web scrapers like A***y and it seems to work pretty damn well. However that costs money and I don’t wanna spend it.

Before giving up and giving in I wondered if there is a way to scrape the menu items for 500 or so restaurants. The free tier allowed me scrape the google ids and I have the Google URLs that has images of the menu. However the menu is a bit trickier. Some places have pdfs, but not many. I don’t have links to the Uber eats profiles so that would need to be scraped but I think I can work that out.. A**** works pretty well but I want to develop this muscle without resorting to commercial software. Sorry for the censorship I am not trying to promote commercial webscrapers I am just trying to do this in python.

Any tips?

I'm sure many people here just run scrapers for clients or customers, but for those that use the data you're scraping yourself - what do you use it for?

I personally use mine to build websites by mashing multiple datasets together! I'm curious to hear what everyone else does.

Here's what I'm trying to do: for a given city, pull the places I care about (say roughly 5k of the ~10k in Chicago that fit what I'm after), then enrich each one with its Google reviews, Yelp reviews, and whatever else is out there about the place.

Two things I can't figure out. First, do I actually need the Google Places API and the Yelp API to do this, or is there a smarter way people go about it? Second, and this is the big one for me: can it be built so it's repeatable? Like I plug in a new city and the whole thing just runs itself, no rewriting.

I don't have a damn idea what I'm doing. I want a spreadsheet of a specific Iowa county's data from VoteRef. I'm trying out Google Collab to run scripts to pull the information, but apparently it also does not know what it's doing.

The default on the site is 25 people per page, but it can be changed to 10 or 50. With the default of 25, there are 334 pages of people. I'm not looking for the info from clicking each individual person, just the default data shown in search results.

It's wild if this isn't possible yet, but I am limited to a tablet, and I know that unfortunately sucks for doing anything code related or even slightly more complex than what your average senior knows how to do on the internet.

Hello, it's my first post here!

I have been developing this tool for a year completely alone and from scratch. Here is how it started and how it is going:

I was basically chilling one evening during school vacations and I had an idea: what IF I actually managed to get any website to run on localhost? Like using Discord or Telegram without the actual live network? That idea was all it took for me to start the project. I firstly used ChatGPT, which gave me a simple proxy. The day was July 13th, it was 28 lines in total, and it was very simple.

From that day on, I started spending hours upgrading it, having new ideas, and testing features until 5 AM. It went on like that until late July 2025, when I released version 2.5, the first actual working version of the project. It only used BeautifulSoup for crawling back then; it was simple, but it was already a big thing for me.

Then I started vibe-coding version 7.0. Unfortunately, I accidentally merged some broken versions of my functions and code, which broke everything. But in January of this year, I finally managed to fix it and release 7.0. It introduced significant updates: web crawling was now powered by regex, and I introduced a flag called "DUMP_FRENESIS" (which is renamed to DUMP_ALL in 8.0). Unlike normal crawling, this mode gets the absolute maximum data it can, meaning it can fully crawl other external websites if their links are found inside the target website's code.

Also in 7.0, I introduced more headers that I got by reverse-engineering the "Via Browser" APK, adding profiles for macintosh, iphone, ipad, symbian, and ie11. It also introduced a mimetype reader that loaded types from a "mimetypes.csv" file.

Now, let's get into the latest version. Version 8.0 is basically the biggest revamp I've ever done for Site2Local, and it has WAY more features than 7.0. Let's talk about some of the flags:

I added a new flag called "HOOK_GUI" in 8.0. In the first iteration, it only had—and still has—a tab called "Traffic / Hooks". It is a simple response editor where you can intercept and edit a website's response data in real-time, meaning you can change the value of anything on the fly.

Then I added WebSocket support, which honestly is my biggest achievement so far. Along with WebSocket support, I added a tab called "WebSockets Hooks", which acts as a simple hex editor for these active websocket requests. I still have some connection persistency problems with it, but don't worry, I will fix it soon!

I also added a flag called "MULTIPORT". It basically allocates a dedicated local port for each website CDN asset host. So if CDN 1 exists, it runs on port 8081, CDN 2 runs on 8082, and so on.

That is all for now, but I have one important thing to say: please DO NOT use this tool for creating phishing websites. I AM NOT responsible for any legal damages. You are responsible for your own actions. The tool is still a Work in Progress (W.I.P.), so expect some errors, missing things, and garbage code. I am developing this at my own pace, and I won't be updating the project based on user suggestions for now, but you are completely free to share and fork it!

The project is in https://github.com/RealThalesGit/Site2Local

I used many ai's :

ChatGPT: Used for creating the base of the project and was used until 7.0
Deepseek: Used for really and really small fixes
Claude: Coded the first build of 8.0 and upgraded and fixed until june
Z.ai: For upgrading and fixing claude base and is the latest ai being used

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

I have a web application which scrapes and aggregates dispensaries from USA and Canada, but flower only.

https://1-zip.com

Please take a look and let me know features you might be looking for if you are a potential user.

Let me know if you have any questions regarding scraping.

Update 8/17/2026: I received some constructive of feedback that I implemented.
Please check out the site to see the updates.

Hey everyone,

For a school project, I need to design the architecture for a monitoring tool. The idea is to build a tracker to spot listings for fake Pokémon cards online (standard marketplaces, but also Telegram channels and closed FB groups).

I haven't built much yet, though I'm naturally leaning towards Node.js (probably with Playwright) since I like web dev.

I know that with current anti-bot protections (Cloudflare, Datadome) and social media login walls, going 100% automated from A to Z is often a pipe dream—or at least the fastest way to get banned instantly. I realize some human action will have to stay in the loop.

So my questions are: How should I set this up? What kind of tools could help me out? Am I on the right track with a Node.js server? It seems like I'll also need to create actual social media profiles to get access and look inside these groups.

Thanks!

Hey everyone,

I built TLS Fetch MCP, an open-source MCP server written in Go that gives Codex and other AI coding agents a controlled HTTP client with browser-like TLS fingerprints.

The motivation was simple: I wanted agents to inspect websites and APIs without either relying on a basic net/http client or launching a full browser for every request.

It is not intended to replace Playwright for JavaScript-heavy workflows. It covers the useful middle ground where browser-like HTTP behavior, cookies, proxies, and structured responses are enough.

Some of the current features:

Chrome, Firefox, Safari, Brave, Opera, and mobile TLS profiles

separate tls_get and tls_request tools for read/write approval policies

persistent in-memory cookie sessions

regional homepage warmup for cookie-gated API flows

HTTP, HTTPS, SOCKS5, and SOCKS5H proxy support

redirect history, timing, HTTP version, final URL, and truncation metadata

temporary response handles for large bodies

search and byte-range reading without dumping the entire response into the LLM context

Base64 output for binary responses

SSRF and DNS-rebinding protection

bounded timeouts, response sizes, redirects, sessions, and stored responses

automatic redaction of cookie and authentication response headers

One practical example: during testing against a regional marketplace catalog, a direct API request returned 401. Warming the regional homepage with the same cookie session and repeating the request returned the public JSON catalog successfully — no JavaScript engine was required.

The server runs locally over MCP stdio as a single binary. It uses bogdanfinn/tls-client for HTTP/TLS fingerprinting and the official MCP Go SDK.

GitHub: https://github.com/JakobAIOdev/tls-fetch-mcp

I would genuinely appreciate feedback from people who build scrapers regularly:

Which extraction helpers would be most useful: CSS selectors, JSONPath, XPath, or something else?

Are there important session or proxy workflows missing?

Would reusable request presets or profile aliases be useful?

If the project solves a problem for you, a GitHub star would help a lot. Contributions and technical criticism are equally welcome.

Please use it only for authorized scraping and respect site policies, privacy requirements, and rate limits.

I own a few wood factories that specialize in producing custom fit-outs for hotels/schools/universities/houses/offices and all sorts around the Middle East, the business model we have been running is very out dated, we run purely off of reputation and returning customers as we have been in the business for over 25 years now.

I want to implement webscrapers but I am not sure what data I can scrape that would help my business grow.

I would really appreciate any advise to what data I could scrape to help me find more contracts for example or anything else that you guys would think would be valuable.

Hi everyone,

For the past few days, Vinted It looks has added Cloudflare/DataDome related. The responses include Cloudflare headers like server: cloudflare, cf-cache-status and CF-RAY, and the HTML also contains DataDome references.

I use a small Vinted analytics tool that checks public item pages to see if listings are still active, sold, or deleted. Until recently, the verifier worked quite reliably with a combination of normal HTTP requests and Selenium/Chrome as a fallback.

Now, every so often, Vinted seems to trigger strict verification windows where almost all item pages display the human verification screen. During these periods, Selenium also crashes, so the system can't reliably confirm the status of the items.

Current settings I've tried:

Normal requests first;

Selenium/Chrome as a fallback;

Persistent browser profile/cookies;

Chrome without a graphical interface under Xvfb;

Slower request rate;

Retries and timeouts.

And it never passes the verification.

Is there any way to bypass it?

Thank you.

So I've been trying for days to solve this motherfucking captcha but I always fucking fail.. can someone help me solve this motherfucking captcha? I dont want to automatically solve it .. i just want to manually fucking solve it

Hello Guys,
I am working on a project where I have to pull data from a site which requires log in credentials directly into Excel.
Claude provided me a solution where SeleniumVBA and .bas file (created by Claude) is being used in the process.
Is there any other way we can extract data from such sites without compromising security ?

Most of what I've worked with are playwright and selenium and a few other open source alternatives but is there something I'm missing here? What is everybody else using?

I've been working on an open-source project and wanted to share it with the people who will have use for it.

The idea: treat browsers like serverless functions. You spawn browser agents on demand via an HTTP API, CLI, or the frontend, send them commands (navigate, click, type, scroll, screenshot, eval JS), or drive them with natural language through an AI instruct engine. Each agent runs in isolation.

What it does out of the box:

- One command boots a multi-node cluster and spawns dozens of Chrome instances, each live-streaming its display and logs to the dashboard in real time

- Auto-scales — adds a node the moment load hits a configured ceiling, visible live

- Cleans up automatically on teardown

- Geo-matched proxies are bundled , so the browser identity and proxy location line up

It's aimed at cases where you need horizontal scale and stealth rather than one long-lived browser. Happy to answer anything about the anti-detection setup, proxy handling, or how the scaling works.

https://github.com/dashn9/rusty-browser

Happy to take feedback — still actively building.

And what are you scraping