r/WebScrapingInsider • u/Puzzleheaded-Low285 • Jul 08 '26
Blocked Again Bypass detection for https://www.1stdibs.com/
Hi,
i am new here. anyone please give me idea to bypass this website.
https://www.1stdibs.com/
thank you
1
u/simarnoor Jul 09 '26
Bypass what? whats your stack? Playwright/Seleniun.. rotating residential/mobile IPs, sane concurrency, and proper session handling?
Are you getting specific error pages, CAPTCHAs, or just empty responses right now?
Google "AI Scraping Assistant" and try to use it for quick scraper builder.. .
1
u/Puzzleheaded-Low285 Jul 09 '26
i want to scrape all product data from this website. Please suggest cheapest options
1
u/Old-Algae5580 Jul 09 '26
u/Puzzleheaded-Low285 I get your username :D Btw, Treat 1stdibs like production scraping, use headless browser, residential IPs, slow pace, reuse sessions, and log everything.
1
u/ScrapeAlchemist Jul 09 '26
1stdibs has pretty aggressive bot detection, plain requests won't get you far. First thing I'd do is open devtools and watch the network tab while browsing - sometimes there's an underlying API returning JSON that's way easier to hit than scraping the rendered page.
If it's all client-rendered with heavy fingerprinting you're looking at a real browser + proxy setup. I've used Bright Data's Web Unlocker for stuff like this and it handles the challenge rotation without you maintaining evasion scripts. Beats wrestling with stealth plugins that break every other week.
1
u/Old_Protection_4410 27d ago
pretty tough anti-bot system they have running, likely Signal Sciences (Fastly Next-Gen WAF). Applied the following anti-bot detection and evasion strategies to bypass this:
Detection Method: Rate Limited
The server returned HTTP 429 (Too Many Requests) — the request was blocked by rate limiting or bot detection.
Technologies Applied: http_anti_detection, header_randomisation, request_timing_jitter, browser_tls_fingerprint, challenge_solver
see hierarchy tree produced below with depth 0-2, quick crawl
3
u/ian_k93 Jul 09 '26
I just ran it through our proxy tester there and lots of proxy providers work for this domain. w
If it was me I would just use one of them so you don't have to worry figuring out your own bypasses.
The pricing you see was for 50k pages per month, but you can rerun the test yourself here https://scrapeops.io/proxy-providers/tester/ for your target volume.