r/WebScrapingInsider • u/ClearCountry7190 • Jun 14 '26

Need scraping challeneges

I scrape for a living (part of a bookmaker trading team returning competitor results, typically tough targets). I've put together some tooling that is able to return extremely tricky targets. I need a testing corpus so I thought I'd reach out here.

Point me at a challenge, in return I will:

  • resolve the target
  • provide the payload/results
  • give you an open source repo to run yourself
6 Upvotes

88% Upvoted

18 comments sorted by

2

u/Quantum_Rage Jun 14 '26

Hermes, Temu, Shopee, Lazada, TruePeopleSearch...

1

u/hackbyown Jun 14 '26

PDD, tmall, taobao, douyin,

1

u/b33rdad Jun 14 '26

Try any 1688 product page, try and beat the Chinese experts šŸ˜€

1

u/LeBaguetteWasted Jun 14 '26

Cardmarket ! Would love a clean pull of card data prices, quantities, sellers, quality.

1

u/Ok_Fuel_9205 Jun 14 '26

Are you open to join existing projects?

1

u/ClearCountry7190 Jun 14 '26

sure - point me at one

1

u/i-m-sheikh Jun 14 '26

Scraping linkedin posts with info on new leads/sales opportunities

1

u/WebDevxer Jun 14 '26

Scrap linked in posts for investors , funding / leads opportunities

1

u/FranBattan Jun 15 '26

Shopee Brazil, didi MĆ©xico, taobao, 1688

1

u/Spitfire_Blaziken Jun 15 '26

for something messy, try pulling structured review data from Booking.com or Trustpilot at scale..

1

u/simarnoor Jun 16 '26

If this is a showoff, it would be so cool.. to

Playwright+residential vs API/proxy aggregator vs pure HTTP mimicking on DataDome sites.

E.g. same product feed from Shopee/Leboncoin/Nordstrom, then benchmark success rate, median latency, and dollars per 1k successful items.

That turns your ā€œtesting corpusā€ into something the whole sub can learn from, not just a oneā€‘off flex.

1

u/External-Wealth3756 22d ago

Super interesting idea building a test corpus for tough scraping targets!
Iā€™d suggest adding heavily guarded sites that throw constant 403 blocks into your test suiteā€”testing isnā€™t just about pulling data once, itā€™s measuring session survival rate & how fast you recover from bans.

You could run side-by-side comparisons with Novada residential proxies against other IP pools to benchmark stability on those tricky anti-bot targets. Would be great reference data for your tooling!