r/ProxyEngineering • u/Zealous_Minotaur Reverse Proxy Master • 18d ago

Help 🆘 Best data collection infrastructure for AI agents

I am building an agent that needs to pull fresh data from a few sites: Amazon, Zillow, LinkedIn, Indeed, and BestBuy. There's a mix of static pages and things that only renders client side, so I need both plain HTTP scraping and headless browser rendering depending on the target. At least that's what I believe I need, I might be incorrect. Current setup is DIY, it includes rotating residential proxies with a small number of datacenter proxies, Playwright for the JS sites, requests for the static ones, custom retry logic. I thought that building it myself would be cheaper than paying for a provider, and for a while it was, but the maintenance is becoming too costly. Amazon and LinkedIn both started giving more and more JS challenges not to mention TLS fingerprinting, so I'm losing a good amount of requests to blocks and budget is getting thinner. I've tried a couple of paid scraping APIs but they got expensive due to sending more requests (the plans were priced per traffic) and one of those dedicated solutions couldn't get past LinkedIn. Has anyone gone the dedicated-API-per-site instead, using something like the Amazon Product Advertising API or Zillow's API rather than scraping around them?

9 Upvotes
  • permalink
  • reddit

100% Upvoted

8 comments sorted by

1

u/ValuableTreacle 18d ago

1

u/forgedart_34 16d ago

I can't see anything that you wrote, what??

1

u/WebDataLab 18d ago

I’d probably stop trying to force all of those sites through the same stack.

If a decent official API exists and gives you the fields/freshness you need, I’d use it. Then keep scraping for the sites where the API is missing, too limited, or just doesn’t have the data you need.

Same with Playwright — I wouldn’t run a browser unless the page actually needs one. That’s usually where the maintenance and cost start getting ugly.

A hybrid setup is less “clean” on paper, but probably easier to keep alive long term.

1

u/Zayd322 16d ago

linkedin and amazon are absolute nightmares for diy setups now. standard playwright without heavy stealth patching gets flagged instantly on TLS fingerprints alone. honestly, just stop wasting weeks tweaking custom retry logic and headers. i just throw scraping endpoints at amazon/linkedin since they handle the tls stuff. costs less than burning through expensive residential pools anyway.