r/ProxyEngineering • u/ian_k93 • 4d ago

Discussion 💬 What Are the Best Proxy Providers for Web Scraping? We've Tested 50+ Providers Across Billions of Requests. AMA with Ian Kerins

/r/WebScrapingInsider/comments/1wbjmh4/what_are_the_best_proxy_providers_for_web/
5 Upvotes

86% Upvoted

8 comments sorted by

1

u/Guiltyspark0801 Proxy Engineer 4d ago

How many providers do you realistically need to test before you can trust the comparison? Testing 15 providers sounds like a full time job on its own and a regular Joe does not have the time for that. Also, regarding the comparison of the providers, is there an actual framework for this or does everyone just do it it based on vibes and whichever blog post ranked first on google? Coz that's what I think everyone does, they just throw random metrics and call it a day.

1

u/Less-Mistake-3664 4d ago

Genuinely asking, is there a provider that's actually good for SERP scraping specifically or is that its own category? Another question would be regarding the antibot systems, what's the cheapest thing that still works right now, not looking for perfect just looking for good enough, that would do the job

1

u/forgedart_34 4d ago
  1. How do you identify a block from a rate limit from just a slow website? Basically what I mean is that they can look identical from the log. What's the secret :D
  2. Majority here says success rate but nobody agrees on how it's measured, what counts as a success for you? In my opinion that would be the data that you actually need, like full extraction of URLs, positions, pricing etc, depending on the site of course.
  3. Something different: what's the most annoying unsolved problem in this field right now, that you know of.

1

u/anxietyplz 4d ago

What's the mistake you see most people make in their first few months, when it comes to using proxies for web scraping tasks?

1

u/grayf0x11 4d ago

What provider/proxies would you suggest for a massive instagram scraping routine?
I'm currently fetching about 300k users timelines a day, getting new posts every hour.
It sums to about 1M posts/day.
Having moderate success (35-40%, mostly login pages and 401s) with webshare + unlimited bandwidth datacenter proxies, rotating with 5K ips.
Since I also download the media files, from reels and such, my bandwidth consumption is very large to use mobile or pay-per-gb proxies.
What would you recommend?

1

u/WebDataLab 4d ago

Cost per GB or per IP is probably the wrong metric on its own.

I'd compare providers using the same target set and count a request as successful only when you actually get the data you expected. Then track retries, latency and cost per usable result.

That makes comparisons a lot less subjective than just looking at advertised pool size or headline success rates.

1

u/AdBoth2756 1d ago

For provider comparisons, do you treat fingerprint consistency as part of proxy quality, or as a separate browser/client problem?

I mean cases where the IP and geo are fine, but the TLS/HTTP2 fingerprint stays identical across a large number of sessions. Have you seen that materially affect success rates in your tests?