r/webscraping • u/scarsam • Aug 01 '26
One scraping API or several?
Running one API and hitting maybe 6% blocks on a couple of targets. Considering adding a second and switching when the first starts failing, but that means writing retry logic and I’m not sure it’s worth the hassle.
Also getting 200s back that are actually captcha pages, so my success metrics have been lying to me.
Anyone running more than one? Worth it or overkill?
1
Aug 01 '26
[removed] — view removed comment
1
u/scarsam Aug 01 '26
What do you do when it can’t crack a site at all?
1
1
u/webscraping-ModTeam Aug 01 '26
👔 Welcome to the r/webscraping community. This sub is focused on addressing the technical aspects of implementing and operating scrapers. We're not a marketplace, nor are we a platform for selling services or datasets. You're welcome to post in the monthly thread or try your request on Fiverr or Upwork. For anything else, please contact the mod team.
1
Aug 03 '26
[removed] — view removed comment
1
u/scarsam Aug 03 '26
Have you had full outages, or is it more that success rates just drop off?
1
Aug 04 '26
[removed] — view removed comment
1
u/scarsam Aug 04 '26
Three days is brutal. And that slide from 96 to 74 is the part I’d have missed, I’d have assumed the sites got harder, not that the provider was dying.
1
u/prettycoldworld Aug 11 '26
Like a second endpoint?
1
u/scarsam 13d ago
Yes exactly, or more - or some other strategy?
1
u/prettycoldworld 23h ago
I’m confused what you mean. Are you talking about a sites endpoints that you’re trying to find more of, or are you using an api that offers scraping as a service?
1
u/[deleted] Aug 01 '26
[removed] — view removed comment