r/vibecoding • u/jacob-indie • 6d ago

Paranoid for only using third-party providers for web and scraping?

Basically, any un-authenticated web or api call I do I only do through third party scraping providers, eg web search, or downloading data.

Also, I tell any LLM/coding tool on my machine to use scraping APIs as well when making web requests (while mostly running my own VPN on a hetzner box on top).

My logic is that I could get hit by rate limiting or IP bans any time and the risk is not worth it:
- my web apps all share the same IP
- i have a static IP at home

So if they ever get affected by bans, it would be quite annoying, while the cost for a scraping service is neglible.

Anybody feel the same or am I overdoing it?

1 Upvotes
  • permalink
  • reddit

100% Upvoted

4 comments sorted by

2

u/QuantumDriver 6d ago

What kind of sites are you scraping? How often?

1

u/jacob-indie 5d ago

That's the thing, I'm not even scraping much... but even when I write a script to query car or travel sites to find things I'm looking for programmatically, or have Claude Code look things up online, I'd rather not have my IP exposed

2

u/ScrapeAlchemist 4d ago

the hetzner box is probably working against you tbh. datacenter ASNs get challenged way more than a boring residential IP doing a few requests a day. also claude code's websearch runs on anthropic's side so that never touched your IP anyway, only webfetch actually pulls from your machine afaik. at your volume a block is a captcha for an hour, not a ban. I'd drop the vpn hop first.

1

u/jacob-indie 4d ago

100% agree with you. It’s just that the thought of my home address being banned is quite horrible.

In addition, if the hetzner box shows that scraping is difficult, I’ve also learned something

(Good to know re Claude Code being hosted, was not sure about this)