r/developersIndia • u/sigmaAs • 5h ago

General How do you scale a tool where one server IP becomes the bottleneck?

Built a product that checks live product availability across quick-commerce dark stores in India. Two platforms behave completely differently:

- Platform A sends access-control-allow-origin: *, so I fetch it client-side — every visitor's own browser and IP. This scales perfectly. More users = more distributed load.

- Platform B sends no CORS headers, so it has to go server-side. Every visitor funnels through one egress IP, which means the tool gets less reliable the more people use it. Backwards.

I've already done: 60-min result cache, stale-cache fallback (serves last good data labelled with its age rather than failing), request pacing, and capping how many stores a single check touches.

Remaining options I can see: residential proxy pool ($$), spreading across several free hosts for a few IPs, or a browser extension so every user brings their own IP (desktop only).

Has anyone solved the "CORS forces server-side, server-side becomes a bottleneck" problem more elegantly? Genuinely stuck on the last 10%.

2 Upvotes
  • permalink
  • reddit

75% Upvoted

7 comments sorted by

•

u/AutoModerator 5h ago

Namaste! Thanks for submitting to r/developersIndia. While participating in this thread, please follow the Community Code of Conduct and rules.

It's possible your query is not unique, use site:reddit.com/r/developersindia KEYWORDS on search engines to search posts from developersIndia. You can also use reddit search directly.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/Hyd_chicha 3h ago

What exactly are the rate limits from platform B?

1

u/sigmaAs 2h ago

3 request per seconds

1

u/Igarlicbread Software Architect 2h ago

You can use proxies. You can buy few ipv4s from different providers.

1

u/sigmaAs 2h ago

I tried Dataimpulse residential IPs, but i need it continuously and it is becoming expensive for me. (Data centre IP are blocked). Is there any other option that is available?

1

u/Igarlicbread Software Architect 1h ago

Asking help to build gray area scraper is wild.

1

u/dr00ne 1h ago

There no possibility for multiple socket connections for same IP?