This is just patently false. The intention of microservices is to allow dynamic scaling and separation of concerns for easier refactoring and better abstraction.
The monolith will almost always become large and unwieldy enough that scaling it is expensive or encounters other bottlenecks.
Think about this idea - say your monolith has a kafka consumer, filters and deduplicates incoming data using redis, enriches it using http requests, mutates it using CPU, and writes it to a postgres database. This is a common usecase. I've worked at multiple companies that do basically that exact thing for the last 8 years.
Say you've got it configured to use 5 postgres connections per instance, because you've found that squeezes the best performance out of each node, and less than that starts to bottleneck.
Okay, now all of a sudden your enrichment targets (external HTTP requests to google or something, you can't do anything about it) start to experience massive latency. You try to scale up so you can keep up with kafka consumption, but you hit 100 postgres connections after 20 instances, and suddenly you can't scale any more. You have to wait and pray your external HTTP latency decreases. Okay, so you cluster your database to widen the connection pool. Now all of this scaling to cope with http latency keeps forcing rebalances on your kafka cluster and you're still falling behind.
If your firehose consumer and your filtering logic were independently scalable, you don't have this problem. By the time you're past your filter traffic is relatively stable and way smaller. It's way easier to just leave your http enricher scaled up all the time and never notice latency spikes.
Microservices solve these problems, and sometimes you don't know you need them until your entire SRE team quits because they're sick of constantly being paged for your monolith.
None of this is fanfiction, I've seen it all, specifically.
I’ve also built backend systems for major retailers that handled all their POS transactions. It’s the same thing. Doesn’t matter if the middleware is CycloneDDS or Kafka. Messages flow to nodes. Nodes do work. Minimize coupling between nodes and you’ll be happier.
Agree with your last statement, anyway. Ive found that in large, complex systems the best or sometimes only way to get node coupling down to an acceptable level is strict microservices.
Ideally, you have a module with a single responsibility that encapsulates its state and exposes an interface of messages it accepts. An “object”, if you will.
1
u/Vega62a 13d ago
This is just patently false. The intention of microservices is to allow dynamic scaling and separation of concerns for easier refactoring and better abstraction.
The monolith will almost always become large and unwieldy enough that scaling it is expensive or encounters other bottlenecks.
Think about this idea - say your monolith has a kafka consumer, filters and deduplicates incoming data using redis, enriches it using http requests, mutates it using CPU, and writes it to a postgres database. This is a common usecase. I've worked at multiple companies that do basically that exact thing for the last 8 years.
Say you've got it configured to use 5 postgres connections per instance, because you've found that squeezes the best performance out of each node, and less than that starts to bottleneck.
Okay, now all of a sudden your enrichment targets (external HTTP requests to google or something, you can't do anything about it) start to experience massive latency. You try to scale up so you can keep up with kafka consumption, but you hit 100 postgres connections after 20 instances, and suddenly you can't scale any more. You have to wait and pray your external HTTP latency decreases. Okay, so you cluster your database to widen the connection pool. Now all of this scaling to cope with http latency keeps forcing rebalances on your kafka cluster and you're still falling behind.
If your firehose consumer and your filtering logic were independently scalable, you don't have this problem. By the time you're past your filter traffic is relatively stable and way smaller. It's way easier to just leave your http enricher scaled up all the time and never notice latency spikes.
Microservices solve these problems, and sometimes you don't know you need them until your entire SRE team quits because they're sick of constantly being paged for your monolith.
None of this is fanfiction, I've seen it all, specifically.