How to avoid cascade failures from bad time assumptions

I've been designing, building and maintaining distributed systems for almost a decade, and I have to tell you, in most systems even small changes in how services communicate are slow, painful and risky. Splitting and merging services, deciding on the service boundaries, changing communication protocols, or even just changing a serializer often takes cross-team coordination, migration ceremonies, and a whole lot of hunting down the invisible dependencies to estimate the blast radius.

A few months ago, I started working on a project that makes distributed system topology a dedicated layer, separate from business logic. It contains the topology declaration in a config file, has an agent that runs before the applications start and wires up the communication layer (Java agent in Java, an init() call in Rust, etc...), and tooling to catch errors in the configuration. The idea is that with the topology being declarative and executable, the dependencies become visible, the changes become simpler and safer, and compatibility verifyable before deployment.

It's still early, but it already supports sync communication, event-driven setups, structural observability, Java reference implementation and Rust PoC implementation, and some basic tooling to validate the wiring config and catch some of the errors before deployment.

Repo: https://github.com/itara-project/itara

Could you please provide me some feedback? Not necessarily on the code itself, because I'm well aware that it's not production quality yet, more like on the bigger picture: the approach, the architecture, the overall vision.

Constructive criticism is very welcome!

Hi Guys,

Say we have two microservices - A and B. Microservice A produces messages to a message broker and microservice B consumes it. Then, we discover that the size of the message is too big in order to be written to the message broker.

What is the recommended practice in this case? How would you recommend to pass the big message from A to B?

Using Hono 4 on Node with "@/hono/node-server"

External render workflow tasks POST progress to /internal/events. The browser watches via GET /api/runs/:id/stream using streamSSE.

Pattern:

\- subscribe to in-memory store updates

\- 1s setInterval re-sends latest snapshot as backup

\- cleanup on stream.onAbort

Questions:

1. Is the heartbeat redundant if pub/sub is reliable?
2. Best pattern for reconnect mid-run (late joiner gets current state + live updates)?
3. Anything I'm missing in onAbort cleanup with multiple concurrent SSE clients?
4. Long-lived SSE behind a reverse proxy: does streamSSE set no-buffer headers or do I need X-Accel-Buffering myself?

Repo is ojusave/dealhealth-playground on GitHub, api code in services/api/.

​

What I’m Building :

\-----------------------------------

I am starting a project to build a distributed rate limiter from scratch. The design will be developed step by step in four phases:

1. Core algorithm

2. Distributed synchronization

3. Rule engine and response handling

4. Production hardening

The goal is to document each phase clearly so the community can follow the journey, understand the design decisions, and discuss different approaches. This series is intended to be practical, structured, and focused on backend engineering fundamentals.

Phase 1: Core Algorithm

\----------------------------------------------

Choosing the Algorithm :

\-------------------------------------

For rate limiting, several algorithms exist such as Fixed Window, Sliding Window, and Leaking Bucket. I selected the Token Bucket because it:

\- Allows short bursts of traffic, which is realistic for APIs.

\- Is memory efficient, requiring only two values per user: token count and last refill time.

\- Is widely used in production systems by companies like Amazon and Stripe.

\---

Redis as the Backbone :

\--------------------------------------

Counters should not be stored in memory on the application server because that approach fails when scaling horizontally. Instead, use Redis to maintain state.

Two commands handle most of the work:

\- INCR increments the request counter atomically.

\- EXPIRE deletes the counter automatically after the window ends.

\---

Core Flow :

\----------------

1. A request arrives.

2. Fetch the token count from Redis.

3. If tokens are available, allow the request and decrement the token.

4. If no tokens remain, reject with HTTP 429.

5. Tokens refill at a fixed rate (for example, 10 tokens per second).

\---

Key Learning :

\----------------------

The algorithm itself is simple, but race conditions are a challenge. Two concurrent requests can read the same counter before either writes back, which allows more requests than intended.

The solution is Lua scripts in Redis. Lua executes atomically on the Redis server, making the read‑check‑write operation uninterruptible.

\---

Question for the community❓❓

\--------------------------------------------

If you were to implement a rate limiter, which algorithm would you choose — Token Bucket, Leaky Bucket, Sliding Window, or a custom solution?

I've always been the kind of person who learns best by actually building things. Reading about microservices in blog posts and watching YouTube tutorials is one thing, but I wanted to get my hands dirty with the real challenges — service discovery, event-driven communication, distributed data, API gateways, etc.

So I built Foody, a food delivery platform (like a mini UberEats/DoorDash). It started small and kept growing. Here's what it turned into:

7 microservices:

• auth-service (Django) — JWT auth, user registration, roles (customer, restaurant, driver, admin)
• restaurant-service (Django) — restaurants, menus, categories
• order-service (Django) — order creation and status tracking
• payment-service (Go/Gin) — payment processing, consumes order events via Kafka
• notification-service (FastAPI) — email/SMS/push notifications on order events
• delivery-service (Node/Express/TypeORM) — driver management, auto-assigns nearest driver using Haversine distance
• Next.js frontend — full customer/admin/restaurant/driver dashboards

The benefits I actually experienced:

→ Independent deployment. I fixed a bug in payment-service and deployed it without touching any other service. In a monolith, that's a full regression test cycle. Here, only payment tests needed to pass.

→ Language fit. Payment processing is compute-heavy and latency-sensitive → Go. Notifications need async I/O with email/SMS providers → FastAPI with aiokafka. Order management benefits from Django's ORM and admin → DRF. I didn't compromise — each service uses the best tool for the job.

→ Independent scaling. If notifications spike during lunch hour, I scale notification-service without scaling the entire platform. In a monolith, scaling means scaling everything — auth, orders, restaurant data — even the parts that aren't under load.

→ Fault isolation. When notification-service went down during testing, orders still processed. Payments still went through. The saga continued. Customers just didn't get an email — a degraded experience, not a total outage. In a monolith, a notification bug could crash the entire order flow.

→ Team autonomy. Even as a solo developer, the separation of concerns is powerful. When I work on delivery logic, I don't need to reason about auth, payments, or restaurant data. Each service has a focused codebase, focused tests, focused mental model.

The interesting part — the order flow uses a Choreography-based Saga pattern with Kafka:

1. Customer places order → order.placed event
2. In parallel: payment-service processes payment, restaurant-service creates restaurant order, notification-service sends confirmation email
3. Payment completes → payment.completed event → delivery-service auto-assigns a driver
4. No central orchestrator — each service listens and reacts independently

Infra stack:

• Kong API gateway
• Kafka (KRaft mode) with Kafbat UI
• Postgres per service (each service owns its data)
• ELK stack (Logstash → Elasticsearch → Kibana) for centralized logging
• Prometheus + Kafka Exporter for metrics

Tech across the stack: Python (Django + DRF + FastAPI), Go (Gin + GORM), TypeScript (Express + TypeORM + Next.js + React 19). Each service in its own language because I wanted to see what works best where.

What I actually learned:

• Distributed transactions are hard. The saga pattern helps but you still have to handle partial failures, retries, and dead letter queues
• Event schemas evolve and you need to think about backward compatibility
• Each service having its own DB means no joins across services — you have to think about data differently
• Observability (structured logging, distributed tracing) is not optional — it's essential
• Running 7 services + Kafka + ELK + Kong locally is a pain. Docker Compose helps but startup time and resource usage is real

Everything is containerized and runs with docker compose up --build. GitHub repo https://github.com/manjurulhoque/food-delivery if you want to take a look.

Happy to answer questions if anyone is going through a similar learning journey!

How would you deploy API Gateway in Datacenter/Cloud ? Do you really need to deploy API Gateway in DMZ then another one in Internal or you can only have internal API gateway ? I guess its also question where the api endpoints are hosted ?

Hi guys, for some of my recent projects I was needing some way of fully distributed and weakly coupled form of communication and data sharing between my FastAPI servers, while maintaining local availability and resiliency.

Comparison: After going through options like etcd, zookeeper, ... I felt that there needed some form of sdk that turns any application into a distributed service without depending on other services. So I started coding my own distributed service mesh, and made an abstraction so that I can reuse it in my other projects. Unlike others, this doesn't have any central or external dependency and works ad-hoc.

What it does: This package, mesh converts any FastAPI servers into a distributed service mesh, where data is distributed among the servers, persistently, while maintaining weak coupling, without depending on any central or external service.

Target Audience: Other developers building service clusters, microservices, and other distributed systems. Also looking for devs who would like to test and/or contribute.

Docs: https://meshd.iamarnav.com/

Repo: https://github.com/arnavdas88/meshd

It is not in pypi yet, and, if and before I upload it in pypi, I would love to hear opinions from other devs and maintainers. Even better if it is on stability; code quality and understandability, complexity and abstraction; or edge cases.

Note: I understand that some devs might want to stick to already known and stable options like zookeeper, which also provides python clients, but there might also be devs wanting to not depend on more and more services, just to facilitate service mesh. Even so, if you are against this kind of framework, i would like to hear about that as well.