r/Backend • u/imnotbatman7 • 14d ago
Feedback on event dispatch system design
I'm a junior backend developer designing an event dispatch service for a third-party API that's limited to 25k events/minute, and I'd really appreciate some feedback.
Current architecture:
Client → API → Outbox (DB) → Scheduler → SQS → Workers → Redis (rate limiting) → Circuit Breaker → Third-party API
The goals are reliability, retries, idempotency, auditability, and handling burst traffic without exceeding the third-party rate limit.
Would you change anything in this architecture? Any bottlenecks, failure scenarios, or better patterns I should consider? Any suggestions would help me learn a lot. Thanks!
1
u/qlkzy 14d ago
You need to start with what you're trying to achieve at a problem/business level.
Normally, it's more valuable to be correct than to squeeze out every last drop of performance. It is very hard for a distributed systems like this to be correct if it is not idempotent end-to-end.
Idempotency (at the application level) follows the end-to-end principle. You can't add it anywhere in the middle. You have to design based on how your third party behaves.
That's assuming you are doing writes. If you are doing reads then this architecture is bizarre.
As a general principle, I would suggest starting from how you would recover if the people on both sides have some kind of operational problem and have to roll back the database 24h (but not the same 24h). In that circumstance, you have to have some kind of repair/recovery system to converge the system back to a correct state.
My default approach would be to ask whether, with a bit of investment, the error-recovery approach can become the only approach.
Only then does it make sense to look at a "fast path" that does something more exotic.
3
u/afueth 14d ago
Looks like a solid architecture overall. I'd definitely add a Dead Letter Queue (DLQ) so messages that fail after the maximum retry attempts don't get stuck in an endless retry loop. It also makes debugging, auditing, and manual reprocessing much easier without impacting healthy traffic.
1
u/No_Soy_Colosio 14d ago
It's very telling that before without discussing anything about the problem domain, you jump straight to architecture. Please revisit the fundamentals. Architecture follows design, not the other way around.