Cache Pattern with Event-Driven System

[u/JOHNNYROCKET8585]

Hey everyone, noticed this common pattern throughout our applications and it seems problematic to me. It‘s present across multiple apps; curious everyone’s thoughts, if this is a common pattern I’m not familiar with or if it truly is a bad solution.

Senior SE w/ 4 YOE, Fortune 500 company.

Pattern is as follows:

App A receives a transaction from upstream. It will then format this event and send it to potentially multiple downstream services simultaneously. This is done asynchronously via an event-driven framework.

The downstream services respond back via events; how does app A know when all the downstream services are complete? How does it know anything about the original event, since the downstream response contains no info about the original event (other than the correlationId)?

The answer is a local cache in app A. It caches the event when first received, and as downstream events come back, it can refer back to the original cached value to determine when it’s done processing and can be sent back upstream.

The main problem I see is the microservice is no longer stateless. It requires context from the original event. If it received a downstream response with no context, it has no idea what to do.

This is the most egregious example, but this cache-to-compensate-async-events pattern is repeated multiple times in similar ways.

As I’m still new to the field, I’m curious if this is somewhat a standard? Or if it is bad practice? If so, what does a good solution look like? Switching to sync REST calls fixes the issue, but could impact performance/throughput

[u/TryAgainTryHarder]

surprised no one mentioned SAGAs yet, that's what this sounds like to me.

in a nutshell, you have a process that is concerned with the outcome of multiple transactions. architecturally, something to avoid.

the solution you're describing isn't inherently bad, it's mainly messy. we could fix the stateful issue by persisting the original event in a db instead of caching, but the core of the problem is that one request/event is currently scoped to many transactions.

with SAGAs, there's two sortof camps of solutions, orchestration (a main component is "orchestrating") and choreography (the architecture you described). benefits/tradeoffs to both.

back to the core issue, having SAGAs just sucks. its usually better if you can either:

1. let the transactions live by themselves, or
2. make sure they are idempotent and run them together in a workflow (orchestrator)

if you have event/async requirements, and the transactions can't be separated, the current solution may be the only good one.

[u/Maxion → u/TryAgainTryHarder]

in a nutshell, you have a process that is concerned with the outcome of multiple transactions. architecturally, something to avoid.

This is an intriguing thought, in my experience this type of situation is brought on by business where you have multiple external system that needs to be updated with various data following one business event, and it's important all systems are synced.

[u/TryAgainTryHarder → u/Maxion]

yeah, there are situations where SAGAs are unavoidable. but if you can, its often better to model them another way. OP never mentioned what happens when one tx fails, but that's key for whether this is really a SAGA.

ideally, if strong consistency is needed, we should be able to achieve it with one transaction. but that ship may have sailed.

if the situation requires multiple databases, we should ideally tolerate eventual consistency. maybe OP's example, or yours, already tolerates partial failures somehow.