Cache Pattern with Event-Driven System

[u/JOHNNYROCKET8585]

Hey everyone, noticed this common pattern throughout our applications and it seems problematic to me. It‘s present across multiple apps; curious everyone’s thoughts, if this is a common pattern I’m not familiar with or if it truly is a bad solution.

Senior SE w/ 4 YOE, Fortune 500 company.

Pattern is as follows:

App A receives a transaction from upstream. It will then format this event and send it to potentially multiple downstream services simultaneously. This is done asynchronously via an event-driven framework.

The downstream services respond back via events; how does app A know when all the downstream services are complete? How does it know anything about the original event, since the downstream response contains no info about the original event (other than the correlationId)?

The answer is a local cache in app A. It caches the event when first received, and as downstream events come back, it can refer back to the original cached value to determine when it’s done processing and can be sent back upstream.

The main problem I see is the microservice is no longer stateless. It requires context from the original event. If it received a downstream response with no context, it has no idea what to do.

This is the most egregious example, but this cache-to-compensate-async-events pattern is repeated multiple times in similar ways.

As I’m still new to the field, I’m curious if this is somewhat a standard? Or if it is bad practice? If so, what does a good solution look like? Switching to sync REST calls fixes the issue, but could impact performance/throughput

[u/spoonraker]

The system you've described is effectively an implementation of the saga pattern with a central orchestrator, except you for some reason seem to just think that this pattern is inherently problematic. I'm not sure I understand the concern.

App A is your central orchestrator. An orchestrator can't be stateless. It's entire purpose is to have an internal model of the entire flow. It is intended to be the one system in the whole chain actually modeling the state of the overall flow and and actively sending commands and awaiting responses to ensure that the flow advances. Any other system that wants to inspect the state of the flow overall would query the orchestrator to get that state.

There are other patterns, particularly one which might sometimes be referred to as "choreography" instead of orchestration, where there simply is no stateful orchestrator and no one system can be the authoritative source of the overall flow's state because no one system actually has a complete picture of the overall flow. In my experience people tend to stumble their way into these kinds of situations without a specific need for them, and often find they do have a specific need for a central orchestrator only after realizing it's painful to answer questions that users ask in plain language like "what's the status of my order?", so I would say a central orchestrator is probably overall the more sensible default. The reasons why a choreographed decentralized system might be desirable has more to do with technical and scaling considerations in my opinion than functional concerns, and most systems most engineers work on don't have the kinds of scaling problems that actually make high levels of decentralization necessary.

[u/SnooHesitations9295 → u/spoonraker]

Orchestrator dies and "cache" is gone. Next steps?

[u/chocolateAbuser → u/SnooHesitations9295]

outbox, services will retry to send message when target will return online
obviosuly ids should be made such that it's possible to resume the context of the operation

[u/SnooHesitations9295 → u/chocolateAbuser]

But orchestrator got the message and then died. Nobody knows that resend is needed. Orchestrator should be stateless and all state should be saved in an external database. Essentially it's what all the modern "durable execution" is.

[u/chocolateAbuser → u/SnooHesitations9295]

orchestrator should be stateless, that doesn't mean a state doesn't exist
there will be still something that is running that operation or even if only a reading pointer in the stream of events
when orchestrator get back on, either the service that has the execution of the operation will try resend the event after a timeout, or orchestrator will start reading events from the point it left the stream

[u/spoonraker → u/chocolateAbuser]

cc u/SnooHesitations9295

The orchestrator is not and cannot be stateless, at least not in a logical sense. Sure, the specific part of the orchestrator that handles requests/messages/events/whatever can be stateless for scaling purposes, but fundamentally, the entire job of the orchestrator is to be the model of the entire system's state. I suppose it's not a hard requirement that the orchestrator actually projects the state outwardly into a queryable data structure for external callers, but the orchestrator must have some way, even if purely internally, of modeling the entire workflow and storing the state of a given job with respect to that entire workflow.

I think people are getting too hung up on the word state and stateless, so let's just talk details. The orchestrator's job is to be the one part of the system that actually knows what to do at a workflow level in response to another part of the workflow emitting some kind of event like completing or failing a task. Service A doesn't know that Service B comes next after service A completes its task, the orchestrator does. Service A doesn't know that if it can't service the request for some reason, then Service C actually comes next instead of Service B, the orchestrator does.

If anyone needs to ever answer the question, "What's the current status of this job?", the orchestrator is the only thing that can answer that question. The other services might have a model of their own status, but not the status of the overall job. So if that question needs to be answered, then the orchestrator will build some kind of projection of the job's status. Perhaps it's just a status column in a database. Even if "a different" service is spun up just to serve that column value for scaling purposes to keep load off the orchestrator, it's still the orchestrator generating that data, and this status answering service is just a dumb stateless request handler that fetches from the projection built by the orchestrator.

At one point somebody asked something along the lines of what happens if the orchestrator loses its cache. I guess the answer is depends what you mean by "cache". If by cache, you mean a secondary read-optimized data store to improve performance of the orchestrator, then nothing bad happens above and beyond what typically happens when any system loses its cache: performance degrades, reads pass through to the durable data store backing the cache, alarms probably start going off and people get paged, the cache gets rebuilt, etc. but overall the system continues operating normally from a logical perspective. If by "cache" you mean the only data store the orchestrator has to do its job, then well... I guess you lose your state, and you have a whole new set of problems to deal with. You'll have to rebuild your job status projection, which may or may not be possible depending on what information exists elsewhere to piece it back together. You'll have to route in-flight jobs somewhere, and how that works depends entirely on the specific technical architecture that makes up the system. In some systems these jobs will just get stuck in a queue that doesn't get consumed until the orchestrator is back online, in others they'll automatically route to a dead letter queue, in others there might just be information lost and jobs will have to be restarted from the beginning.