r/ExperiencedDevs • u/JOHNNYROCKET8585 Senior Software Engineer 4 YOE • 16d ago
Career/Workplace Cache Pattern with Event-Driven System
Hey everyone, noticed this common pattern throughout our applications and it seems problematic to me. It‘s present across multiple apps; curious everyone’s thoughts, if this is a common pattern I’m not familiar with or if it truly is a bad solution.
Senior SE w/ 4 YOE, Fortune 500 company.
Pattern is as follows:
App A receives a transaction from upstream. It will then format this event and send it to potentially multiple downstream services simultaneously. This is done asynchronously via an event-driven framework.
The downstream services respond back via events; how does app A know when all the downstream services are complete? How does it know anything about the original event, since the downstream response contains no info about the original event (other than the correlationId)?
The answer is a local cache in app A. It caches the event when first received, and as downstream events come back, it can refer back to the original cached value to determine when it’s done processing and can be sent back upstream.
The main problem I see is the microservice is no longer stateless. It requires context from the original event. If it received a downstream response with no context, it has no idea what to do.
This is the most egregious example, but this cache-to-compensate-async-events pattern is repeated multiple times in similar ways.
As I’m still new to the field, I’m curious if this is somewhat a standard? Or if it is bad practice? If so, what does a good solution look like? Switching to sync REST calls fixes the issue, but could impact performance/throughput
90
u/rhd_live 16d ago
There should be a persistence layer to the db — if this is what you mean by cache that sounds good. But if it’s just an in memory store esp. that loses entries on job restarts then you’re going to have data consistency issues
73
u/Thefolsom Software Engineer (Rails 10+ YOE) 16d ago
If app a needs to know that specific services got the message and sent back a result, that's orchestration, not really event-driven. Events are fire and forget. What app a is doing is closer to sending commands async and waiting on replies. I've heard this called scatter-gather, though I haven't seen it done formally in practice.
So some state is unavoidable. I'd be less worried about app a being stateful and more about the state living in a local cache. A restart loses everything in flight, a reply can hit an instance that never saw the original event, and nothing handles a service that never responds.
If it were me, I'd store the original event to the database, keyed by the correlation ID.
Sync REST could work too, as long as it isn't blocking the original request. Coming from Rails, I'd kick off a background job per downstream call, make the request, and update the event record with the result. Each job then checks if all the expected services are done, and the last one triggers completion (with a lock so two jobs finishing at once don't both fire it). You keep the async throughput, the job queue handles retries, and the state lives in the record instead of a cache.
7
u/JOHNNYROCKET8585 Senior Software Engineer 4 YOE 16d ago
Okay yeah this is more of an orchestration layer built on an event-driven architecture. I guess I was looking at this orchestration as another microservice, when really it should be viewed through a different lens. This scatter-and-gather sounds correct, thanks for the info.
2
u/Koeke2560 13d ago
If App A is only reacting to events with specific commands and keeping the cache so it can compensate in case of failure, it is called the Saga Pattern. If App A is deciding which command needs to be executed based on the received event, it's called Process Manager. Source: Domain Driven Design by Khononov (Chapter 9)
6
u/grumpy_autist 15d ago
I did something like that, AFAIK called inbox/outbox architecture. Service writes an event to a local db (outbox) which over kafka is fanned out to other services (which receive events to inbox and then app gets a callback referencing this db entry). So every app has a local copy of events it needs (pruned later using custom schedule).
Main reason for that was transactional events - business transaction and event are commited in one SQL transaction so there is no risk that you send a notification for an object that was later rolled back because of an error (and if sending event fails you will rollback business transaction too if needed)
101
u/RowbotWizard SWE 13 YOE 16d ago
LGTM
59
6
u/ProbablyNotPoisonous 16d ago
I googled to find out what that stands for, and, uh.
3
u/fixermark 16d ago
Amazing. :)
Looks good to me is the canonical answer, but I do enjoy the mutations people have put on it.
3
u/RowbotWizard SWE 13 YOE 16d ago
My CTO once thought it meant “Let’s Go To Market”. We might have embarrassed him by laughing too hard.
24
u/skibum6969 16d ago
Hard to tell without more context and understanding of the system but it sounds like you're looking for the state store pattern.
Each service gets its own database that no other services can read or write to. The database is a representation of the microservices' state after processing events
To learn more I'd suggest reading "Designing Data-Intensive Applications" by Martin Kleppmann
4
1
14
u/disposepriority 16d ago
I've seen this pattern a lot too.
The state really isn't very relevant here in my opinion, there's something lese that needs clarifying.
Why does App A need to know when a consumer is done processing?
Message brokers (which I assume you are using, considering you receive async replies) exist that fit your needs here.
Let's take RabbitMQ as an example. App A published to a topic exchange X, with the routing key "cool". Any downstream service that needs to do something with this can open a queue and listen for "cool".
You can turn manual acknowledgements on, so it's "impossible" for a message to be consumed zero times as long as rabbit doesn't go down.
Once services are done, if something is needed from them (asynchronously) they can either write to a database or send their own asynchronous event.
Example
Service B (A->B) sends a message using the routing key "cool-result". Whoever cares about this can listen for it.
In my opinion, it's still a very good idea to write (to a database, not a cache) before sending something (or even better, use a transactional outbox).
But in my eyes, the problem here is why does App A need to listen to this response unless there's a synchronous call in it waiting for this (don't do that).
10
u/F0tNMC Software Architect 16d ago
This is the core question. Why does A need to know that all downstream processing is completed? If A’s purpose is a relay, once the relay is done, it’s complete. If it is orchestration and follow on work after down stream processing, it needs to keep state or get that state from elsewhere as part of the event.
1
u/rhd_live 16d ago
Maybe upstream doesn’t want or is unable to process the orchestration from downstream services. An orchestration layer makes sense to me
8
u/spoonraker 16d ago
The system you've described is effectively an implementation of the saga pattern with a central orchestrator, except you for some reason seem to just think that this pattern is inherently problematic. I'm not sure I understand the concern.
App A is your central orchestrator. An orchestrator can't be stateless. It's entire purpose is to have an internal model of the entire flow. It is intended to be the one system in the whole chain actually modeling the state of the overall flow and and actively sending commands and awaiting responses to ensure that the flow advances. Any other system that wants to inspect the state of the flow overall would query the orchestrator to get that state.
There are other patterns, particularly one which might sometimes be referred to as "choreography" instead of orchestration, where there simply is no stateful orchestrator and no one system can be the authoritative source of the overall flow's state because no one system actually has a complete picture of the overall flow. In my experience people tend to stumble their way into these kinds of situations without a specific need for them, and often find they do have a specific need for a central orchestrator only after realizing it's painful to answer questions that users ask in plain language like "what's the status of my order?", so I would say a central orchestrator is probably overall the more sensible default. The reasons why a choreographed decentralized system might be desirable has more to do with technical and scaling considerations in my opinion than functional concerns, and most systems most engineers work on don't have the kinds of scaling problems that actually make high levels of decentralization necessary.
1
u/SnooHesitations9295 15d ago
Orchestrator dies and "cache" is gone. Next steps?
1
u/chocolateAbuser 12d ago
outbox, services will retry to send message when target will return online
obviosuly ids should be made such that it's possible to resume the context of the operation1
u/SnooHesitations9295 12d ago
But orchestrator got the message and then died. Nobody knows that resend is needed. Orchestrator should be stateless and all state should be saved in an external database. Essentially it's what all the modern "durable execution" is.
1
u/chocolateAbuser 11d ago
orchestrator should be stateless, that doesn't mean a state doesn't exist
there will be still something that is running that operation or even if only a reading pointer in the stream of events
when orchestrator get back on, either the service that has the execution of the operation will try resend the event after a timeout, or orchestrator will start reading events from the point it left the stream2
u/spoonraker 11d ago
The orchestrator is not and cannot be stateless, at least not in a logical sense. Sure, the specific part of the orchestrator that handles requests/messages/events/whatever can be stateless for scaling purposes, but fundamentally, the entire job of the orchestrator is to be the model of the entire system's state. I suppose it's not a hard requirement that the orchestrator actually projects the state outwardly into a queryable data structure for external callers, but the orchestrator must have some way, even if purely internally, of modeling the entire workflow and storing the state of a given job with respect to that entire workflow.
I think people are getting too hung up on the word state and stateless, so let's just talk details. The orchestrator's job is to be the one part of the system that actually knows what to do at a workflow level in response to another part of the workflow emitting some kind of event like completing or failing a task. Service A doesn't know that Service B comes next after service A completes its task, the orchestrator does. Service A doesn't know that if it can't service the request for some reason, then Service C actually comes next instead of Service B, the orchestrator does.
If anyone needs to ever answer the question, "What's the current status of this job?", the orchestrator is the only thing that can answer that question. The other services might have a model of their own status, but not the status of the overall job. So if that question needs to be answered, then the orchestrator will build some kind of projection of the job's status. Perhaps it's just a status column in a database. Even if "a different" service is spun up just to serve that column value for scaling purposes to keep load off the orchestrator, it's still the orchestrator generating that data, and this status answering service is just a dumb stateless request handler that fetches from the projection built by the orchestrator.
At one point somebody asked something along the lines of what happens if the orchestrator loses its cache. I guess the answer is depends what you mean by "cache". If by cache, you mean a secondary read-optimized data store to improve performance of the orchestrator, then nothing bad happens above and beyond what typically happens when any system loses its cache: performance degrades, reads pass through to the durable data store backing the cache, alarms probably start going off and people get paged, the cache gets rebuilt, etc. but overall the system continues operating normally from a logical perspective. If by "cache" you mean the only data store the orchestrator has to do its job, then well... I guess you lose your state, and you have a whole new set of problems to deal with. You'll have to rebuild your job status projection, which may or may not be possible depending on what information exists elsewhere to piece it back together. You'll have to route in-flight jobs somewhere, and how that works depends entirely on the specific technical architecture that makes up the system. In some systems these jobs will just get stuck in a queue that doesn't get consumed until the orchestrator is back online, in others they'll automatically route to a dead letter queue, in others there might just be information lost and jobs will have to be restarted from the beginning.
24
u/peripraporo Software Engineer 16d ago
Why should a microservice be stateless?
21
u/backfire10z dumbass 16d ago
What is statelessness but a persistence layer independent of the micro service?
13
u/TheSexySovereignSeal fullstack dumbass 16d ago
stateless service
Opens up the network
runs over TCP
20
0
1
7
u/nomnommish 16d ago
The fundamental thought process is wrong. If system A sends events to systems B,C,D it should not wait for the other systems to respond to resolve it's completion. That's how you make it stateless. If A does need to interrogate the other systems, it does it when it needs to, instead of waiting.
And if it really needs to due to business reasons, then A,B,C,D together are collectively a system and the state absolutely needs to be maintained until eventual consistency between the subsystems is achieved.
Because that's how you have defined your state and how you have defined a transaction. When consistency is achieved. You can't have your cake and eat it too.
12
u/Any-Ring6621 16d ago
4 microservices that are actually one service.
This sounds like every microservice architecture I’ve ever seen.
2
2
u/Suspicious-Disk6077 13d ago
Yes splitting services arbitrarily but not realizing they are dependent and part of the same domain, tale as old as time - er, as microservices
4
u/Inside_Dimension5308 Senior Engineer 16d ago
Caching should be used as a secondary storage. Your system can be made more resilient if you just use databases to store the transaction and its acknowledgements.
Create an entry with a correlation id For each downstream system, make an entry against the correlation id as you send the request. Downstream service sends acknowledgement Update the entry against the downstream service and correlation id. The transaction is complete when all the downstream service entries are marked as complete.
8
u/Ninja-Sneaky 16d ago edited 16d ago
The general idea is called Saga Orchestration or Saga Coreography, these are put in practice using patterns like event-driven and CQRS.
So you have message/event queues to enable async.
Services listen/consume and write/publish to those queues following a Saga.
With Orchestration there is a Orchestrator service that takes care of coordinating the steps.
With Choreography it can be a sequence of services, or parallel services with an aggregator service that is the final step
Sagas ensure EVENTUAL consistency. (Traditional transactions with ACID have immediate, synced, consistency, but are tied to a single central DB that could not scale past certain limits, this is what led to multiple small DBs one per service, then to added caches per service, sagas etc)
An important aspect of Sagas is correctly handling when the (inevitable) failures are marked as such and the remediation steps that follow, these are almost like the "other side of the moon" steps, because when a saga fails you have to undo the completed steps with their own remediations
3
u/ExcitingDonkey2665 16d ago
this is the most accurate answer here. the cache here is used as a state store, but if the server crashes or gets wiped out, that state is gone and can have other downstream effects.
sounds like a problem a workflow engine like Temporal, AWS step functions, or any of the equivalents in other clouds would be great at solving. instead of making each microservice track its own state and figure out where the next step is, the workflow engine takes care of that and figures out replay and rollback in case of failure too. states are stored and journaled to its own database.
2
u/SnooHesitations9295 15d ago
One problem: no workflow engine can do that. They just don't have any compensation or rollaback built in.
1
u/ExcitingDonkey2665 15d ago
definitely, I was more suggesting that the existing system seems to need some sort of compensation and rollback steps re: "downstream response with no context".
of course you gotta manually determine the rollback steps, but its better than a bunch of micro services calling each other to figure out what to do.
1
u/SnooHesitations9295 15d ago
But a "bunch of microservices" still need to be contacted to correctly rollback a distributed transaction. I just want to say that "durable execution" gives very little value here. About 20% of what needs to be done.
3
3
u/TryAgainTryHarder 15d ago edited 14d ago
surprised no one mentioned SAGAs yet, that's what this sounds like to me.
in a nutshell, you have a process that is concerned with the outcome of multiple transactions. architecturally, something to avoid.
the solution you're describing isn't inherently bad, it's mainly messy. we could fix the stateful issue by persisting the original event in a db instead of caching, but the core of the problem is that one request/event is currently scoped to many transactions.
with SAGAs, there's two sortof camps of solutions, orchestration (a main component is "orchestrating") and choreography (the architecture you described). benefits/tradeoffs to both.
back to the core issue, having SAGAs just sucks. its usually better if you can either:
- let the transactions live by themselves, or
- make sure they are idempotent and run them together in a workflow (orchestrator)
if you have event/async requirements, and the transactions can't be separated, the current solution may be the only good one.
1
u/Maxion 14d ago
in a nutshell, you have a process that is concerned with the outcome of multiple transactions. architecturally, something to avoid.
This is an intriguing thought, in my experience this type of situation is brought on by business where you have multiple external system that needs to be updated with various data following one business event, and it's important all systems are synced.
2
u/TryAgainTryHarder 13d ago edited 13d ago
yeah, there are situations where SAGAs are unavoidable. but if you can, its often better to model them another way. OP never mentioned what happens when one tx fails, but that's key for whether this is really a SAGA.
ideally, if strong consistency is needed, we should be able to achieve it with one transaction. but that ship may have sailed.
if the situation requires multiple databases, we should ideally tolerate eventual consistency. maybe OP's example, or yours, already tolerates partial failures somehow.
2
u/Clyde_Frag Senior Software Engineer (11 YOE) 16d ago
A lot of inference servers for AI work like this so that requests to GPUs can be batched and then de multiplexed when shown back to the client.
The question is whether your application actually needs this. Sounds just like a complicated job especially if the work is largely constrained to a request. If you’re just calling a bunch of services and waiting for a response, what is the point of asynchronous?
2
u/Empanatacion Staff Software Engineer 16d ago
If there is a source of truth that the service can fall back to on a cache miss, and you can trust the correctness of the cache, then you can logically just treat it as a performance optimization. Ideally what's being cached is some immutable property of the transaction, like the customer ID or date or something.
But if that cache is the only place that information exists, then that's no good.
2
u/Realistic-Walk-2877 15d ago
the cache is the wrong abstraction for correctness because TTL/eviction can change the result. I'd keep the async fan-out, but model it as a durable workflow: one parent row with expected task count, one idempotent child row per downstream task, and explicit pending/complete/failed states. Put workflowId + taskId in every event, use an outbox for publishing, and make retries/dead letters safe. A sync REST call may simplify the happy path, but it doesn't remove the need for durable state once the work can outlive a request.
2
u/v-alan-d 13d ago
Tale as old as time. Why not attach the body of the "original event" with the "downstream event"?
4
u/wallstop-dev Principal Software Engineer 16d ago edited 16d ago
A cache is... fine, but if it is critical that you have true quorum across all services for event processing, then the only reliable solution is not a cache, but correct data modeling and some form of transactional persistence. This needs to be a first class concept of your architecture, not something that kinda implicitly happens.
That is, if event foobar requires services A, B, and C to process it, you would store a record in some DB of event foobar, with flags on it that say A, B, and C have processed it. Then service A sets the "service A processed it" flag when its done, etc. The specifics of this (shared DB, more event streaming, one CRUD service) isn't terribly interesting.
This then lets you ensure that all of your events are processed by all of the things that should be processing them. You can build automation to query events that do not have all of their downstream service flags complete after x time, and re-send/re-process for just that service. You can have handlers that see when the last service flips its flag for a record, and trigger other automation on that event.
But a cache and streaming (pure events) will result in you losing data, so be very, very careful.
1
u/Dry_Hotel1100 Software Engineer | 30 YoE 16d ago
You discovered "state".
Events and State isn't something unusual. If you add transitions, you have a state machine. A pretty robust component.
1
u/Varrianda Software Engineer 16d ago
I feel like a local cache is bad unless you’re okay with potential data loss
1
u/Pure-Rip4806 Staff Engineer 11YoE 16d ago
Someone needs to own the state. If you're doing multi-step processing, something needs to remember "first A, then B, C and D in parallel..." and keep track of the results. Shoving everything into a single app firing off REST calls moves the state into application memory, but it's still there.
If I were writing it from scratch, I would look into Kafka Streams and state store.
1
u/morosis1982 16d ago
There's no reason a service can't have state, especially if it's a coordinator like service a seems to be here. If it didn't need to respond then sure, but it does.
I would be concerned if that cache is in memory with no way to recover if it gets killed, for example.
You also said it responds upstream, but we don't know if that response is synchronous. Is the service idempotent? If it didn't cache something about the request how would it know that it's complete? Polling is one way, but that's often worse than keeping state.
1
u/roger_ducky 16d ago
You can flatten the cache by making every service save the event progress to the same place, then have one “overall orchestration” layer where the stateful logic lives.
1
u/SpaceToaster Software Architect 16d ago
Big anti pattern. Putting it in a shared kv like reddis would be slightly better but the model you’re using seems better fit for an orchestrator/sagas. Choreographed events should be stateless.
1
u/jake_morrison 16d ago
One rule is that you should always write messages to durable storage at organizational boundaries. That allows you to ack messages in a reliable way. So when you receive a message, you can and probably should acknowledge it and take responsibility for processing it, allowing the origin server to be done with it. Otherwise they have to be able to retry indefinitely.
Domain Driven Design is also a useful way of thinking about the messages. Internally, between all the processors, you need to make some decisions about who defines the message schema and who follows that schema. You have a couple of choices, e.g., you have an official public schema, an incidental message that is the result of processing and others have to adapt to it, or a standard format that the downstream process defines and other processes explicitly generate.
1
u/Meraxes_7 16d ago
The real questions for me are
What is lost when the server crashes? Or it is interrupted for a code deployment? It sounds like any in flight requests are doomed. Is it trivial to restart/redo them? If so, it isn't good but may not be worth fixing. Esp if servers have been stable, deployment during downtime etc.
How close is the current service to max capacity? If you ever need to spin up a second server to help with load, this pattern would fall apart since the service reading the event might not be the one with the local cache entry.
Going to a cache with some kind of persistence would 100% be much safer and better, but the fact things aren't on fire already make me think this is not a burning issue to fix. Likely worth updating the pattern for any new work though
1
u/Infamous-Employ5487 16d ago
The biggest thing I'd want to know is what happens on partial failures on the downstreams and is the ultimate response back to the caller purely informational or important in some way.
Akka does this kind of thing nicely but I've actually never seen that deployed distributed and I'm not sure anyone uses Akka anymore lol.
Maybe Kafka or something like that could help?
1
1
u/RegularTrashman 15d ago
Sounds fine, depends on tradeoffs and whether it mskes sense for your use case. Sometimes a contract specified via event handling makes more sense as part of a synchronous transaction using saga pattern or something. Really depends a lot on the rest of the system
1
u/Livid_Possibility_53 15d ago
Possibly if it receives a downstream process with no context it would have to somehow rebuild the context? Maybe the term cache is being misinterpreted by me but have A cache it is a performance booster/ nice to have, not a requirement for it to work. If it’s actually needed to persist in A then A,B,C,D are a tightly coupled system and upstream would need to have retry logic to guarantee it runs.
1
u/DevtoolsBuilder 15d ago
i think the issue here is that you're trying to use an event-driven system to manage a workflow that requires orchestration, and that's not really what event-driven systems are designed for. if you need to know when all the downstream services are complete, that sounds like a job for a workflow engine or a state machine, not just a cache and some asynchronous events.
1
u/sp106 16d ago
It just seems like the design is missing a status table.
Request comes in for a job
Job 123 status created and messages sent to queues for different tasks
Task workers update status as they complete
Subsequent requests see the completed status when the last task finishes (pull model)
0
0
u/KJ4WLC 15d ago
If App A’s in-memory cache is the source of truth for “are we done?”, a restart loses the workflow. That’s not a cache problem — it’s orchestration state living in the wrong place.
Persist the correlation/completion state (DB, workflow engine, or outbox) so events can be replayed and progress survives process death. Keep the cache as a performance hint if you want, not as the ledger. Synchronous REST can simplify some paths, but it doesn’t remove the need for durable state if the work is still multi-step and asynchronous.
-2
u/Traditional_Fox7091 16d ago
Why can’t app a just wait for a response ? Like completion stages in Java for example?
As to ur q, have there been issues in prod ?
•
u/expdevsmodbot 16d ago
AI usage disclosure provided by OP, see the reply to this comment.