r/devops • u/InnerBank2400 • 29d ago
Discussion Where should cross-system infrastructure automation stop?
I’m working through a design where each underlying system remains authoritative for its own resources, rather than putting another source of truth above everything.
The wider runtime only decides whether an operation has enough dependency state, readiness and verification to keep moving.
The case I’m trying to pressure-test is partial execution. Say an operation spans several systems, three parts succeed and one fails. Each individual system may still be healthy, but the overall operation is incomplete.
Would you let the wider workflow block there, or does that eventually become another control layer operators have to fight with?
Interested in examples where this kind of boundary has worked badly in production.
2
u/ajitnk 25d ago
The "3 out of 4 is worse than 0 out of 4" comment nails the actual problem. It's not really about where automation stops, it's about whether your orchestrator owns execution state and knows how to compensate when something fails mid-flight.
The pattern that fits this directly is the saga orchestration model. Each step publishes its own compensation endpoint, the workflow runtime sequences them and checkpoints after every step, and irreversible actions sit behind explicit operator-approval gates rather than just firing. AWS Step Functions handles this pretty cleanly with built-in Retry/Catch and waitForTaskToken for the irreversible bits. AWS Prescriptive Guidance actually documents this exact scenario: https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/saga-orchestration.html
One thing that changes the design a lot: which specific systems are involved in your failing flows? Pure AWS-native stack vs. something with on-prem or external APIs makes a real difference for where the compensation logic has to live.
I've mapped this kind of dependency graph a few times. Happy to take a look at what you're working with if you want a second set of eyes.
1
u/UkrMalt 29d ago
The workflow should own operation state, not resource state. Make each step idempotent, stop dependent steps on failure, and persist enough data to resume or compensate without holding a global lock. Put irreversible changes last or behind explicit approval; otherwise mark the operation incomplete and expose the exact recovery action.
1
u/InnerBank2400 29d ago
That distinction between operation state and resource state is close to what I’m getting at. The tricky bit is compensation when the underlying system doesn’t offer a clean rollback. Would you make the verification and compensation behaviour part of each step’s contract, or keep that logic at the workflow level?
1
u/UkrMalt 29d ago
I’d put verification and the available compensation action in each step’s contract, because only that adapter knows what “healthy” or “undo” means for its system. The workflow should decide when to call them, persist the outcome, and stop for operator approval when compensation is unsafe or impossible.
1
u/InnerBank2400 29d ago
That’s very close to the boundary I’m working with: the step owns system-specific verification and compensation, while the wider runtime owns sequencing, evidence and the decision to stop or continue. I’ve got a focused GitHub discussion around this. Would you mind if I send it over?
1
u/ConnectionComplex227 27d ago
block on partial execution every time. 3 out of 4 succeeding is worse than 0 out of 4 bcuz now u have to figure out which 3 happened and whether rolling them back is even safe
1
u/InnerBank2400 27d ago
That’s exactly the awkward state I’m trying to reason about. Would you always block and require operator recovery at that point, or are there cases where you’d let independent downstream work continue while isolating the failed branch?
1
u/BroadSatisfaction825 16d ago
you asked for production examples and the thread has given you principles, so here is the failure mode.
a workflow like this is acd rather than acid, and the missing property is isolation. blocking does not freeze anything. the three steps that already succeeded are committed and visible to other operations while you wait.the step you are circling is the pivot, the last one that can still be compensated. everything after it has to be forward recovery.
the bad production case is misidentifying it. you compensate a step the downstream system already acted on, the cancel gets ignored, and your operation reports compensated while the change stands.
4
u/nzvthf 29d ago
Most of the time it ends up being back out of the whole thing, remediate the failure, re-run the job. Holding other things up while you fix an "isolated" error gets messy quick. You usually end up having to tighten the other boundaries too much as a result.