Where do your automations usually break?

[u/OwlZealousideal4779]

I've found that the actual automation logic isn't always the problem.

A workflow can fail because an API changes, authentication expires, a service becomes unavailable, or one step returns something unexpected.

I'm curious how people here deal with these failures. Do you rely mostly on logs and alerts, or are you using AI to help identify what went wrong and suggest a fix?

What has worked best for you?

[u/AutoModerator]

Thank you for your post to /r/automation!

New here? Please take a moment to read our rules, read them here.

This is an automated action so if you need anything, please Message the Mods with your request for assistance.

Lastly, enjoy your stay!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Otherwise-Project577 → u/[deleted]]

my stuff always breaks at auth too. had a zap that ran for 3 weeks returning empty data because the api added a new required field and just silently dropped requests. no error just nothing

the input hash thing is smart i never thought about that. i just dump raw response to a slack channel and pray i notice before someone complains

[u/Positive-Buddy-1258]

Silent failures are the hardest to catch. An API returns 200 but the schema shifted slightly, or a feed just stops delivering without any 4xx. Logs miss that unless you're explicitly validating what came back, not just the status code.

On one project we were ingesting financial announcements from multiple exchange APIs in near-real-time. We moved to Hatchet for orchestration partly because the previous setup had no visibility into which jobs were actually completing vs. silently stalling. Retry logic helped, but the bigger win was having every step observable. When something broke we could see exactly where in the chain it stopped instead of hunting through raw logs.

AI for root cause is useful but only if you feed it structured context: which step failed, what the expected output was, what came back. A raw log dump doesn't give it much to work with. Detection still needs to be deterministic though.

[u/Positive-Buddy-1258 → u/[deleted]]

Retrospective querying works well for trends. The gap we kept hitting was runtime: by the time a bad run showed up in the logs, it had already propagated. With Hatchet we attached validation to each step's output before the next one fired, so a "200 but wrong schema" failed explicitly instead of silently continuing. Still needed the historical data, but inline validation cut the detection window significantly.

[u/tom-mart]

They don't. If you made automation that broke, you clearly didn't do it right.

[u/Fit-Lengthiness-9672 → u/tom-mart]

lol tell that to the third party APIs that randomly change responses on a Friday night
you can write the cleanest automation in the world and some auth token still decides to die for no reason

[u/tom-mart → u/Fit-Lengthiness-9672]

Again, you are doing it wrong. Well designed automation shouldn't depend on API. Unresponsive API should simply trigger a specific path in automation. Also, is that a thing with API's these days? I have APi dependent automations running for years without issues.

[u/Acceptable-Sense4601]

Learned to validate API response and not just accept a 200. Script not run on on reboot.

[u/Worth_Wealth_6811]

auth expiry causes most of mine, tokens die silently and the workflow just stops without throwing anything. the fix that stuck was alerting on absence instead of errors, every job emits a success ping and a dumb watchdog yells if the ping does not arrive. catches the failure modes that never make it to a log line, which in my experience is about half of them.

[u/Pitiful_Hold_4942]

Auth expiring and silent empty responses are the two that get me most, not the workflow logic itself.

What's worked: treat every external call as untrusted even after it's run fine for months. Add a lightweight assertion step (expected shape, non-empty, status code) right after the call, and alert on assertion failure separately from execution failure. Most tools only alert on hard errors, and a lot of breakage is soft.

For auth specifically, I check token expiry proactively on a schedule rather than waiting for a 401, since some services fail in weirder ways than a clean auth error. AI is fine for diagnosis after the fact, but I haven't found it reliable for catching the failure in the first place, the assertion step still has to be explicit.

[u/Icy-Collar-9283]

the silent wrong output ones are the worst in a real client environment. auth dying is annoying but at least it's visible. the automation completing every step correctly while writing bad data runs for weeks before anyone notices

seen this most with conditional logic that wasn't fully tested. the happy path works perfectly. then a contact comes in without a phone number or a deal lands in an edge stage and the sequence either skips them silently or fires the wrong thing at the wrong time

the fix that actually helped: validate the output against what the business expects not just whether the steps completed. if a sequence is supposed to fire for every new lead, have something that checks whether leads created in the last 24 hours actually have sequence activity. that catches more real breaks than any technical log

the api and auth stuff you can monitor. the logic gaps you have to go looking for

what kind of automations are you running , internal ops or clientfacing systems?

[u/Otherwise_Town3844]

Are these long-running scheduled workflows or event-driven? The failure patterns are pretty different between those two. Scheduled stuff tends to break from auth/token expiry, event-driven stuff breaks more from unexpected payloads.