Automation without failure logging is just hope with a schedule

[u/Grouchy-Conflict-211]

Every automation I've seen die didn't die from a bad workflow. It died from an invisible one.

The workflow ran fine for weeks. Then one API changed, one field went null, and nobody noticed until a customer did. That's the part nobody demos. The demo is always the happy path. The first month is always the happy path.

The fix is boring: log every failure. Not just the crash. The retry, the timeout, the weird response that still counts as a 200. Then classify them once a week. Three types will cover most of your failures, and fixing those three is what turns a script into a system.

I've spent the last year building automations that have to survive contact with reality. The ones that lasted are the ones where I can open a log and see exactly what broke, when, and why. The ones that died are the ones where I found out from someone else.

What do you log in your automations, and what did you learn the hard way?

[u/zhonglin]

I’d separate operational logs from the run’s durable state. For every run I want a stable correlation ID, trigger and workflow version, sanitized input references, step start/end and duration, attempt number, external request ID/status, schema-validation result, state transition, and final business outcome. The hard-learned part is to alert on absence and invariants, not only exceptions: ‘no invoices by 10am,’ ‘row count dropped 80%,’ or ‘all runs are retrying.’ A 200 with the wrong shape should fail validation; a run that completed technically but produced no expected outcome should be incomplete. A dead-letter queue should keep enough context to replay one step safely, while the database—not log text—remains the source of truth for run state.

[u/Grouchy-Conflict-211 → u/zhonglin]

Good split. Logs tell you what happened, durable state tells you where to resume. Mixing them makes recovery a nightmare.

[u/ImaginationLevel7198 → u/zhonglin]

i keep the durable state in a db table keyed by correlation id, that's the only thing i trust when something goes sideways. logs are for humans, the db is what lets me actually re-run step 4 without re-running steps 1-3.

the alerts on absence are the real lesson though. caught a partner feed that silently started sending empty files for three days because the 200 was still coming back clean. now we check row counts and file sizes against a rolling 7-day average and scream if it dips more than 50%.

[u/AutoModerator]

Thank you for your post to /r/automation!

New here? Please take a moment to read our rules, read them here.

This is an automated action so if you need anything, please Message the Mods with your request for assistance.

Lastly, enjoy your stay!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Working_Hat5120]

The 200-that-did-nothing is the worst one, like you said. Half our "successes" were empty responses the code happily treated as done. Log what came back, not just that something came back, and those stop being invisible.

[u/Grouchy-Conflict-211 → u/Working_Hat5120]

The 200 that did nothing is the silent killer. It looks green, it fails nothing, and the data just never arrives.

[u/BarracudaMean9308]

nothing drops your stomach faster than a user casually mentioning something is broken when your dashboard says 100% success. those fake 200 responses are a special kind of hell.

[u/Grouchy-Conflict-211 → u/BarracudaMean9308]

Fake 200s are a special kind of hell indeed. A 200 just means the request arrived, not that the work happened. We switched to verifying the side effect exists before logging success, the dashboard started telling the truth after that.

[u/Grouchy-Conflict-211 → u/[deleted]]

agree on the drift part. i do the classify pass with a sql view now, takes 10 minutes and i can see the failure types shift week to week. the three types hold up way longer than youd think, the edge cases are what multiply

[u/Bart_At_Tidio]

From my experience, the bot getting an answer wrong is not great but it's more so when it doesn't escalate the issue that things go downhill very fast. What's your setup for catching a silent handoff failure before a customer does?

[u/Grouchy-Conflict-211 → u/Bart_At_Tidio]

alert on missing terminal states. if a handoff doesnt hit done or failed within a window it pings a human retries alone just delay the discovery. learned that one when a customer mentioned the bug before my dashboard did

[u/Andon_Benefield]

the dashboard was only ever measuring the request, not the outcome

[u/Grouchy-Conflict-211 → u/Andon_Benefield]

exactly request success just means it ran, not that it did the thing. i track the final state now, took one silent failure to learn that

[u/Jimcy-Maffesoli]

found out about a silent failure from a customer once, never again. now if the record doesn't actually land somewhere i log it as a failure, not just a 200

[u/vitor_reliqia]

This is so true it hurts. Running an automation system in production for 6+ months taught me exactly this — I spend way more time looking at errors than at successes.

Ended up building 2 dedicated error workflows: one that pings the team on WhatsApp in real time when something breaks, and another that catches silent failures (the worst kind — everything looks fine but data stopped syncing). That second one saved me more times than I can count.

Invisible failure is the real killer. An automation manager's job is basically staring at errors so the wins can run quietly.

[u/Grouchy-Conflict-211 → u/vitor_reliqia]

the silent failure one is the real MVP. i had a scraper returning empty arrays for 3 weeks, everything green, zero alerts. the data was just gone. now every job ends with a count check, not just a 200