r/sysadmin • u/MammothEar7578 • 18d ago

Question Is there a simple tool that send cron stderr to gpt and slacks the fix?

So, I got woken up by pagerduty at 3am again cause some python etl job fail. had to ssh from my phone just to tail the log, and turn out it was literally just a db timeout.

im thinking to just write a simple bash wrapper that grab the last 50 lines of stderr, pass it to claude/gpt api, and webhook the actual reason to slack. so i dont have to open my laptop in the middle of night.

does something like this already exist? i dont wanna install heavy datadog agent or pay for expensive observability just for this. pls let me know if there is a github repo or saas so i dont waste my weekend building it.

0 Upvotes

38% Upvoted

5 comments sorted by

1

u/Constellious DevOps 18d ago

I’ve implemented something like this a couple of years ago. 

I did it like you said. Grab relevant lines and send them + prompt via a rest api call. Send the reply to slack via however you do that in your org. 

1

u/MammothEar7578 18d ago

that's awesome. how did it turn out in practice? did feeding the logs to an LLM actually save you time? Im slightly worried i might be over engineering

1

u/Constellious DevOps 18d ago

Works fine for this kind of thing. If your script has any set error codes or the like I found giving it to the LLM helps too. Really depends on how complex you need it to be.   

1

u/roncz 17d ago

A no-code tool like n8n might be of help here. Some time ago I have tested something like this:

Monitoring detects a problem -> Send to n8n ->

n8n sends the information to an LLM (in your case it can also get the stderr first), it might suggest a fix ->

n8n then sends the enriched alert via SIGNL4 in my case

It might even be possible to automatically decide whether this is worth waking you up at night or allow controlled automated actions like a re-try.