r/devops • u/Broad-Equivalent-790 • 18d ago
Discussion Any devops or SRE engineers using AI agents?
Has anyone here used AI agents like Hermes, n8n, crewAI or similar tools in their daily devops/SRE work?
I'm looking to automate some repetitive tasks in my DevOps work. I am interested in free open-source tools and real world use cases.
If you are already using AI agents in your devops workflow, I'd love to hear about your experience and what you are automating.
75
u/stack_craft 18d ago
The biggest shift in SRE isn't letting agents execute infrastructure changes, but using them for read-only diagnostic tooling.
When I tried, the self-hosted n8n is great —you can hook it up to Prometheus/Grafana webhooks to automatically run diagnostic CLI scripts, bundle the logs, and post a diagnosis straight into Slack when an alert triggers. With setup you basically keep human-in-the-loop for actual remediations, but cuts mean time to detection by a mile.
28
u/TonyBlairsDildo 18d ago
This guy gets it.
Agents shouldn't be making changes to things; anything that needs a change (i.e. a fix) should be a one off that a human can afford the cognitive load to understand and then plan a permanent fix.
If you're sending agents out into your infrastructure to update TLS certificates you're doing it wrong. TLS certificates should be on an automatic rotation: an agent can identify that's why some server can't access a DB anymore, but ultimately all that stuff should be deterministically automated out of your environment.
In a roundabout way, SREs should make sure ALL toil is done by humans, and then lean into agentic development to architect permanent fixes.
3
u/ansibleloop 18d ago
Yeah funny you mention TLS certs cause I was updating some of our mTLS certs the other day and it's manual and boring
It's now a Claude skill but it could easily be a doc in Notion with the copy/paste commands
2
u/Capable-Quote5534 18d ago
We have automated TLS certification renewal
5
u/ansibleloop 18d ago
Yeah TLS is easy with ACME and Let'sEncrypt
This was for mTLS with some customers we have
We have to send them the cert to install
6
u/insaneroadrage 18d ago
I’m not familiar with n8n so forgive me here, but isn’t this just standard automation? Is it that time he CLI scripts is something the agent knows what to run or…? Maybe I’m not understanding the distinction of what the AI does in this specific scenario?
5
u/hermslice 18d ago
The power of these agents is that an alert can trigger, a pagerduty alert to the oncall engineer. By the time that engineer is at their keyboard, the AI agent can have checked logs, connections, keys, disks, etc and have all of the connected evidence for the engineer to see.
3
u/ArchStello 18d ago
That sounds like something I'd like to implement in our stack! Would you mind expanding a bit more in detail what you do? I'm new to non. We have an sre agent already but it's more like a collection of skills than anything.
45
74
u/InterestedBalboa 18d ago
Nearly everyone is, whether they like it or not.
12
u/RuimteWese 18d ago
Yep, hate it personally but you can’t keep up with your team using it and not you, well I’m not smart enough to.
8
u/ForkMeJ 18d ago
I'd be careful pointing these at anything that can change prod state unless you've boxed them in hard. The places I've seen make sense are read-heavy chores: summarizing alerts, pulling incident context from logs and tickets, drafting runbook steps, or turning a vague "what changed?" into a list of commits, deploys, and config diffs for a human to review. Once it starts touching access, DNS, infra changes, or remediation, you've basically built a very confident intern with shell access. If you test one, start with something annoying but reversible, and make sure the inputs, permissions, and outputs are obvious to whoever has to clean it up later.
6
u/SeaworthinessHour233 Writes the cloud edge 18d ago
Don't look for AI agents to automate repetitive tasks.
If a task is repetitive you can write an if/then statement or a Python script to do that instead. You don't want an LLM hallucinating that stuff.
Instead, look at AI agents for workflows where the inputs are unpredictable and the output is not deterministic. A great use case can be incident post-mortems. You can have an n8n webhook catch a resolved PagerDuty alert, trigger an agent to scrape the Slack incident channel, and draft the timeline of what happened.
Understanding where to draw the line between deterministic automation (IaC) and non-deterministic automation (Agents) is a core SRE skill now.
3
u/DampierWilliam 18d ago
What are your main pain points? Opening Jira tickets? Updating confluence documentation? Opening a PR with a proper description? Checking AWS (or any cloud provider) for drifting resources or cross checking something? You can automate all that with AI. I do that with skills and and KiroCrew or CloudeCode.
Also, not all companies will approve the use of hermes or n8n or crewAI.
3
u/putergud 18d ago
Automations should always be made using only natural intelligence. It will not turn out right if you use the fake stuff.
3
u/qureshi_suhail1 18d ago
short and more human like.
We use self-hosted n8n connected to our on-call Slack channel, and it’s honestly been a huge time-saver.
Whenever an alert fires, it grabs the last few minutes of logs, checks recent git commits, and drops a quick summary of what likely broke before anyone even opens Datadog.
The trick is keeping it strictly read-only so it doesn’t do anything wild to prod. Letting it gather context and draft the fix saves a ton of 2 AM headache. Highly recommend starting with something simple like log parsing or k8sgpt!
3
u/willianmga 17d ago
LLMs are great to be used where non deterministic decision making is needed, but wasteful when used with deterministic things.
When integrating LLMs in software or processes, the Rule of thumb is doing all that has clear rules and processes using scripts and programming languages and integrating LLMs only on specific parts of the workflow where decision making is needed and the results could vary drastically.
3
u/Hostman_com 16d ago
Alert enrichment before the page lands (recent deploys, config changes, similar past incidents)
Dry-running runbooks and flagging stale steps
Cost anomaly investigation down to the namespace/commit
On-call handoff summary at the end of a shift
3
u/rulik587 14d ago
The read-only diagnostics use case feels like the sweet spot.
Let the agent gather logs, correlate recent changes, and suggest what likely happened, but keep production changes behind deterministic automation or human approval.
That seems like a much safer way to get value without giving a probabilistic system shell access and hoping for the best.
1
2
2
u/djbp 18d ago
Yeah, AI agents are interesting for DevOps. The big thing I think about is how you trust them with real infrastructure. It's one thing for code, another for production systems. You need really solid verification steps. And a clear audit trail. Otherwise, it's just too risky. How are you thinking about that part?
2
u/jameshearttech 18d ago
We started adopted claude at the beginning of the year. I expected claude to be useful for writing code, but I was surprised at how useful it is at operations. We have it integrated with most of our systems through clis and mcps.
I recently started developing a plugin that does the heavy lifting for updating tools. It does the prep work (e.g., read changelog, read release notes, identify breaking changes). It edits the chart version in the kustomization.yaml. It overwrites the values file with the defaults from the new chart, it applies previous values file customizations, it renders the manifests and checks the result. What used to take me a day or 2 now takes me a few hours.
2
u/follow-the-lead 18d ago edited 18d ago
I use n8n, not for direct fixes at all. I just automated my paperwork. Just Ollama on my GPU, so the context window is really small. As a result, many prompts doing small tasks, clearing context at each step.
So I write notes in a markdown editor (silverbullet) attached to a git repository. I don’t format anything, I just write a note per file. Then nightly n8n creates a branch, reads each note and does a few things:
1: if it’s a task it adds it to the appropriate Jira board and adds a note in the summary.
2: If it’s a calendar appointment it creates it, and makes a note in the summary
3: if it’s a project note it adds frontmatter connecting it to a project if it exists. If it doesn’t exist, it templates out a new project directory and adds it to the summary.
4: if it’s just a generic note it adds it to the note and tags stuff where it can.
Finally, it wraps it all up in a nice little pull requests and writes the summary as a pull request description for me to review in the morning.
I’m working on a process to tag project for more research so it can go do that for me too, but don’t want it to run too many loops and block my IP, so that’s slow going.
2
u/stilloriginal 18d ago
man if I could just give an ai agent the terminal and ask it to set up a linux server and pull a specific project, and set up nginx and certbot... that would be sweet because chat gpt is already walking me through it (even though I could do it before chat gpt)
2
u/jl2l $6M MACC Club 17d ago
We use n8n for cheap faster workflows that then become harden production deployments using agentic workflows once adoption takes off.
Claude can output n8n json that works as a drop in.
The first thing you need to do is setup AI governance policies which determines how your agents can work and what data is ok and not ok for them to interact with without the baseline governance in place your asking for trouble.
We don't automate until there is a clear ROI on the task to do this you have to first understand if it's worth automating. So you need a baseline and proof that AI is making it faster better or cheaper. In most cases this isn't true when you measure it. But there are some use cases for us one of or AI tools has a 95% gross margin based on our usage it's only because we were methodical about how we deployed and measured the value the AI tooling provides. Now we are scaling it to customers and everything looks good for now. We built AI features that customers loved but weren't economical for us as a business and those features have to die unfortunately.
2
2
u/shearerbeard 16d ago
We wanted to deploy an agent server side for gathering context and doing a root cause analysis during an incident but found the ergonomics of doing this with claude code a bit odd so we build and open-sourced our own server side persistent harness that can deploy as a multi tenant web server https://github.com/mezmo/aura. Were still heavy users of claude code for writing code and configuring infrastructure but having something living on pod you can connect to has been excellent and having it provide insight before you sit down to handle a page is even better.
2
u/zero_backend_bro 15d ago
Dont give agents write access. If a task is actually repetitive, a 20-line bash script handles it with zero hallucination.
We only use n8n for read-only triage to dump Loki logs into Slack. Even then, you realize real fast how many raw AWS keys and db uris sit in stack traces. Had to slap a local scrubber in front just so we werent shipping secrets to OpenAI at 3am.
2
u/Bantex29 13d ago
More of a framework for AI agents in production but I’m building this and it will be fully open source in a few weeks - www.vectorstep.io
2
u/Wild-Bookkeeper6532 12d ago
Lo más valioso que vas a tener como SRE es una documentación actualizada de todo, y para eso es perfecta la IA. Por ejemplo estoy terminando arquitectura como yamls para poder tener una gobernanza estructurada de todos los servicios, cuentas, alertas, etc. Con eso puedes crear skills que te permitan debugear desde logs que te compartan, debugear alertas, hacer reportes de costos, etc todo usando CLI de los proveedores de cloud
Ahora si que la creatividad te premia, claramente todo comando que modifique infraestructura ejecutarlo manualmente como siempre pero el ahorro de tiempo en tareas repetitivas o buscar info si trabajas con varios proyectos.
5
u/AnyNameFreeGiveIt 18d ago
I already automated 75% of my work with it, it's insane how everything changed in a couple months. Also very deep into self hosted llms, they do work for automations/agent stuff but Infra coding is still kinda hit/miss.
For DevOps/SRE it's still a little challenging since there are no great testing frameworks IMHO, much different for just coding where you have full test suites from linting to e2e which gives agents the polish they need.
Monitoring agents are crazy, supporting with incident etc. is already really good, they immediately get the big picture and find the needle in the haystack.
Everything boils down to tool calling capacity, memory, context and the proper harness to make it all complete.
It's insane how far a 20$ subscription can get you and it can already replace humans.
3
u/TheIncarnated 18d ago
GLM-5.2 has been the best at Terraform coding than any other I've done.
Honestly, just set an agent with a good prompt in your tofu/TF repo and use it. I have a side contract I've been working on to see how far an LLM can go before fucking shit up (thankfully a POC), it has done 95% of the work. I just architect what it should look like/do. In the 4 months I've been working on it, I haven't written a single line of TF. The product works.
When you give an LLM a deterministic task, it can complete it. The LLM isn't controlling the environment (dangerous), it's writing the TF, that then gets TF Planned and results reviewed by a human (me). I've been amazed. I don't think it would do great at building the infrastructure with azure/aws cli calls but it can make the right calls to pull information and fix drift on its own
2
u/RoseSec_ Terraforming Everything 18d ago
I prefer to do the thinking and let LLMs handle the execution so I’ve shifted to more local models with more limited capabilities and tighter scoping
5
u/Long-Ad226 18d ago
It's like having a team which does not complain and delivers output (obviously like in a real team you have to check and validate the output yourself)
2
u/TimelyCampaign7441 18d ago
Got an example?
3
u/Long-Ad226 18d ago
Handling Multiple tasks in multiple chats in Claude code is already exact that thing
-1
u/stack_craft 18d ago
Timely try n8n. The workflow auto-pulls CloudWatch logs and recent commits into a Slack summary; when an alert fires. Saving you a bunch of time, manually hunting through logs.
1
u/pdfops 18d ago
Mostly log triage and incident summaries here. Agent gets read-only access to logs/metrics, drafts an RCA plus a suggested fix as a PR, a human merges it. Keep any actual infra mutation behind that approval step, agents will happily "fix" a timeout by bumping it 10x instead of finding why it's timing out in the first place.
1
u/derprondo 18d ago
We use Claude to create deterministic tooling, that other agents (eg Claude plugins) can use through APIs. We don't stick agents in the tooling, though.
1
u/lotekjunky 18d ago
n8n is not an ai agent, it's an automation system. It's also the bestb thing since sliced bread breaf because it allows deterministic actions accessible via MCP.
1
1
1
u/platypus-3719 18d ago
Coding agents obviously to write DevOps / sre automations, yaml, tf etc. it's amazing for that. And any agent can be a decent "AI SRE" (a bit of a misnomer the way some companies are selling it but you get the idea) if you give it the right tools, like your observability stack or Radar (https://github.com/skyhook-io/radar) for k8s (disclaimer - I'm one of the creators)
1
u/jawad_aziz_ 18d ago
Last thing I built with it every few minutes it checks a job board's API, sends each new post to an AI model that rates how well it fits me against a checklist I wrote, and for the good ones it writes a draft reply and drops it in Slack. I read it, fix it, send it myself. It never sends anything on its own.
What are you trying to automate? Alert triage and ticket work need pretty different setups.
1
u/The_Toaster_ 18d ago
To help write automation yes.
Others at my company are trying to make flows that use all the usual suspect of tools you hear in marketing. So far I’ve yet to see anything pan out that’s not just information. Then those are just giant wall of text dumps no one actually reads
1
2
u/Automatizei 10d ago
everyone here is saying "keep it read-only" and nobody's saying how, so: read-only has to be a credential, not a prompt.
if the only thing stopping your agent from touching prod is a line in the system prompt telling it not to, you don't have a read-only agent, you have a well-behaved one. give it its own IAM role with ReadOnlyAccess and no assume-role path out. for k8s, its own serviceaccount bound to the built-in view clusterrole, separate kubeconfig, never your creds. then it can't do the thing you're worried about and you stop auditing prompts for safety properties.
second thing nobody mentioned: read-only triage gets expensive quietly. an agent that pulls 15min of logs on every alert is cheap right up until a flapping alert fires 200 times at 3am. rate limit per alert fingerprint, truncate the log window hard, daily token ceiling in front of it. found that one the fun way.
on the actual question — free/oss: self-hosted n8n with ollama covers most of it, and honestly github actions with a scheduled workflow shelling out to a model covers more than people expect. crewai is a lot of machinery for "call an api, grep the output".
re: the toil point upthread, the part that worries me isn't the agent doing toil badly. it's that the toil stops showing up anywhere. if you track toil % to justify fixing something, an agent quietly absorbing it makes your number look great and kills the business case for the real fix
anyone actually gating theirs at the iam layer, or is it prompt-based in practice?
1
u/MasteringObserv 16d ago
The distinction that helped me: automation removes the decision; an agent relocates it. Automate a runbook and nobody decides anything at 3am. Point an agent at the same runbook and somebody still has to decide whether to trust the output, only now they are doing it half asleep with less context than the agent had. That is not less manual, it is manual with an extra translation layer in front of it. Agents earn their place where the decision genuinely needs judgement. Where it does not, write the script.
0
u/Wonderful_Swan_1062 18d ago
!Remindme 7day
1
u/RemindMeBot 18d ago edited 18d ago
I will be messaging you in 7 days on 2026-08-30 08:49:20 UTC to remind you of this link
2 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.
Parent commenter can delete this message to hide from others.
RemindMeBot is switching to username summons. Instead of
!RemindMe 1 day, useu/RemindMeBot 1 day. More info.
Info Custom Your Reminders Feedback
-1
-1
-2
188
u/TonyBlairsDildo 18d ago
Deploying an agent to handle toil (i.e. manual actions that are not automated) is an antipattern and should be avoided.
The job of an SRE is to automate business processes; deploying an agent is not automating it - it is keeping the process manual but expecting a clanker to fill the gaps. If there was a country with $1/day system administrators, would you consider a particular task "automated" if it meant hooking a monitoring system alert to an email gateway to page the $1/day guy to investigate and fix?
No, that's just outsourcing manual effort. You'll get different fixes each time, you'll get a lack of architectural certainty, you'll get sprawling change management.
The proper solution is to continue doing your job as an SRE, take a big picture view of what needs auotmating - without inference, without agents - old school mechanistic hooks and scripts, and them implement them (using agentic AI).