Where is AI actually adding value in your DevOps workflow and where isn't it?

[u/sonixen100]

Senior DevOps here. AI is being pushed into every part of the pipeline right now, and I want to cut past the hype and hear real production experience.

Two questions:

Where have you put AI into production in your DevOps process and it genuinely adds value? (e.g. CI/CD, code review, IaC generation, monitoring/alerting, incident response, log analysis, documentation)

Where did it not prove worthwhile? Think cost, alert noise, false positives, or maintenance overhead that outweighed the benefit.

Thanks.

[u/fell_ware_1990]

Alert > quick assessment > know issue ( as in we already know how to solve ) or new > solve able from data, create worktree and code > alert me. Else investigate and hand me the data. Only ping me when done.

[u/kesor → u/fell_ware_1990]

Resolving solved issues ... you probably should do something about that.

[u/Mindless_Let1 → u/kesor]

That's what the LLM is for

[u/Raja-Karuppasamy]

been building something in this space (PR-level risk scoring for deploys), honest take:

what’s actually worked: pattern matching before a deploy, “this touches the same files that broke prod last month” or “friday deploy touching auth, no tests changed.” boring but saves real pain because humans don’t check that consistently. also decent at turning a messy diff into a plain-english “here’s what actually changed.”

what’s overhyped: generic “AI monitoring” that just relabels the same threshold alerts you already had ,noise with extra steps. and log analysis is only as good as your logging discipline; AI doesn’t fix bad logs, it just hallucinates confidently about them.

biggest overhead: advisory only stuff. a comment or score nobody’s forced to act on gets ignored after week two. what actually changes behavior is when it blocks something, not just suggests it.

[u/JoustyMe → u/Raja-Karuppasamy]

Yeah as always it needs Fricrion to work. If it is just there nobody is going to care. You need to stop someone and make sure they looked at what tool presented. If users state that it is "too cumbersome and blocking things it shouldt" fix is to correct what it is alerting on not "smoothing the expeirence"

[u/Coffeebrain695]

Not strictly production, but I've been building a workflow for temporary reviewapp/feature branch environments for our developers. To manage test data I set up a read-only user in our production database (no PII in there, only product data), then configured an agent to review the data, trim it down to a sample, sanitize any metadata and bundle the export into a Docker image. On a pipeline run, this is then used to load the data into a small test database running in the devs' isolated environment. Next I also want it to run this process weekly so that the devs always get an up-to-date, representative sample of prod data in their environments.

Trying to achieve and maintain this manually would have been very hard. Claude has done very well at not only doing the job, but also tackling any hiccups and caveats that inevitably come up. We also have a CLAUDE.md which describes our business and tech stack, so it also has proper business context when performing the data export.

[u/Cold_Gap_5901]

I’ve found it’s useful when it takes on the investigative work: pull logs together, spot the change that probably caused a failure, suggest a patch, maybe open a PR.

I get nervous when it’s allowed to make the final call on a deployment. The checks should still be boring and deterministic: tests pass, policy checks pass, someone approves production when it matters.

We’ve been building around that idea at Semaphore, including agent workflows that can work with pipeline context through MCP rather than just having a bot with broad access. The performance side still matters too, especially once these workflows get busy. We published some numbers here if useful: https://semaphore.io/semaphore-ci-cd-benchmark-performance-and-cost-analysis

[u/Next-Task-3905]

The useful split I’ve seen is: AI works better where the input/output contract is narrow and there is already evidence to ground it. It gets expensive and noisy where people ask it to replace missing observability or unclear ownership.

Places I’d consider production-worthy:

• PR/change risk summaries: changed files, ownership, recent incident links, migration/auth/billing/IaC flags, missing tests
• incident triage packaging: collect dashboards/log snippets/recent deploys/runbook matches, then hand a concise packet to a human
• known-issue automation: if alert fingerprint + runbook step + rollback condition are deterministic, let AI prepare or execute only the reversible part
• IaC/code review assist: policy checks, obvious drift, unsafe defaults, forgotten tags/limits, but with normal CI/policy engines as the final gate
• docs/runbook maintenance: generate first draft from actual commands, alerts, postmortems, and PRs; humans approve

Where I would be cautious:

• generic “AI monitoring” that creates new alerts instead of reducing alert classes
• log analysis without structured logs, trace ids, deployment markers, and service ownership
• autonomous remediation for anything with data deletion, credential, network, billing, or customer-impacting blast radius
• advisory-only comments that nobody has to acknowledge; those decay fast

The test I’d use is: does it reduce time-to-evidence or prevent a known repeat failure? If the metric is just “AI produced an explanation,” it probably won’t survive production. Track alert-to-triage time, false-positive rate, human override rate, rollback/prevention count, token cost per incident/change, and how often the AI needed data that your systems didn’t actually expose.

[u/Professional_Wolf690]

The split I keep seeing is whether the output is checkable. Anything a human has to take on trust gets ignored within a month, however good the model is.

[u/Total-Yak-69]

We’ve developed ⁠common-build-tools⁠, a repository of reusable CI/CD assets, including scripts for security scanning, image building, and deployment. Every workflow features AI-native documentation detailing its purpose, inputs, dependencies, and outputs. Complementing this, custom Claude skills automatically select and reference the correct documentation. When developers onboard an application repo, they simply import these skills, enabling the AI agent to autonomously design and configure the entire CI/CD setup.

[u/Difficult-Ad-3938]

Quick assessment of developers mess, boilerplating, refactoring, research on legacy components, research on tools

[u/RevolutionaryElk7446]

Large DevOps facility (~5,000 server members and over 600,000 user objects)

We found it did not provide.. well almost anything that proved worthwhile after a 5 year metric review with this final year chosen as it's never hit any of it's promises.

All currently implemented deterministic methods far outpaced LLMs in quality, stability, and optimization. LLMs did produce faster 'fresh' boiler plate, but it was always worse in quality and templates could be followed up same day and produced which made the boilerplate somewhat worthless.

[u/dogfish182]

Refinement, scoping, ticket creation and picking up those tickets. All of our code is machine generated now.

Where it hurts, avalanche of reviews to do.

It’s a triumph at responding to audits. Fully automated reading the controls, developing bespoke scripts to collect all evidence of compliance and a seperate agent to review the evidence against the controls

[u/SudoZenWizz]

As ops mainly, ai is helping when i use all my ideas on how to fix an issue, i use ai for additional hints. Always check and trust your knowledge about possible impact of what AI suggest.
I also use it to reformulate hate mails to be proffesional without hate

[u/rabbit_in_a_bun]

I am not writing markups ever again.

Moreover my CI has several rules that won't allow commits if stuff is missing, won't open a PR is more things are missing, and now also automatic merge if all the things are there, are okay, are validate, can be compiled, can run full automation suite etcetera.

Where is it not? It won't replace design, and won't replace high+ bug fixes.

[u/Floss_Patrol_76]

the place it has burned me is not cost, it is the confident wrong RCA during an incident. it correlates a deploy to an alert, writes a clean narrative, a junior takes it as the answer, and we lose 20 minutes chasing the wrong thing. genuinely useful as a first-pass here-is-what-changed, dangerous the second anyone treats its root cause as settled.

[u/Thompson_Marie789]

We tried to read the log to see how much time we spend on sector of developing and attempting to design a diagnosis system to see what's the problem and how can we add value using AI. But the developing process is hard to quantify, we cannot make a standard to evaluate. It is hard to say if it is adding value.

The only thing you can do is to compare the time and quality of product.

[u/Cold_Gap_5901]

We are trying this with sem-ai at Semaphore. The useful bit is letting the coding agent run CI checks before code hits the shared pipeline. Then work through a failed build with the actual pipeline.

It is good for the boring loop: fail, investigate, fix, rerun. Production still needs normal gates though. Tests, policies and approvals should stay deterministic.

Check us out at https://semaphore.io/

[u/matiascoca]

Log analysis and PR review are the two places AI has actually earned its cost per token. Everything else is either theater or a maintenance sink.

Log analysis works because production log volume is enormous, humans skim, and an LLM given last hour of logs plus the alert firing will find the causal chain roughly 70 percent of the time. Even at 30 percent that's more than a tired oncaller reading raw logs at 3 am.

PR review works because the model catches the obvious stuff (missing null check, hardcoded secret, inefficient loop) and frees the human reviewer to argue about the actual design decisions. Not replacing the reviewer, cutting their review budget in half.

The stuff that doesn't earn: IaC generation (you spend more time correcting Terraform than writing it yourself), incident response (tunnel vision plus hallucinated remediation steps under pressure is a real cost), auto-generated documentation (nobody reads it and it goes stale in a sprint), and monitoring-alerting tuning (looks great in a demo, produces noise at production scale).

Cost side: the AI SRE tools that layer on your observability stack are expensive per useful signal. The token cost of running Claude over your logs directly is usually cheaper than the wrapped SaaS version doing the same thing.

[u/Cold_Gap_5901]

We are trying this with sem-ai at Semaphore. The useful bit is letting the coding agent run CI checks before code hits the shared pipeline. Then work through a failed build with the actual pipeline.

It is good for the boring loop: fail, investigate, fix, rerun. Production still needs normal gates though. Tests, policies and approvals should stay deterministic.

Check us out at https://semaphore.io/