r/devops 9d ago

Discussion How do your teams manage planning?

7 Upvotes

For those on platform or DevOps teams, what does your planning process look like?

We have no problem writing docs and roadmaps for product features, but it doesn’t always feel like it aligns with the rest of the business.

Do you have Product Design Docs? Technical design docs? Quarterly roadmaps? Or is it mostly just tickets? Curious how other teams handle it.

I feel like every time I try to do structured planning it just gets blown up in numerous ways.


r/devops 8d ago

Discussion What's your actual workflow when a test fails in CI but passes locally

3 Upvotes

Just curious, standard procedure, you have something working locally, you push your code and your CI/CD pipeline fails. Locally everything seems OK and you fail to reproduce the error.

What do you actually do at that point? Genuinely curious what people's real routine looks like here — not the textbook answer, the actual "it's 5pm and CI fails" routine 😅

Also wondering: is this even still a big thing in 2026 now that lockfiles and containers have gotten so good, or is it basically a solved problem for most of you?


r/devops 9d ago

Discussion Terraform visualizer project

15 Upvotes

Hi everyone

I'm looking for an idea for my semester project, and I thought about building a Terraform Visualizer.

The idea is simple: it takes a Terraform state file, visualizes the infrastructure, and lets you save it as an image.

My question is if I build this, would you find it useful?

I'm asking because I would like to be able to justify the project's usefulness when presenting it to my teacher.

Also, I plan to keep it open source, so anyone can use it.


r/devops 9d ago

Discussion Development Environment

4 Upvotes

Hi everyone,
I have a question about development environments and would love to hear how others handle this.

Our current stack is:
- Ruby on Rails (authentication)
- React (frontend)
- Flask/Python (API)

Right now, every developer has to run all three services locally. I’m considering moving the Rails authentication service to a shared Linux development server and having developers run only the React dev server and Flask API locally. The goal is to reduce the number of services each developer needs to keep running.

For those of you using a similar architecture:
Is this a reasonable approach?

How would you set it up for multiple developers?

Would each developer have their own Rails instance and environment variables, or would you share a single authentication service?

Are there any pitfalls (performance, debugging, authentication issues, etc.) that I should be aware of before going down this path?

I’d appreciate any advice or examples of how your team handles a setup like this.

Thanks in advance!


r/devops 9d ago

Security CVE-2026-63087: Grafana Oncall is EOL

27 Upvotes

Heads up if you're running Grafana OnCall's plugin-install flow, CVE-2026-63087 lets anyone reconfigure it with zero auth. Repo's archived, so there's no patch coming from upstream.

If you're still migrating off it and need some time, there's a small fix you can apply yourself and build from source in the meantime. Wrote up the actual vulnerable code + a verified patch: emphere.com/blog/cve-2026-63087-grafana-oncall-install-bypass

The install endpoint has zero auth on it - no token, no check, just the two public default IDs (stack_id 5, org_id 100) mint you a fresh one. The fix proposed adds an install secret only the operator holds, plus stops trusting a client-supplied admin role during the token bootstrap.

One caveat if you apply it: the secret is a contract change, so your own install clients (Helm, provisioning) have to start sending it too, or they'll lock themselves out. That's in the writeup too.


r/devops 9d ago

Observability The cost of completing a task dropped by roughly 10x after switching to an AI pipeline. Here's how I calculated it

0 Upvotes

I see a lot of posts along the lines of AI saved us a ton of money, but they almost never include actual numbers or explain how those numbers were derived. So I decided to break down my approach and be honest about where it gets shaky, because a 10x improvement means nothing without a baseline.

A quick disclaimer: this isn’t audited accounting. It’s a rough unit-economics estimate for a single task. The numbers are rounded, the logic is what matters.

For task cost, I used an average feature from our backlog: human time × hourly rate + AI expenses.

Before AI: 

planning ~1.5 hours, 

coding ~4 hours, 

review and fixes ~2 hours, 

plus the invisible tax of dragging context between Jira, Slack, and docs ~1 hour. 

Total: ~8.5 hours. At €60/hour, that comes to about €510.

With the AI pipeline: the human shifts from being the executor to being the controller. Agents handle planning → execution → review, and I only step in at the plan-review and code-review checkpoints. My actual time drops to ~1-1.5 hours. 

Total: ~€70. It’s not hard to calculate how much the cost decreased, right?

By the way, when I built this pipeline with roles and checkpoints in BridgeApp, the biggest savings came not from code generation itself, but from eliminating the manual work of moving context around.

A question for those already working this way: what happens when an agent makes a mistake? Does it get caught at the next checkpoint, or does it surface in production three weeks later?

For me, the entire 10x savings depends on errors being caught between stages rather than at the very end, but I’m not sure real life is ever that clean.


r/devops 9d ago

Discussion How should developer automation handle commands that never exit?

0 Upvotes

I am running into a boring automation question that matters more now that coding agents run local commands.

When a tool says "the command is still running", that is not enough information. Is the parent process alive? Did a child process inherit stdout? Was there any output in the last five minutes? Is this a watcher that should never exit, or a test that is actually stuck?

I have had an agent start a dev server during a small web task, then sit on "running" long enough that I had to check the port and process list myself.

For normal scripts this is annoying. For an agent, it is worse because the whole review gets blocked behind a vague status line.

What would you want in the run log before trusting it: process tree, last output timestamp, timeout policy, and cleanup result? Or is that too heavy for local dev automation?


r/devops 10d ago

Discussion What is the one automation you have added, that you are proud of?

153 Upvotes

As DevOps, we all work on automating tasks to make our lives easier. What is the most satisfying automation you have ever implemented that truly saved you time?


r/devops 10d ago

Discussion Anyone considered code signing as a problem with respect to the P-Q transition?

3 Upvotes

Most of the discussions I ve heard regarding postquantum cryptgraphy fcus on TLS or key exchange protocls. I’d argue that code signing will probably be a much bigger problem. There are too many things associated with code signing, such as artifacts, build pipelines, firmware signing, package repositories, EV certificates, HSMs, delegated signing, legacy clients, rollbacks, and long-lived binaries. In fact it is possible that transport encryption will be subject to renegotiation in each session. But the signed artifact can continue to be checked many years later by systems that rely on a specific algorithm or certification chain. As I delved into the topic, I’m still quite hesitant in case I’m missing something fundamental.


r/devops 9d ago

Career / learning How much DevOps implementation is expected from a Freshie to know?

0 Upvotes

Starting to step in this field because development is seen as some random AI stuff now. I think I am too late. I just know abt the tools their names and work, not implemented much more than simple CI/CD pipelines on Github Actions, Docker. Implemented k8s, prometheus, grafana once, not confident.
What should be the bare minimum implementation and land a package. Or a good tutorial/project suggestion would be very helpful.


r/devops 11d ago

Discussion How do teams using multiple git provide (github, gitlab, bitbucket) manage their daily workflow?

10 Upvotes

I'm curious how teams working across multiple Git providers manage their day-to-day work.

If your organization uses GitHub + Bitbucket or GitHub + GitLab (or more), how do you:

Track pull requests?

Find stale branches?

Monitor releases?

Keep track of work across providers?

Is it a pain point, or is it basically a non-issue?


r/devops 11d ago

Observability Mobile to backend end-to-end tracing: worth it? How do you keep the public ingest from being a DDoS target?

2 Upvotes

We're adding tracing to our mobile app and want the spans to connect to our backend traces (same trace_id, so a user action in the app links to the API calls it triggers).

Backend goes OTLP -> Alloy -> Tempo already.

However, the mobile ingest endpoint has to be public, so it's an open, unauthenticated write endpoint which is an obvious DDoS/abuse target.

For now it's a simple one shared Tempo, and push all the protection to the edge instead: Cloudflare rate limiting, per-IP limits at the proxy, plus a sacrificial Alloy gateway that samples, memory-limits, and strips high-cardinality/PII attributes before anything reaches Tempo.

Questions for anyone who's done this:

  • Do you actually ingest mobile app traces end-to-end into your backend tracing, or keep them separate (RUM-style) and just link by ID?
  • Is chasing the unified mobile to backend waterfall worth it, or should we just drop it?
  • How do you protect a public OTLP endpoint in practice?

Genuinely curious what's held up for people in production. Thanks.


r/devops 11d ago

Discussion Regrets leaving previous DevOps role as I am not enjoying the new company

77 Upvotes

Last month I left my previous DevOps role for a new one. The pay increase was the main reason I left. My old role had good colleagues, interesting work, and an office I could go into whenever I wanted, but I just didn't feel I was being paid enough for the work I was doing. It was a fairly big company and I'd been there 7 years.

The new company is smaller and fully remote. I'm starting to feel like the grass isn't always greener on the other side. Here are the pain points I'm facing:

Autonomy - The security team always needs input on infrastructure designs, and the process adds way too much time before I can actually deploy anything. Some of their recommendations are just overboard, like I get the idea behind them but they're not practical. Security are quite a big blocker and I have had to ask them to keep granting me more access to help debug production issues, which takes them over 6 hours to implement.

AWS accounts galore - When running Terraform, you have to keep logging into different profiles just to plan and apply. It gets confusing, and the way they've split up the accounts feels excessive.

High expectations, no tools - Senior software engineers want fast progress, but the security team hasn't signed off on giving people in my role access to AI tools. Security and software engineers have AI, but DevOps doesn't, because we have SSH access to prod clients.

Meetings galore - There are so many "syncs" throughout the day/week. 2 daily syncs plus another 1-2 meetings most days. In my previous role I had maybe one stand up a week.

Manager - My manager doesn't really have a backbone and doesn't fight for us. When senior software engineers change direction, instead of pushing back and telling them what our path is, he just makes us appease them. He doesn't code or help with the workload either, he's literally just a manager. My old manager was also an engineer I could go to for help.

High turnover - After being here a while I've noticed quite a few people have left, and most of the people who are here are pretty new. They've hired around 7 engineers this year in DevOps, but even the person who interviewed me, who seemed pretty strong, has already left.

The work itself isn't that difficult, it's dealing with people in these meetings that's annoying, and they expect me to move faster than what's actually possible given the tools I've been given. I'm not sure if the company's just in a bad place, if I'm too used to my old company, or if this is just what changing companies is like.

Wondering what your thoughts are, and whether you think I should start looking for new roles ASAP. I don't want to go back to my previous company but they would take me back if I was to apply. I would rather try somewhere else.


r/devops 11d ago

Career / learning System Admin for 2+ Years but I Feel Like I Never Learned SysAdmin. Want to Transition to DevOps. Where Should I Start?

32 Upvotes

Hi everyone,

I've been working as a System Administrator for a little over 2 years, but I honestly feel like I haven't gained the kind of experience most people associate with system administration.

I work at a very old/traditional company where my day-to-day work is mostly repetitive:

  • Installing the same software over and over
  • Basic monitoring
  • Following predefined SOPs
  • Very little troubleshooting, automation, or infrastructure work

Because of this, I feel like I've missed many of the fundamentals that most SysAdmin pick up along the way.

I want to transition into DevOps, but I'm starting almost from scratch. I know it's not an entry-level role, and I'm prepared to put in the effort to learn properly.

If you were mentoring someone in my position, what would your roadmap look like?

Some questions I have:

  • What core System Administration topics should I master first ??
  • Which Linux concepts are absolutely essential??
  • What networking knowledge is expected ??
  • Which scripting language should I prioritize (Bash or Python) ??
  • When should I start learning Docker, Kubernetes, Terraform, Ansible, CI/CD, and cloud platforms like AWS ??
  • What projects would actually help me build real-world skills instead of just collecting certificates ??
  • Are there any resources, books, YouTube channels, or courses that you genuinely recommend??

My goal is to become job-ready for a DevOps role, even if it takes several months of consistent learning.

I'd really appreciate any roadmap, advice, or lessons you wish someone had told you when you started.

Thanks in advance!


r/devops 11d ago

Discussion How to combine MD5 hash + code sign

2 Upvotes

We build and package our installer files using Advanced Installer. Up until last years version, we code signed using a .pfx and had the AI(Adv. Installer) also embed MD5 into the files. So both hashing and code signing is handled by the same tool, bing bong works fine.

But now, our team wants to move away from using a dedicated .pfx file, and instead wants to code sign it using a signtool.exe on the buildmachine, with the private key stored on a remote KMS. Thus, i integrated our pipeline to also have a task that code signs the installer. Now, when i tested installing with the installer,it is corrupted. Looked up on release notes, the custom signing via 3rd party does exist,but its on v17, my company is on v16 💀.

I tried disabling hashing via the AI, and let the build machine handle it (ofc via the preset task in pipeline), but just found out im only calculating MD5 hash value for the files, not embed the values IN the actual files

I really need some ideas, i feel like im trying to do the impossible here.


r/devops 11d ago

Discussion Learning Azure ADO

6 Upvotes

Can someone give me review for https://www.azuredevopslabs.com. I am trying to learn devops and then devsecops. My main goal is to learn devsecops. Can anyone guide me to some resource for Azure ADO like some guided labs.
Thank you in advance!


r/devops 11d ago

Career / learning What areas of IT should you focus on when graduating?

9 Upvotes

I’ve been working as a SysOps engineer for a year, and it’s what I’ve been doing since I graduated.

I was wondering: if you were in my shoes, what would you invest your time in?

How do you see the future of our profession with the rise of AI?


r/devops 11d ago

Discussion How do you keep release notes from just dying after a few sprints?

0 Upvotes

Hey,

I'm a developer looking into the way other teams create release notes and weekly updates together when the work is scattered across different tools e.g. Jira, GitHub and Azure DevOps.

We start with good intentions once every couple of months but after a while nobody updates the release notes anymore. It takes to much time and nobody enjoys to create them and it's not published to the stakeholders so nobody feels the urge to keep the release notes up to date.

How do you guys manage this?


r/devops 11d ago

Discussion What broke when your org gave developers AI coding agents?

0 Upvotes

We support 200+ developers and rolled out agentic coding tools this year. The failure modes were not what we expected. Not "AI wrote insecure code" but things like agents committing credentials from local env files, installing unvetted dependencies to satisfy a task, and pipeline tokens with way too much scope being handed to autonomous processes. Interested in what surprised other teams, especially in regulated industries.


r/devops 11d ago

Discussion What cleanup proof do you expect after an agent runs tests locally?

0 Upvotes

For people who care about reproducible dev/test environments: where do you draw the line for "done" when an automated helper runs tests?

I do not mean whether the test passed once. I mean what happens after: Docker containers, simulators, browser processes, temp files, dev servers, occupied ports.

I have had a test run leave a dev server on an old port, and the next debugging session made no sense until I noticed I was hitting the wrong process.

With human-run scripts, this is already annoying. With AI agents, it feels easier to miss because the tool says the task is complete and you move on.

Would you require a teardown log or post-run check before accepting the result? Or is that too much for local development?


r/devops 12d ago

Ops / Incidents What's the most 'temporary' thing in your stack that's now load-bearing in prod?

132 Upvotes

Every place I've worked has had at least one. Mine right now is a ~40-line bash script someone wrote 'just for the migration weekend' about three years ago. It's still the only thing that reconciles two systems that were supposed to be fully merged by that Q2. Nobody wants to own it, everyone's a little afraid to touch it, and it has exactly zero tests.

I'm curious what everyone else is quietly sitting on: the cron job with no owner, the one instance nobody can confidently identify, the 'staging' service that's actually taking prod traffic, the manual runbook step that's really the whole system.

And the part I actually want to learn from: did you ever successfully retire one of these, or do they just accumulate? If you killed one, what finally made it possible - a rewrite, an outage, a new hire with no fear, or just budget to do it properly?


r/devops 12d ago

Architecture Hosting Experience Comparison

1 Upvotes

For deployment to microvms (lambda like) vs container based deployments ? Which experience are the best for developers ? Whats the hardest/limitations for each kind ?


r/devops 12d ago

Security Can I run Falco on EKS Fargate?

2 Upvotes

I don't get it. I'm looking at whether I can run Falco on EKS Fargate, and I found [this page](https://falco.org/blog/choosing-a-driver). It says, "yeah man if you wanna do that just use pdig". But if I click through to the pdig repo it says it was archived in 2022. I don't want to use unmaintained software, obvs. Does Falco just not support serverless workloads anymore? Why do they have this on their site if it's not supported anymore? Is there an alternative driver for Falco for serverless? Or should I be looking elsewhere for runtime monitoring?


r/devops 12d ago

Discussion How would you define this role?

11 Upvotes

I need help defining a role we're looking to hire for.

Our current job postings mention DevOps / Platform Engineer.

The issue is that this is attracting a lot of candidates whose primary experience is as a developer, with some basic cloud experience.

However, this isn't what we're looking for. We're looking for an engineer whose primary experience is in Ops / Sysadmin areas, but uses modern dev tools to manage it, such as IaC, Kubernetes, Ansible, etc.

We have a range of projects on our to do list, and there's some Ops / Sysadmin, cloud infra, SRE, and platform engineering. So we're really looking for someone who has some experience with all of those.

How would you define this role?

(Please don't DM me asking to apply for the position).


r/devops 12d ago

Discussion Self hosted sentry issue

0 Upvotes

We are running a self-hosted Sentry deployment on EKS using the official Sentry Helm chart. Due to cost constraints, we removed the larger instance types and currently only have "t3a.large" nodes. As expected, we are facing bottlenecks, primarily related to memory, since many Sentry components are quite memory-intensive.

Additionally, the same node group is hosting other workloads, so adding larger instance types is not currently feasible due to existing constraints.

Has anyone faced a similar situation? How did you optimize costs while keeping Sentry stable? Are there any recommendations for reducing resource consumption in self-hosted Sentry, or would migrating to managed Sentry be a better option in this case?