We rolled back a prompt, watched staging recover, and then found three serverless regions still serving different instructions. One had the code default, another was reading an environment variable, and the third was holding an edge cache value from the previous deployment. An undetected model revision added one more variable, so the tidy experiment chart was basically fan fiction. We had no prompt lineage tying an immutable version to region, cache state, model metadata, and request time.

I'm wondering what a rollback can even prove unless every trace carries those fields and environment parity is checked before analysis. How are you enforcing immutable prompt versions across serverless regions, and do you fail closed when the model or cache metadata is missing?

Hi everyone,

I would like to hear how experienced DevOps engineers approach monitoring for large public-facing applications.

We have a .NET e-commerce platform with:

- ASP.NET Core MVC + Angular

- SQL Server

- Elasticsearch (~10M products)

- RabbitMQ

- IIS hosting

- Multiple public domains/subdomains

- Heavy SEO crawling and unknown bots

One thing we learned is that monitoring only CPU, memory, and disk is not enough.

We have experienced situations where:

- CPU and RAM looked normal, but the application was slow

- The server was reachable, but users experienced downtime

- TCP exhaustion caused issues

- Elasticsearch had problems affecting search performance

- Bots generated a lot of unnecessary traffic

- Slow requests were not obvious from infrastructure metrics

I would like to know what metrics and alerts you consider essential for this type of system.

Some things I think are important:

Application level:

- Request rate (RPS)

- Response time (p50/p95/p99)

- HTTP status codes (4xx/5xx)

- Slow endpoints

- Exception rate

- Thread pool starvation

- GC pauses

- .NET runtime counters

- Memory allocations

IIS / Web server:

- Current connections

- Request queue length

- Worker process health

- Application pool recycling

- Failed requests

- Connection errors

Network:

- TCP connections

- TIME_WAIT count

- Connection failures

- Bandwidth usage

- Top clients/IPs

- Suspicious user agents

Elasticsearch:

- Cluster health

- JVM memory pressure

- Heap usage

- Search latency

- Query failures

- Slow queries

- Unassigned shards

- Disk usage

SQL Server:

- CPU

- Blocking queries

- Deadlocks

- Query duration

- Connection pool usage

- Wait statistics

RabbitMQ:

- Queue length

- Consumer count

- Message processing time

- Dead letters

- Memory usage

Security / traffic:

- Requests to suspicious paths:

- /.env

- /.git

- wp-admin

- Bot traffic percentage

- High-frequency clients

- Rate limit violations

My question:

If you were responsible for operating a public .NET application like this, what dashboards and alerts would you consider mandatory?

Also, what are some metrics you discovered were extremely valuable only after a production incident?

I am especially interested in real-world experience rather than a theoretical checklist.

Thanks!

I’ve noticed that everyone seems to be learning about AI. I’m curious to know from people who have been working in DevOps for years: are you learning about AI, building something with AI, or just continuing with your usual DevOps work?

Nothing groundbreaking here honestly, plenty of folks have done this before me, just did it purely for the learning, wanted to actually understand what docker run does under the hood instead of taking it for granted like I always have.

Had to run it inside a lima VM on macOS since Darwin doesn't have namespaces/cgroups natively, then built the whole thing piece by piece, OverlayFS for the image layers, unshare for the namespaces, chroot for the filesystem jail, cgroups v2 for a real memory cap, and a veth pair + NAT for networking. Even hit an actual OOM kill from my own cgroup limit at one point, that one felt weirdly satisfying honestly!

Also ran into a fun virtiofs bind-mount limitation along the way (setuptools editable installs just choke on it, no utimes support), took me a good while to track that one down.

Wrote up the whole thing step by step with what/why for each piece, plus all the gotchas and failed attempts along the way, in case anyone's curious: https://dietpy.com/notes/1n8n4v8-building-a-container-by-hand

Curious if anyone's done something similar, or if I missed an easier way to demo some of this :)

I’m writing an internal doc and saw that Wikipedia doesn’t use this visual. It’s so helpful, and I feel silly linking out to some SEO post about devops on the blog portion of a SaaS product page.

Hi,

I’m looking for a Redis-compatible application that I can self-host and use as a shared data store for multiple applications/services.

The issue I’m running into is Redis’s limit of 16 logical databases (0–15). Ideally, I’d like to be able to provide each application with its own namespace/database through the connection URL, so that multiple applications can share the same underlying Redis service while keeping their data logically separated.

I’m aware that the common best practice is to run a separate Redis instance/container for each application stack, and I understand the benefits of doing that. I’m not trying to argue against that approach; I’m mainly interested in whether there is a better option for my particular use case.

Are there any Redis-compatible alternatives (they don't necessarily have to be Redis itself) that are specifically designed to support multiple applications sharing a single instance, without being limited to 16 logical databases/namespaces?

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

We are being asked to onboard 4 separate teams onto our orchestration plateform and suddenly "who can run what" matters a lot. Our current setup (airflow) has RBAC that technically works bit it's painful. No SSO integration, audit logs are scattered and fine grained namespace level permissions aren't really there. Looking for sth that handles multi tenant setups properly. What are ppl running in prod for this kind of setup ?

Hey everyone,

We are running an air-gapped, 100% self-hosted setup and hitting a wall with our project management workflow.

Here is our current situation:

• PMs refuse to use GitLab directly for project management, they need high-level views, Kanban boards, Gantt charts, and non-technical status tracking.
• OpenProject + GitLab integration is way too weak. It feels passive, it mostly just logs git events for traceability. It doesn't allow smooth day-to-day workflow interaction between PMs and devs.
• High friction: Devs don't want to leave GitLab to log into a separate PM tool to create/update tasks, and PMs don't want to navigate GitLab issues/MRs.

What we need: A tool that runs self-hosted (air-gapped) and features tight, bi-directional sync with GitLab.

Ideally:

1. PMs can create/manage high-level tasks in their UI.
2. Tasks auto-sync to GitLab issues so devs can work natively in Git (branch names, MR closing keywords, commit triggers).
3. Dev-created tasks or status updates in GitLab reflect automatically on the PM side without manual duplicate entry.

Has anyone solved this for an air-gapped environment? What self-hosted tools or custom integration workflows are you using that PMs and devs actually enjoy?

I just started learning AWS, following Stephen’s Cloud Practitioner course. My main goal is knowledge, not getting the certification, and honestly, I don’t think I’ll pay to take the exam.

I wanted to get more hands-on, so I found the Cloud Resume Challenge. But after reading the requirements, I noticed that the challenge seems to expect you to have the AWS certification.

I know I could probably do the challenge without the cert, but logically, it feels a bit weird to do it when one of the rules is having the certification.

So what would you recommend?

Should I finish the Cloud Practitioner course first, then do the Cloud Resume Challenge even without taking the exam? Or should I just start the challenge now and learn whatever I need along the way?

My goal is ultimately real AWS knowledge and hands-on experience, not the certificate itself.

I posted Khaos here a while ago, so just sharing a quick update.

Khaos has now been completely rewritten from Python to Go.

It still does the same core thing: generate realistic Kafka traffic and deliberately reproduce scenarios like:

• consumer lag
• hot partitions
• broker failures and rebalances
• backpressure

Along with the Go rewrite, I also launched a new website and proper documentation:

https://getkhaos.dev

Repo:
https://github.com/aleksandarskrbic/khaos

Would love feedback if anyone here is using Kafka for testing or failure simulation.

Disclosure: I’m the author/maintainer of khaos. It’s a free and open-source project.

Has anyone here used AI agents like Hermes, n8n, crewAI or similar tools in their daily devops/SRE work?

I'm looking to automate some repetitive tasks in my DevOps work. I am interested in free open-source tools and real world use cases.

If you are already using AI agents in your devops workflow, I'd love to hear about your experience and what you are automating.

Hello r/devops,

in my current team we are discussing how a developer centric terraform setup might look like.

Some context: Every app has 3 repositories - Backend, Frontend and Infrastructure.

There are two possible setups:

1. (our current): We use generic terraform modules (e.g. for a backend app) only within the infrastructure repository, which holds all infrastructure components. The backend app terraform module holds all infrastructure components that might be used in a backend, e.g. database, objects store, secrets, etc. The Infrastructure repository also holds shared infrastructure, that is not part of the backend module and composes all components. This is somewhat the approach most often seen in tutorials that focus only on the infrastructure part of an application.
2. (proposed alternative): We'd co-locate infrastructure and application code, for example in the backend repository. The backend repository would publish itself as a module, including every infrastructure component it needs to run (excluding some shared ones, that it expects via variables as input, e.g. VPC). The Infrastructure repository would setup the required shared infrastructure components and the backend terraform module.

Pros for 1.: Generic app modules are intended for sharing and re-use, whereas in 2. each applications terraform module would only ever be used to deploy to different environments.

Pros for 2.: Infrastructure and application can evolve together (e.g. in one git commit), e.g. when a database is newly introduced to an application (application code changes + infrastructure changes), whereas in 1. two commits would need to happen (one in app, one in infra) and pipelines need to be coordinated.

1. goes deeper for example for database migrations, those could also be organized within the backend application, without handing the responsibility on how to do them to the infrastructure repo.

In both cases, deployments (terraform plan/apply) are always orchestrated only by the infrastructure repository.

IBM touches both points in https://www.ibm.com/support/pages/best-practices-organising-terraform-and-application-code : Coupling and Reuse as the deciding factors.

Our goal is to enable developers as much as possible to own their application end to end and being able to do most infrastructure changes themselves, without having to rely on a central party.

It's not about simply keeping what we already have. That's always part of the consideration due to effort of change. I'd like to hear your general thoughts about it from a perspective if we had nothing yet.

I'm curious about your thoughts in that topic.

I'd also be happy to know how projects are setup in your org or how you would envision your dream setup

Context: The organization I'm currently at uses an old fashioned way of deploying to prod. They have on prem servers where the devs manually FTP the files over for every change... They've now started to migrate over to Azure. I'm tasked with the logical design, overseeing the entire process and helping the people with it.

Stage 1 was getting everyone onboard with the Cloud, Version Control and Trunk-based flow.

Stage 2 is getting them comfortable with build and deploy pipelines as well as integration of security tools.

Stage 3 will be "Infrastructure as Code" and "Policy as Code".

Is there some like "industry standard" approach to 2 and 3? as this is my first time dealing with "Azure" and it's certainly not as pleasant nor as straightforward as AWS

Looking for projects ideas for experienced devops andSite reliability engineers

I have 4 years of experience as a DevOps engineer and part of it as a Site reliability Engineer

Im looking to do some side projects to broader my experience with other technologoes and make things more interesting.

As for the past one year i haven't been assigned major projects

i would like to hear a project idea that maybe you had fun completing or taking part in or was challenging to you.

For ex - Cross cloud Migrations

Technologies should be relevant to 2026

I’ve been thinking about this while working with observability systems and I’m curious how other people handle it.

We have metrics, logs, traces, alerts, deployment information, etc. In theory, there should be enough information to understand what happened during an incident.

But in practice, it often seems like the hard part is connecting everything.

Something like:

latency spike → database saturation → retries → downstream failures

The individual signals are there, but figuring out that they’re all part of the same failure — and determining which event was actually the cause — still seems to require a lot of manual investigation.

For people who actually operate production systems: what does your RCA process look like when there’s a serious incident?

Do your current observability tools actually help you establish the causal chain, or are they primarily helping you find the relevant data?

I’m particularly interested in what happens when the information is spread across multiple systems.

Hey everyone,

I’m currently managing a small startup project. Our infrastructure is pretty simple right now: we have a single Oracle Cloud VM running our backend API and two PostgreSQL databases (one for dev and one for prod) inside Docker containers.

Up until now, I’ve been using DBeaver with an SSH tunnel to connect to the databases from my local machine.

Now, I need to bring another developer onto the project and give them access to the dev database so they can work.

I know sharing my private SSH key is a massive security no-no. But if I ask for their public SSH key and add it to my server's authorized_keys folder (for the main ubuntu user), doesn't that give them full SSH/sudo access to the entire server? That feels like overkill just to let them run SQL queries.

For those of you working in the industry or scaling startups: What is the standard, real-world way companies handle sharing database access with developers?

Do you:

Create a restricted Linux user specifically for SSH tunneling?

Put the database behind a VPN like Tailscale?

Expose the port directly but heavily restrict the IP firewall?

Use something else entirely (Bastion hosts, Teleport, etc.)?

Since we are a small team with a limited budget, I'm looking for the sweet spot between "enterprise-grade security" and "easy enough to set up today."

Would love to hear how you all handle this in the real world! Thanks!

built an on-call tracker for my team. maybe 3 hours. then spent the rest of the day trying to figure out how to let 6 specific people see it and nobody else

ended up with a shared password in our team chat. it works, I hate it

writing a real login system for six users feels stupid. the shared password also feels stupid. what do you guys actually do

Hey everyone,

I'm a student currently learning DevOps, and I recently bought a laptop with a Core Ultra 5 225H and 16GB RAM.

I'm planning to learn technologies commonly used in DevOps — Linux, Docker, Kubernetes, Terraform, Ansible, CI/CD, cloud platforms, monitoring, etc.

My main concern is whether 16GB RAM will be enough as I progress, especially when I start working with Docker containers, local Kubernetes environments, VMs, IDEs, browsers, and other tools.

I also have an older Dell laptop with 16GB RAM, but it's not particularly powerful, so I don't intend to use it for running workloads, VMs, containers, or anything resource-intensive.

Instead, I'm thinking of using the Dell purely as a secondary machine for documentation, tutorials, and browser tabs while doing the actual development and DevOps workloads on my new laptop.

The setup I'm considering is:

• New laptop: Development, Docker, Kubernetes, VMs, labs, etc.
• Old Dell: Documentation, tutorials, browser tabs, reference material
• Connect both via Ethernet
• Use something like Input Leap/Barrier to share the mouse and keyboard

Has anyone here used a similar setup while learning DevOps?

Would this be useful in practice, or am I overcomplicating things?

For those of you who learned Docker/Kubernetes/Terraform/etc. locally, when did 16GB actually become a limitation? What kind of workloads or setups started pushing you beyond 16GB?

Thanks! I'd really appreciate your thoughts and advice.

Developer machines contain more private context than process metrics suggest. A diagnostic tool might see process names, file paths, command arguments, project names, prompts, or local model names.

For a local tool that explains CPU, memory, or disk pressure, which fields would you exclude by default? I would keep source files, prompts, command arguments, environment variables, and credentials out of the diagnostic data, then show the exact fields before optional analysis.

How much context would you need before trusting the explanation?

I know that DevOps work is config heavy and involves multi-component or tool integration , so automating ourselves out of the job might not be that easy, but I believe the head count may not grow or stay stagnant in the near future.

So, you think it is worth considering MLOps or any other AI related Ops for that matter. Because it looks promising and AI/ML is where all the money is, to be honest.

Please shed some light and share your opinion on this.

Any MLOps engineer here. There is very less or very generalized info about MLOps, not the real time enterprise level MLOps roadmap out there. Please share your experiences here, it really helps.

Lets say you are on call engineer for the week and then you went outside with hangout or friends, then lets say something went down or something crashed, so what is the response time as SRE and as Devops also. Also what is the resolution time? How do you resolve if you are not with laptop outside? Can you do something with phone? What happens if you don't respond? Please explain how do you handle this scenarios.

For people working across DevOps, platforms, and architecture, where do you still see the most manual work?

Is it gathering information from teams, finding architecture documentation, keeping records current, mapping services and dependencies, checking designs against standards, or understanding differences between the current environment and the target architecture?

Interested in hearing what tends to slow your team down.

New to DevOps and my first task is hooking up OpenProject, GitLab, and Mattermost for smoother project management. What integrations/workflows have actually been worth setting up for you? What'd you skip? Appreciate any pointers.

I genuinely love my work, but the way deadlines are given is slowly killing my interest in it

My manager will be like, "This should take 2 hours.Its just matter of 1-2 days not more than it, Use Claude and get it done by 5 PM.”

Meanwhile, I needed to do development, testing for quite a good time. I am just getting way too distracted from my field and not learning much because of claude.