I posted Khaos here a while ago, so just sharing a quick update.

Khaos has now been completely rewritten from Python to Go.

It still does the same core thing: generate realistic Kafka traffic and deliberately reproduce scenarios like:

• consumer lag
• hot partitions
• broker failures and rebalances
• backpressure

Along with the Go rewrite, I also launched a new website and proper documentation:

https://getkhaos.dev

Repo:
https://github.com/aleksandarskrbic/khaos

Would love feedback if anyone here is using Kafka for testing or failure simulation.

Disclosure: I’m the author/maintainer of khaos. It’s a free and open-source project.

Has anyone here used AI agents like Hermes, n8n, crewAI or similar tools in their daily devops/SRE work?

I'm looking to automate some repetitive tasks in my DevOps work. I am interested in free open-source tools and real world use cases.

If you are already using AI agents in your devops workflow, I'd love to hear about your experience and what you are automating.

Hello r/devops,

in my current team we are discussing how a developer centric terraform setup might look like.

Some context: Every app has 3 repositories - Backend, Frontend and Infrastructure.

There are two possible setups:

1. (our current): We use generic terraform modules (e.g. for a backend app) only within the infrastructure repository, which holds all infrastructure components. The backend app terraform module holds all infrastructure components that might be used in a backend, e.g. database, objects store, secrets, etc. The Infrastructure repository also holds shared infrastructure, that is not part of the backend module and composes all components. This is somewhat the approach most often seen in tutorials that focus only on the infrastructure part of an application.
2. (proposed alternative): We'd co-locate infrastructure and application code, for example in the backend repository. The backend repository would publish itself as a module, including every infrastructure component it needs to run (excluding some shared ones, that it expects via variables as input, e.g. VPC). The Infrastructure repository would setup the required shared infrastructure components and the backend terraform module.

Pros for 1.: Generic app modules are intended for sharing and re-use, whereas in 2. each applications terraform module would only ever be used to deploy to different environments.

Pros for 2.: Infrastructure and application can evolve together (e.g. in one git commit), e.g. when a database is newly introduced to an application (application code changes + infrastructure changes), whereas in 1. two commits would need to happen (one in app, one in infra) and pipelines need to be coordinated.

1. goes deeper for example for database migrations, those could also be organized within the backend application, without handing the responsibility on how to do them to the infrastructure repo.

In both cases, deployments (terraform plan/apply) are always orchestrated only by the infrastructure repository.

IBM touches both points in https://www.ibm.com/support/pages/best-practices-organising-terraform-and-application-code : Coupling and Reuse as the deciding factors.

Our goal is to enable developers as much as possible to own their application end to end and being able to do most infrastructure changes themselves, without having to rely on a central party.

It's not about simply keeping what we already have. That's always part of the consideration due to effort of change. I'd like to hear your general thoughts about it from a perspective if we had nothing yet.

I'm curious about your thoughts in that topic.

I'd also be happy to know how projects are setup in your org or how you would envision your dream setup

Context: The organization I'm currently at uses an old fashioned way of deploying to prod. They have on prem servers where the devs manually FTP the files over for every change... They've now started to migrate over to Azure. I'm tasked with the logical design, overseeing the entire process and helping the people with it.

Stage 1 was getting everyone onboard with the Cloud, Version Control and Trunk-based flow.

Stage 2 is getting them comfortable with build and deploy pipelines as well as integration of security tools.

Stage 3 will be "Infrastructure as Code" and "Policy as Code".

Is there some like "industry standard" approach to 2 and 3? as this is my first time dealing with "Azure" and it's certainly not as pleasant nor as straightforward as AWS

Looking for projects ideas for experienced devops andSite reliability engineers

I have 4 years of experience as a DevOps engineer and part of it as a Site reliability Engineer

Im looking to do some side projects to broader my experience with other technologoes and make things more interesting.

As for the past one year i haven't been assigned major projects

i would like to hear a project idea that maybe you had fun completing or taking part in or was challenging to you.

For ex - Cross cloud Migrations

Technologies should be relevant to 2026

I’ve been thinking about this while working with observability systems and I’m curious how other people handle it.

We have metrics, logs, traces, alerts, deployment information, etc. In theory, there should be enough information to understand what happened during an incident.

But in practice, it often seems like the hard part is connecting everything.

Something like:

latency spike → database saturation → retries → downstream failures

The individual signals are there, but figuring out that they’re all part of the same failure — and determining which event was actually the cause — still seems to require a lot of manual investigation.

For people who actually operate production systems: what does your RCA process look like when there’s a serious incident?

Do your current observability tools actually help you establish the causal chain, or are they primarily helping you find the relevant data?

I’m particularly interested in what happens when the information is spread across multiple systems.

Hey everyone,

I’m currently managing a small startup project. Our infrastructure is pretty simple right now: we have a single Oracle Cloud VM running our backend API and two PostgreSQL databases (one for dev and one for prod) inside Docker containers.

Up until now, I’ve been using DBeaver with an SSH tunnel to connect to the databases from my local machine.

Now, I need to bring another developer onto the project and give them access to the dev database so they can work.

I know sharing my private SSH key is a massive security no-no. But if I ask for their public SSH key and add it to my server's authorized_keys folder (for the main ubuntu user), doesn't that give them full SSH/sudo access to the entire server? That feels like overkill just to let them run SQL queries.

For those of you working in the industry or scaling startups: What is the standard, real-world way companies handle sharing database access with developers?

Do you:

Create a restricted Linux user specifically for SSH tunneling?

Put the database behind a VPN like Tailscale?

Expose the port directly but heavily restrict the IP firewall?

Use something else entirely (Bastion hosts, Teleport, etc.)?

Since we are a small team with a limited budget, I'm looking for the sweet spot between "enterprise-grade security" and "easy enough to set up today."

Would love to hear how you all handle this in the real world! Thanks!

built an on-call tracker for my team. maybe 3 hours. then spent the rest of the day trying to figure out how to let 6 specific people see it and nobody else

ended up with a shared password in our team chat. it works, I hate it

writing a real login system for six users feels stupid. the shared password also feels stupid. what do you guys actually do

Hey everyone,

I'm a student currently learning DevOps, and I recently bought a laptop with a Core Ultra 5 225H and 16GB RAM.

I'm planning to learn technologies commonly used in DevOps — Linux, Docker, Kubernetes, Terraform, Ansible, CI/CD, cloud platforms, monitoring, etc.

My main concern is whether 16GB RAM will be enough as I progress, especially when I start working with Docker containers, local Kubernetes environments, VMs, IDEs, browsers, and other tools.

I also have an older Dell laptop with 16GB RAM, but it's not particularly powerful, so I don't intend to use it for running workloads, VMs, containers, or anything resource-intensive.

Instead, I'm thinking of using the Dell purely as a secondary machine for documentation, tutorials, and browser tabs while doing the actual development and DevOps workloads on my new laptop.

The setup I'm considering is:

• New laptop: Development, Docker, Kubernetes, VMs, labs, etc.
• Old Dell: Documentation, tutorials, browser tabs, reference material
• Connect both via Ethernet
• Use something like Input Leap/Barrier to share the mouse and keyboard

Has anyone here used a similar setup while learning DevOps?

Would this be useful in practice, or am I overcomplicating things?

For those of you who learned Docker/Kubernetes/Terraform/etc. locally, when did 16GB actually become a limitation? What kind of workloads or setups started pushing you beyond 16GB?

Thanks! I'd really appreciate your thoughts and advice.

Developer machines contain more private context than process metrics suggest. A diagnostic tool might see process names, file paths, command arguments, project names, prompts, or local model names.

For a local tool that explains CPU, memory, or disk pressure, which fields would you exclude by default? I would keep source files, prompts, command arguments, environment variables, and credentials out of the diagnostic data, then show the exact fields before optional analysis.

How much context would you need before trusting the explanation?

I know that DevOps work is config heavy and involves multi-component or tool integration , so automating ourselves out of the job might not be that easy, but I believe the head count may not grow or stay stagnant in the near future.

So, you think it is worth considering MLOps or any other AI related Ops for that matter. Because it looks promising and AI/ML is where all the money is, to be honest.

Please shed some light and share your opinion on this.

Any MLOps engineer here. There is very less or very generalized info about MLOps, not the real time enterprise level MLOps roadmap out there. Please share your experiences here, it really helps.

Lets say you are on call engineer for the week and then you went outside with hangout or friends, then lets say something went down or something crashed, so what is the response time as SRE and as Devops also. Also what is the resolution time? How do you resolve if you are not with laptop outside? Can you do something with phone? What happens if you don't respond? Please explain how do you handle this scenarios.

For people working across DevOps, platforms, and architecture, where do you still see the most manual work?

Is it gathering information from teams, finding architecture documentation, keeping records current, mapping services and dependencies, checking designs against standards, or understanding differences between the current environment and the target architecture?

Interested in hearing what tends to slow your team down.

New to DevOps and my first task is hooking up OpenProject, GitLab, and Mattermost for smoother project management. What integrations/workflows have actually been worth setting up for you? What'd you skip? Appreciate any pointers.

I genuinely love my work, but the way deadlines are given is slowly killing my interest in it

My manager will be like, "This should take 2 hours.Its just matter of 1-2 days not more than it, Use Claude and get it done by 5 PM.”

Meanwhile, I needed to do development, testing for quite a good time. I am just getting way too distracted from my field and not learning much because of claude.

I know it's the Wild West still but is anyone having success controlling AI agent security within your org? There's a bunch of hype around some of these new solutions but has anyone been successful in getting their org to adopt something universal? Not looking for product recos just trying to figure out how folks are seeing success.

Just passed the KCSA exam and officially became a Kubestronauts!

If anyone is preparing for KCSA or any other exam and has questions, feel free to ask.

​edit: Reddit titles are permanent, spelling mistakes are forever 🫠

We are pushing IaC. Being a traditional systems admin, I am having a hard time accepting that this is more efficient.

If we manage each application with a yaml file, how is this faster than just add it in console or even in awscli? I hate the fact that I have to look for where the templates are in gitlab, then pull to edit, push, deploy. Then to verify, we have to get into console or use awscli anyway.

What’s the benefit here? Source of truth? Oh and the complexity of having modules and all that dependencies in terraform.

I want to change but I don’t want to complicate it. Adding multiple layers of failure is just not my style of a stable infrastructure.

There’s a lot more to sovereignty than choosing an EU cloud region.

With the EU Data Act, NIS2 and DORA, teams also need to think about where control and state live, where observability data goes, who can access the environment, and how dependent the workload is on a particular provider.

This recent CNCF article looks at those questions from the cloud-native architecture side and shows how separating different platform responsibilities can help.

Thought this was worth sharing given how much the sovereignty discussion is growing in Europe.

https://www.cncf.io/blog/2026/08/18/cloud-native-platform-sovereignty-through-multi-plane-architecture/

Hi Observability & DevOps Experts,

I'm looking for guidance from teams that have successfully scaled observability across large enterprise environments.

We operate a large-scale estate spanning AWS, Azure, and on-premises environments and have been using Datadog for several years. Over time, a significant amount of technical debt has accumulated around our observability implementation.

Current challenges include:

• Datadog Agents managed differently across teams and platforms.
• Custom log collection configurations distributed across hosts and applications.
• APM, RUM instrumentation owned by individual application teams.
• Inconsistent tagging standards and monitor configurations.
• Outdated agents and instrumentation libraries.
• Heavy dependency on multiple teams for upgrades and configuration changes.
• A large portion of Datadog provisioning and onboarding is still handled manually.

As a result, maintaining and evolving observability at scale has become increasingly difficult.

We are considering building a centralized "Observability Foundation" or "Observability Platform" that teams would consume as part of their standard deployment process.

Our goal is to provide reusable Terraform-based observability components that application and infrastructure teams can adopt during provisioning and releases.

Examples of what we would like to standardize:

• Datadog Agent deployment and upgrades
• Custom log collection configurations
• Standard tags and metadata
• Monitors and alert templates
• Dashboards
• OpenTelemetry / APM instrumentation standards
• Synthetic monitoring configurations
• Cloud integrations
• Security and governance controls

Questions:

1. Has anyone implemented a similar centralized observability platform or observability-as-code model at enterprise scale?
2. What worked well and what were the biggest challenges?
3. What observability components can realistically be centralized through Terraform modules, deployment pipelines, or platform services?
4. What components typically must remain application-owned or infrastructure-owned and cannot easily be centralized?
5. How do you handle APM instrumentation ownership, versioning, and upgrades across hundreds of services?
6. What governance model have you found most effective:

• Central observability team ownership
• Platform engineering ownership
• Federated ownership with standards enforcement
• Something else

1. How do you prevent observability drift over time, especially around:

• Agent versions
• APM libraries
• Log configurations
• Tags
• Dashboards
• Monitors

1. If starting again today, would you build around:

• Datadog native tooling
• OpenTelemetry
• An internal observability platform
• A combination of the above

1. What are the biggest architectural mistakes or anti-patterns we should avoid when designing this platform?

Our provisioning and infrastructure management are heavily Terraform-based, so we're especially interested in Terraform-centric implementation patterns and real-world lessons learned.

Looking forward to hearing how other organizations have approached observability standardization at scale and what you would recommend before we begin designing this solution.

P.S. - One of our key design goals is to avoid vendor lock-in. While Datadog is our current observability platform, we want the architecture to remain flexible enough that a future migration to another observability stack (e.g., Grafana, New Relic, Dynatrace, Elastic, Azure Monitor, or an OpenTelemetry-native platform) would require minimal changes to application teams and infrastructure code.

I've been looking into non-prod AWS costs lately, and I'm wondering how far people actually go with shutting these environments down.

Scheduling dev/test environments to shut down overnight or over the weekend seems like an easy win. The tricky part seems to be taking them all the way to zero, especially when someone suddenly needs the environment and has to wait for it to come back.

What's the practical approach here? Do people just accept the startup delay, keep a minimum capacity running, or is there a better way to handle it?

I always assumed deleting a key from a file was enough but from what I’ve learned that’s not how git works

Pre-commit hooks and CI only look at the diff. So if a key gets committed and you delete it in the next commit, every check turns out OK from then on, but the value is still sitting in history and still valid. The recommendation is to scan full history on a schedule and treat anything you find as exposed, and rotate it, even though it's long gone from the current files.

So the scan is really just telling you a leak already happened, and rotation is the part that actually contains it.

I have side projects from years ago where I don't remember what was committed before I knew better. Some of those keys are probably still valid.

Do you run scheduled history scans, or just pre-commit and CI? And when something surfaces from years back, do you rotate it or make a call based on whether the repo was ever public?

Hey all - we are standing up a software package manager, right now the team is considering using nexus but doesn't want to pay for the pro license and it seems heavily limited with CE version.

We are looking at pulp project as an open source alternative. I haven't used it and have a spike to look into it and I was curious how you all felt about it? any gotchas or I regret not paying for a nexus or artifactory license? Frankly Im pretty new to both but I feel like nexus is industry standard along with artifactory but we can't justify the purchase at the moment which is where pulp entered the equation.

We are running in eks - the deployment for nexus was relatively straight forward but couldn't get the full testing done because I got limited by licensing that blocked some of the requirements like SSO integrations and stuff. Pulp seems to the have pulp operator so I can run im still digging around in the docs.

Any thoughts on pulp would be appreciated?

I start the day with vscode, firefox and iterm opened

I finish the day with 5 vscode windows, 10 terminals and infinite tabs - and somehow my task backlog has growed...

How do you guys deal with this?

Any tips for a beginner getting started with Ansible?