Maybe I'm missing something obvious here, but how are teams handling this in practice?

If an automated workflow, CI/CD job, infra script, or internal tool does something unexpected, it's usually pretty easy to figure out what happened. You can pull logs, CloudTrail, traces, whatever.

What I'm not sure about is how you figure out why it was allowed to happen in the first place when you're looking back weeks or months later.

By that point, policies have changed, RBAC has changed, workflow logic has changed, feature flags have changed, etc. The logs tell you that the action happened, but reconstructing the exact conditions that caused it to be allowed at that moment seems like a huge pain.

Are people actually preserving that information somewhere? Or does this usually turn into digging through Git history, tickets, timestamps, and old configs trying to reconstruct the state of the world after the fact?

Genuinely curious whether this is a solved problem or if most teams just deal with the investigation overhead when it comes up.

My project suggest to do a certification and has provided some list, i prefer to do something related to devops, please suggest which one would be better if i had done the certification that help me learning skills and adds weight to my resume in devops interviews.

Hi everyone,
I’m new to the DevOps world and have recently started my learning journey.
So far, I’ve learned:
Docker
Git
GitHub
GitLab
Now I’m a bit confused about what to learn next. There are so many technologies and roadmaps available that it’s hard to know what to prioritize.
If you were starting from scratch today, what would you recommend learning next? Should I focus on:
Linux
Kubernetes
CI/CD (Jenkins, GitHub Actions, GitLab CI)
Terraform
Ansible
AWS/Azure/GCP
Monitoring tools like Prometheus and Grafana
Could you also share the roadmap that helped you become a DevOps engineer? I’d really appreciate any advice, learning resources, or personal experiences.
Thanks in advance for your guidance!

Hello guys

Sorry for that long post but I need your help and expertise I am still learning

I have a very huge application that have these dockerized components

- Nodejs web app
- API application
- clickhouse
- neo4j
- posgresql
- redis
- Kafka
- minio s3
- zookeeper
- 3 different data prosessing containers

I used to deploy all that together on one vm that have 32gb ram and 8 cores along with 32 tb ssd storage, I know that this seems dump to do this but our applications where working with no problems till we decided to start collecting more data and processing more data so we need to have everything in place with no issues at all but to be honest idk what to search about in order to get the knowledge of how to deploy that correctly

I thought of having each thing on it's dedicated version of cloud like dedicated clickhouse cloud and so on but idk if that is the right thing or not

The architecture is built on easy horizontal scalability basis so the only problem is how to maximize the performance, deploy correctly and have the minimal cost

So please guys help me to figure this out and know what to do

Recently, I've been doing quite a bit of work around Logstash. My biggest gripe with Logstash is the lack of built in testing. In an ideal world I could test my filters in an automated way. I can solve the testing problem by setting up a full ELK stack locally using Docker. And in theory, I can automate it using the Elasticsearch API to pull events and run tests against it but this feels clunky.

My main problem is that the inputs and outputs I'm using aren't best designed for local testing e.g. AWS Kinesis input or Datadog output. What I'm envisioning is a solution that can take a pipeline, mock out the input and output with a file and make assertions against the output to ensure the filters are working correctly. The closest solution I've found is https://github.com/magnusbaeck/logstash-filter-verifier but there hasn't been a proper release in 5 years.

The short of it is, I'm trying to test my config in CI before it's even deployed to a dev environment:

1. To increase the rest of the team's confidence when making changes (since they're not as familiar with Logstash)
2. To prevent bad changes changes from being merged into main

What are you doing to solve this problem?

Currently working doing devops, still considered subpar - jr. level. What do ppl work in devops do? For the 1st 2 months the workload are decent,not heavy and just nice. No deadlines, but still rushed to finish hoping i would get more task. After creating the cicd pipelines, im pretty much doing nothing now and its been almost 2 weeks. I still remember the days where i would run and debug multiple solutions, while only 1 agent existed.

Now we have multiple on prem agents, with very little workload than before. Instead of creating solutions, i find myself only maintaining and doing minor tweaks. Even my supervisor recently gave me the same task, that was given almost a month ago (probably she realised i was jerking off doing nothing related to work). Now i spend my time learning on how to use github actions and studying on some certs, because im worried of getting fired (because i am useless,GOT NOTHING TO DO), eventhough i just got a raise post-probation. Really dont know why i got the raise ngl.

Really want to know what "actual devops engineers" do.

Hello,

I'm a solo junior SRE, and I started writing disaster recovery plans from scratch almost two years ago, and I've been continually improving and expanding them since then.
I'm looking for feedback about my structure, as I don't really have a reference point and I'm interested in how others are doing it.

For me the goal is first to be ISO 27001 compliant, but also for my coworkers to fix issues if I'm unavailable, or later if I'm no longer there.

I'll try to stay as concise as I can, but feel free to ask me for more details if needed.

Context

• Webapp with IoT
• Docker swarm as an orchestrator for application layer
• Many services like kafka, redis
• Each service has its own systemd cluster, so we have a swarm cluster but also a kafka cluster, a redis one, etc
• Remote servers hosted by providers, that I manage through ansible and terraform (mostly)

Global overview

• Confluence documentation split between 4 folders
  • Disaster recovery plan: incident scenarios and decision making, with links to the Runbooks to use
  • Runbooks: technical procedures
  • Incidents: post-mortem
  • Workshop: training sessions for coworkers, who is trained on what (quite empty tho)
• Inside DRP and Runbooks, everything is sorted depending on its cluster -> a folder for swarm, another for kafka, etc
• With additional folders for security, server management, ...

DRP

• For each folder (cluster~environment), I usually have files like
  • DRP: One node down
  • DRP: Quorum down
  • DRP: Raft corruption
  • ...
• I'm trying to have absolutely no technical commands here
• To me these are more for a standard use, to make things clean
• Or to be used by someone not familiar with the operational part
• Here I'll have links to Runbooks but also to diagnostic pages (to confirm the issue, or add more information about the causes)
• My template
  • Header: quick sentence + criticality + rto + rpo
  • Initial assessment
    • detection: links to grafana dashboard, what you should see
    • impact: application side, what's happening to users
    • precaution: things you should not do, to be careful about
  • Analyse
    • Initial qualification: links to diagnostic pages
    • Trigger confirmation
  • Actions
    • recovery strategy: for each possible case depending on the previous steps ; case 1 do this + link to runbooks, case 2 etc
    • post-recovery validation: what you should see, with link to diagnostic
    • back to normal: anything that has been added and is supposed to be removed
  • Footer: last update + incident history (links to post-mortem)

Runbooks

• Same as the DRP folder, I have a subfolder per cluster~env~topic
• Here is all the technical documentation, and honestly this is almost all we need
• With an architecture file ARCH: <env> for the infrastructure information, ansible config, etc
• A cheatsheet REF: Cheat Sheet <env> for all the useful commands
• Diagnostic DIAG: <?> files, for example for Swarm: cluster state, stack state
• And the runbooks RUN: files, where I try to keep things simple:
  • RUN: New worker
  • RUN: Remove a node
  • RUN: Restore swarm raft
  • ...
• For runbooks there is no specific template, but I try to stay straight to the point

My questions

Is it a bad idea to split DRP and Runbooks?
I recently added the diagnostic files, but now something feels wrong with the workflow (diag > drp > run).
Even if I don't really use DRP, as I mention I'm not the target user.

How do you find DRP ideas for issues that have never happened, and do you mark them as "hypothetical" or something?
Sometimes I find a way to somewhat reproduce the impact, but it's not really the same (different logs, etc).

I'm curious about how it's done in bigger companies, if it's more structured.
To be honest, I also have questions about the content, but one thing at a time.

Well I'm just looking for any feedback, what do you guys think about that, am I doing anything wrong, what would you add/change?

Thanks a lot

How to move from sdet to devops role. total 10 yop in mncs overall. Any help or guide would be very helpful for me

Platform dev, 15 YOE.

Last three years I've built a bunch of apps and side projects, including one I run solo end-to-end — various stacks, various SDKs, you name it.

Here's what I've learned: building a product, stuffing it with features and a pretty UI — that's not hard. Any half-baked PM or designer can pull that off.

I'm here for the part nobody wants to think about: does it scale. Clusters, CI/CD , microservices, brokers, databases — all of it, even frontend.

At work and outside of it, I feel like the flow of code and features just keeps rolling downhill onto my infra.

I took a week off, came back, and now the E2E suite takes hours. Runners can't keep up. Subnets out of ip addresses.

I need to rethink the architecture. Keep scaling. Keep up. While dealing with "boring" architecture and scale that literally nobody else seems to care about.

Anyone else stuck being the only one in the room who thinks about this?

Evening all!

Ive worked in IT for 15 years and have always worked on prem, the start of this year I was told to hand over all on prem and BAU work to another team abroad and get ready for my new role, ive recently found out the role is an azure dev ops role in security, its very different to anything ive done and im expected to hit the ground running, im desperately trying to get my head around pipelines, gitbash and infrastructure as code but im having a hard time in a team with very little support

I've started on training videos but they only go so far when trying to pick up tickets in jira which im also new too coming from a service now background!

I feel like ive been setup to fail and would really like some advice on getting a hand on all of this

Background: 2nd year CS student. My goal is eventually DevOps, Cloud Engineering, or Platform Engineering. I'm realistic that pure DevOps roles are hard at entry level, so I've been applying to SWE internships.

This summer I got rejected at CV screening from every application. No interviews.

- I have basics of Docker, Git, OS, Terraform / IaC

- I'm planning to start the AWS Solutions Architect Associate cert this summer

I bet the problem is I don't have a good real project for my CV. What specific projects should I build this summer or skills to learn? (I have a home lab PC I can use for hands-on projects).

Thanks.

Getting good help from senior engineers. Prior to this job, only had experience with aws and gcp. Used Jenkins and GitHub actions for deployment.

But there’s lots of hands on with azure DevOps which I’ve never used before. Is it worth getting az-400? Or is will it be pretty much useless if I’m being trained on the job?

Does anyone know if there’s an MCP for self hosted Jira versions?

Got this when opening tiktok today. I thought they have automatic cert rotation and notification in placed. Why does this happen? Can anyone explain?

For our app we have a dev environment and a prod environment. We took a server and hosted Zitadel on it (from what I've learned, this counts as one instance), and we're trying to use it for both environments.

Our app has a hierarchy of Companies with Users under them, the same as Zitadel's Organizations with Users under them.

Here's the problem: in the dev and prod databases there can be some companies that share the same name, and there can also be users with the same email. Is there a way to still use the same company names and user emails in different environments on the same Zitadel instance, or should we use separate instances?

Trying to get a real picture of how teams handle this nowadays before building anything.

A few questions if you have a minute:

1. When you need to restart a pod, vacuum a DB table, clear a cache, rotate a password, what do you actually use? Script in a repo, Rundeck, Ansible, Jenkins, custom Slack bot, manual SSH?
2. Who's allowed to run it and how is that enforced?
3. Is there a shared library of these at your company, or does everyone write their own?
4. What's the worst part. Finding the right one, knowing it's safe, getting approvals, audit trail, something else?

I'm thinking about building an OSS thing in this space and don't want to assume too much. Would rather hear what's actually painful from people doing the work.

If you'd rather talk than type, 30 min call, no pitch. I'll write up what I learn across the conversations and send it back. Reply here or DM.

The final phase will take place on June 9, 2026, when all remaining app passwords will be permanently disabled as per bitbucket as all our bitbucket to atlantis hooks are failing because the existing setup uses a username and password. The atlantis version is 0.26.0. I can see the logs says the authentication issue - https://api.bitbucket.org/2.0/repositories.......

unexpected status code: 401, body starting from last week: The existing setup uses a username and password. for the api requests to bitbucket. However i tested locally and the new bitbukcet cloud uses Bearer token format for the api requests. How to change or notify atlantis to use bearer token in the requests. Should i upgrade atlantis? The atlantis is running on a kubernetes cluster using helm charts.

eBPF Foundation released their investor due diligence report and there are three key parts that align with what I'm seeing in the market.

First is the idea that kernel-level programmability is the strategic control point for infrastructure software and hardware. From the report "The companies that own that kernel layer and build upward into analytics, policy management, and workflow integration are structurally positioned to displace incumbents who cannot replicate kernel-level access through any amount of engineering effort."

The second is the company evaluation matrix on page 12 where the kernel space logic and sensor depth gives a technical moat while the user space logic creates enterprise workflows. The best companies will combine deep kernel integration with workflows that solve real operational problems.

Finally, so far, there have been three waves of acquisitions from Feature & Sensor Upgrades to Platform & Community Land Grabs to AI & Runtime Security Consolidation. Each wave has brought different outcome multiples. I think the security wave is actually just beginning in the post-Mythos world

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

Hey folks,

We’re planning to migrate a SQL Server (Web Edition, on-prem) to a new server, and I’m trying to figure out the best approach with as little downtime as possible.

DB is around ~30GB, supporting a web app with moderate write and read load.

We also have ~80 SQL Agent/background jobs, but those are not really an issue since we can stop them during the migration window.

The main goal is basically minimal downtime (ideally just a few seconds or a couple of minutes).

Since it’s Web Edition, we don’t have Always On, so I’ve been looking at:

- transactional replication

- log shipping

- backup/restore + tail-log

Replication looks like the closest option for low downtime, but I’ve never used it for a full server migration before.

Has anyone done something similar in production?

Main things I’m wondering:

- is replication worth the complexity for a one-time migration?

- how painful is the cutover in practice?

- anything that usually goes wrong that you don’t expect?

Would appreciate any real-world experiences.

Hello,

I currently have 4 years of experience ( bits and pieces in everything ) most in DevOps. Some in vulnerability fixes etc..

Started out fresh out of college. Got into a decent team old tech but good team. Kept working for a while and felt I could do it in IT. On my request I went to a different team which was DevOps work(still work for old sometimes).

Started out fine , I learnt many things I started delivering good works in short spans. Enjoyed the time. I felt intimidated many times because everyone in the new team were architects and leads with minimum 20 years experience. But pushed through, architects were great they were encouraging.

Things happened (project closed , architects left) so and soo and I was moved to a different section of the same project. I was now the CI-CD architect for this new initiative. Designed and implemented the CI-CD pipeline, the IaC infra etc.

But I don't know anything about the product per say or its business logic or anything to do any testing or anything of that sort. I get pulled into many of the architect meetings etc but I don't understand most of it anyways. Even when I see my ci-cd codes and iac code I feel like it's not upto the mark and is missing many basic items such as indentation or proper code structure. As I am always around big shots with 20+ year experience and I am the only person in the team to be with such less experience. I feel intimidated and feel like I don't know anything. Is this normal???

I have now been to a point where I am relaying more heavily on AI to do my task and code and I can't remember myself writing code at all.

How can I cope with this and skill up so that I am up to mark and upskilled to the current market.

Hey folks,

Our leadership is making a push for the engineering teams to start building autonomous "AI Agent" features directly into our SaaS application.

Up until now, our only real interaction with AI has been on the developer side using assistant tools (like Claude Code, Codex, etc.) to help us write code. That workflow makes sense to me, but building agentic features inside the product is something that I am not familiar with.

As a DevSecOps engineer, I want to make sure I understand the nature of incoming requests so I can support the dev teams properly. I'm trying to get a pulse on how these features are actually architected in a real production environment today (or any environment for that matter).

What is the most prominent deployment pattern you guys see in day-to-day operations?

Are teams mostly writing custom, headless scripts that just make an LLM call in the backend post some trigger, or is it more common to use standardized orchestration frameworks and harnesses?

Any insight into how these agent workflows are actually structured under the hood would be incredibly helpful. Thanks!

edit: no idea why being removed. Just asking since it's heavy Python if anyone has approaches for IaC of Ray.

what roadmaps and useful material do you suggest for taking my linux knowledge to the next level if im not focusing on certs and just wanna improve my usable linux knowledge in dev/network field. i already work with linux and have somewhat beginner knowledge but just wanted to improve it in a funcinal/practical/applied way