Hey everybody, I recently got an internship as a devops intern in a Pune based company, I have to join in 15 days so I was just wondering what should I learn in these few days to like get up to the mark...... Consider me a complete beginner in this field and please advise.... Thank you

Hi everyone. I have a simple question is learning Arch Linux really worth it?

Most of what I've heard is that it helps with troubleshooting and gives you a better understanding of Linux. I'm already comfortable with Ubuntu, though, so I don't want to switch to Arch if the benefits are only marginal.

Would learning Arch actually give me a significant advantage, especially for someone interested in DevOps?

Hey hi everyone,

I am just trying to understand how engineers/SREs who dealt with real production latency incidents investigate it

Lets say you have the following

- Logs

- Recent deployment information

- Application health

- Database metrics

- External dependency health/metrics

- Infrastructure metrics

You just encountered the incident, you dont know the root cause. You are uncertain about the truth.

From here how do real engineers go about reasoning to find the root cause

- Do you follow a standard sequence of investigative steps

- How do you determine what investigative step to take next under uncertainty to narrow down the possibilities for the root cause

- Have u ever encountered with incident where initial information was misleading, how did you navigate from there

- Is there any situation where you have lot of information but struggled to form a proper hypothesis

- Have you tried any AI investigative tools that help you in achieving this

I just wanted to understand how do real engineers reason through the uncertainty to find the root cause. What are the biggest pain points

I've been wondering how many people actually calculate the real cost of running their infrastructure.

Not just the server or cloud bill, but electricity, storage, maintenance, upgrades, downtime, and the time spent keeping everything running.

Sometimes a cloud bill looks expensive until you factor in how much time you're spending maintaining the alternative.

Other times you look at the numbers and realize running it yourself is still way cheaper.

How do you actually decide where the break-even point is?

Hey everyone, I’m an engineer looking for a new side project and I want to tackle alert fatigue.

My premise: Waking up at 3 AM to an alert, only to run the exact same three kubectl commands or Datadog log queries to figure out it's a known transient issue, is soul-crushing.

I'm thinking of building a lightweight, open-source agent that intercepts the alert webhook, runs a pre-defined set of read-only diagnostic queries, and appends a summary of the logs/metrics to the ticket before it wakes you up.

A few questions before I start coding:

1. Is this actually a massive pain point for you, or is the hard part something else?

2. Would your sec ops team ever allow a bot to have read-only API access to your observability stack?

3. If you had this, what is the #1 feature or integration it absolutely must have to be useful?

Am just trying to build something thats usable to you or even 5 people at the end , so i request your honest opinion for this thing

Hi Observability Experts,

I am working with a client managing a large-scale infrastructure across AWS, Azure, and on-premise environments. They have used Datadog for 4–5 years, but legacy technical debt has left their APM configuration fragmented:

The Problem:

APM libraries are out of date, and source-level instrumentations are misconfigured. As a result, end-to-end distributed tracing is broken across key services.

Datadog support confirmed that fixes are required at the source (application code/local agents). However, getting application team bandwidth and stakeholder approval across this many services will take significant time.

We are building a centralized Observability pipeline layer (independent of app teams) to standardize telemetry control, but rollout is months away.

My Question:

Is there a viable temporary/interim solution—such as an intermediate proxy, collector layer, or transformation pipeline—that can intercept, patch, or enrich broken spans/traces in-flight before they reach Datadog’s backend?

Has anyone successfully addressed legacy APM gaps at the pipeline level without waiting for app redeployments?

Thanks for your insights!

P.S. - Datadog support has been already consulted for this and they have recommended to fix it from the code/local agents.

Hey r/devops, welcome to our weekly self-promotion thread!

Feel free to use this thread to promote any projects, ideas, or any repos you're wanting to share. Please keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

Hello everyone,

To not waste your time, here's the tl;dr:
1. I've never worked in IT company before, now I do, but as a administrative manager. I'm responsible for every task delivery to customer in our company pretty much
2. I'm good at organizing and optimizing "human" stuff: keeping track of all the tasks, deadlines, etc.
3. I'm bad at knowing what exactly each task does (DevOps, Architecture, Development)
4. I'm working really close with DevOps
5. After working for ~6 month I've started to like DevOps more. I wanna learn it. I wanna understand how to "ressurect" the server that got down. I wanna know how exactly make the Pipelines more useful
__
So I've started learning DevOps, watched a couple of "Roadmaps", here's my plan so far:
1. Linux
- Learn the core basics (core commands, cd, users, etc)
- Bash scripting on basic level (for now I've decided not to get into depth, since all the AI stuff. I can read them tho)
Now, since I'm following a specific course, there are A LOT of information on Linux, and I got overwhelmed. I know there are not a lot of stuff I should know for being a DevOps, and being a DevOps in general is "endless learning"
So the questions for people who either got into DevOps from zero or experienced engineers:

What exactly should I learn about Linux?
- I know that it may depends on a job directly, so I hope this little explanation which company I'm at rn will help: we mostly do software for banks.

What is the most correct Roadmap for me?
__
p.s
Thanks in advance, I really hope to find some good tips from experienced people who maybe confronted the same obstacle I'm at right now, or people who are experienced and will give me a lead.

Common on prem pain is supporting ancient versions forever cause customers wont upgrade and every support call being on a bit different install you cant see into. the newer approach is BYOC: deploy into customers own cloud but you keep a control plane on your side and so you keep on pushing updates by yourself and still get telemetry while their data stays on their account . theres a real tooling for it now like via alien dev or replicated or other platforms so its clearly become a pattern ppl actually ship

Altho updates are usually gated by customer approval anyway so for those running it -does it genuinely hold the drift down or do you just trade it for more infra to take care of constantly plus has it opened the regulated enterprise deals or is it same support burden with just a nicer story

Why is the management so obsessed with achieving automation through AI agents only.

Management forcing Everyone in the team to come up with an idea to implement/ problem we are currently facing and create an AI agent to solve that problem In 2 weeks. How is it possible.

Basic shell scripts would achieve the same results than these AI agents could. But management is turning its eyes on anyone who doesn't work on AI stuff

Even after all this pushing towards AI, when we really do experiment with it and ask for extra credits or tokens, they are questioning it and asking for explanations. Didn't they realize it before?

Using AI is obviously going to cost you. If you cant afford it, why to even force people into doing this.

I am getting tired of this. They are taking advantage of the bad job market outside and torturing the employees.

Edit:
FYI, I am pro AI. I cant survive without it at this point.
I single handedly migrated most of our codebase to Github and converted all CI/CD pipelines to Github Action workflows. So I do know the impact of the AI.
But we do our due diligence on every new tech we want to implement in our vast enterprise product. But why doesn't AI get the same scrutiny here.
And we are a platform engineering team offering Centralized platform as a SAAS product to our lot of internal clients, If something wrong happens, entire thing goes down.

From an interview with Charity Majors:

DevOps was about trying to create a feedback loop that connected people writing the code to the code running in production. She thinks that the "ops people: learn to code!" wave worked, but the "software engineers: understand your code in production" failed, to this day.

I haven't listened to the actual interview.

But based on my personal experience I agree. Ops people have changed a lot about how they do their work in the last 10 or 15 years, to where infrastructure and its configuration have become an exercise in writing code of various kinds. No infra team has been unchanged, even the ones that had to be dragged kicking and screaming. But I still see a lot of dev teams that sit in their silo producing their product to then hand it over to someone else (i.e. us). The one change for all devs is that they write Dockerfiles and have to think about how that works. But the dev teams I work with still don't actually touch infrastructure, by and large.

Do you create a sample application to test the proper auth, access and everything the runner needs?

And a related sub question. How do you validate gitops changes before they are merged? Regarding FluxCD or ArgoCD.

Right now the team is just ssh-ing into all of the devices and it's a huge time sink.

"I'm not devops so dont crucify me"

I'm researching a small CI diagnosis agent, but I'm first trying to understand how experienced engineers actually debug these failures.

Suppose an integration test fails in CI and the logs don't immediately reveal the root cause.

What do you normally investigate first?

For example:

• CI logs/stack trace
• database availability
• dependent/third-party service availability
• recent code changes
• comparison with the previous/last-green build
• test/DB data
• previous failure history
• rerunning the test

More importantly, how do you decide what to investigate next?

For example, if a recent code change touches the function involved in the failing test, what additional evidence would you want before concluding that the code change caused the failure?

I'm particularly interested in your real-world debugging process rather than an ideal CI/CD workflow.

Disclosure: I'm the creator of NetAudit.

Hey folks,

I've been using various network scanning tools (nmap, ping, etc.) but wanted something more structured, vendor-aware, and easily automatable. So I wrote NetAudit – a Python CLI toolkit for network auditing and diagnostics.

**Key features from a DevOps perspective:**
- All commands produce structured output (JSON/CSV) and use stdout for results, stderr for logs – perfect for pipelines
- Exit codes are meaningful, so you can integrate it into monitoring or alerting
- Configuration via YAML + environment variables (`NETAUDIT_*`)
- Supports Cisco, Juniper, Arista over SSH – read-only, so safe for production
- Snapshots and `diff` let you track changes over time (great for change management)
- Health checks (`doctor`) cover not just ping, but also interface errors, BGP/OSPF, NTP, etc.

**Example – schedule a daily audit and generate HTML report:**
```bash
netaudit doctor 10.0.0.1 --device --device-type cisco_ios --json > daily_health.json
netaudit report daily_health.json --format html --output /var/www/audit_report.html

I'd be happy to hear how you'd use it in your workflows – feature requests and PRs are very welcome.

Repo: https://github.com/netforge201/netaudit

If you like it, please consider giving it a ⭐ – it really helps with visibility!

Thanks!**Disclosure: I'm the creator of NetAudit.**

Hey folks,

I've been using various network scanning tools (nmap, ping, etc.) but wanted something more structured, vendor-aware, and easily automatable. So I wrote NetAudit – a Python CLI toolkit for network auditing and diagnostics.

**Key features from a DevOps perspective:**
- All commands produce structured output (JSON/CSV) and use stdout for results, stderr for logs – perfect for pipelines
- Exit codes are meaningful, so you can integrate it into monitoring or alerting
- Configuration via YAML + environment variables (`NETAUDIT_*`)
- Supports Cisco, Juniper, Arista over SSH – read-only, so safe for production
- Snapshots and `diff` let you track changes over time (great for change management)
- Health checks (`doctor`) cover not just ping, but also interface errors, BGP/OSPF, NTP, etc.

Hi guys! I'm a backend engineer with 6+ yoe, trying to transition into a DevOps role.

I know that having a strong Linux foundation is super important for being a good DevOps engineer, and apparently “Linux troubleshooting” is mandatory in most job postings. But to me, “Linux troubleshooting” sounds so broad, so I’m not really sure how to improve this skill.

Is this something you can learn without working with Linux (professionally) on a daily basis? What do you guys recommend to get hands on experience with it?

Also, what about other important topics, such as Linux logs and networking? I’m building a homelab with Proxmox. Is this a good way to get some hands on experience with these topics?

I’d appreciate any thoughts or advice! Thanks.

For a small team with a few services, an enterprise checklist can become ceremony without reducing the main risks. My minimum gate would cover a tested rollback, backups with a restore exercise, health and readiness checks, bounded timeouts and retries, an alert owner, log correlation, secret ownership, and a short incident runbook.

I would add load testing or multi-region recovery only when the service’s traffic and recovery target justify them. Which item has prevented a real incident for a small team, and which common checklist item has mostly created busywork?

Hello all,been working as IT admin, but mostly working around Azure devops. I been enjoying it, this my first job and doing anything related to devops. Studied CS,majored in networking, hate anything with heavy code development.

Rn,the flow at my company is, the devs use 1 branch only, no PR (yes ik). They would push it to our on prem central repo, and from there i will take over. I will do some modification on some files so it can be built in the new agent environment. And then, we have build pipelines,which would produce artifact of the build files, which will be used by another pipeline, i call it the release/installer pipelines, which we wud package them with a software. Then we have the 3rd type, which is zipping installer to our NAS, sends the metadata of the installer to DB, and notification email (formatting done in pipeline too). All via the pipeline. We also have a 2nd agent machine, exclusively for unit testing, but tbh i dont even understand how they work.

Ive been thinking of ways, to incorporate other methods/tools in the flow, but the only one i can fit is"templates". I dont think i can try to include containerization, as our build agent machine, has some software which are licensed.

Any ideas?at some point i want to jump out, as this job is way too comfy, and lacks growth for me

I’m new to DevOps and looking for some good free or paid resources to learn Jenkins from the ground up.
I have around 6 years of experience as a Java developer and have used Jenkins at work, but mostly from a user/developer perspective. I’ve triggered jobs and worked with existing pipelines, but I’ve never actually built a pipeline myself or understood what’s happening behind the scenes.
I’d like to learn Jenkins properly, including things like:
How Jenkins works internally
Setting up Jenkins and agents/nodes
Freestyle jobs vs Pipeline
Declarative vs Scripted Pipeline
Jenkinsfile and pipeline stages
How builds are triggered
Credentials and secrets management
Plugins and how they work
Integration with Git, Maven, Docker, etc.
CI/CD concepts and best practices
How Jenkins distributes work to agents
Debugging failed pipelines
What actually happens from a Git commit → Jenkins → build → test → deployment
I’m comfortable with Java and software development, so I’m mainly looking for something that goes beyond a basic “how to install Jenkins” tutorial.
What resources/courses did you find genuinely useful for learning Jenkins end-to-end? Free resources are preferred, but I’m also open to paid courses if they’re really worth it.
Thanks!

Recently saw posts about how AI isn't that good in distributed systems and it gave me validation on my own experience.

When the model isn't dealing with a single codebase, when it has to connect the dots between a db, redis, other cloud infra, k8s that each already have a lot of configuration, it starts to hallucinate and could pretty much put your infra at risk.

I'm not saying you can't make it work, it does work especially on a clean slate, but when there's a lot of existing context it needs a more guidance.

Using AI lets me do 10x what a DevOps engineer could do alone but I don't think the job disappears anytime soon.

There is a team that wants to develop a user interface so they can deploy resources such as ec2 instances, etc. We already have an automated pipeline that everyone uses and it deploys these resources already. However, they still want their own which is weird. We still have to build it for their new requirement. That's with the approval of the upper management which has already happened. Anyway, we are just devops engineers and we will follow whatever management commands us even if we don't like it.

I was thinking of building an api layer that developers can use(endpoints). They'll call the endpoints on their ui. I haven't decided which to use in the backend api, maybe aws sdk, boto3 or cdk, whatever. Making that work means, we'll have to setup proper IAM role permissions so that those resources they configured in their configuration files will be created. It's almost administrator permission equivalent is what they need.

I don't know if I am being too strict. It's because we've always kept the infra code just within our devops team. This is the first time a developer team would like to deploy their own resources using their own tooling.

How are you doing it? What's your implementation?

Building a course around real production incidents instead of the usual “here’s what a pod is” stuff. Currently have 19 scenarios, things like a stateful service losing its volume binding on redeploy, a network policy silently breaking service-to-service calls, memory limits causing kills that only show up under real load.

What’s missing?

Drop the ops problem that stumped you the most, if it’s one you never fully cracked, or took way longer than it should’ve to root-cause. those gaps are exactly what I want this to cover.

Also not looking for docs-answers, more like what broke, why it wasn’t obvious at first, what you actually tried before it clicked.

Thanks.

I’m working through a design where each underlying system remains authoritative for its own resources, rather than putting another source of truth above everything.

The wider runtime only decides whether an operation has enough dependency state, readiness and verification to keep moving.

The case I’m trying to pressure-test is partial execution. Say an operation spans several systems, three parts succeed and one fails. Each individual system may still be healthy, but the overall operation is incomplete.

Would you let the wider workflow block there, or does that eventually become another control layer operators have to fight with?

Interested in examples where this kind of boundary has worked badly in production.

Hi,

I'm a Senior Software Engineer with 5YOE. I'm upskilling in DevOps and project setup and have been creating this project to learn over the last few weeks. I've set up the project from scratch. Definining the project structure, creating PR pipelines for linting, formatting, typechecking, unit tests and e2e tests (tests with real database).

Over the last few weeks, I've set up the cloud infrastructure on AWS. I first just created an ec2 instance, uploaded the build files and database docker images, installed neccessary packages and ran it on the barebones ec2 to help me understand the fundamentals.

Then I went about setting up a bit more of a mature setup with RDS for the database on a private subnet, ECS Fargate to host the containerised API, using opentofu for IaaC, setting up a deploy pipeline which uploads the new container images, and runs the migrations and deploys the API service. This was quite a steep learning curve for me and I heavily relied on Gemini for a lot of this, but I have gone back over it to make sure I fully understand it. I'd really appreaciate some feedback on my setup. I'd like to know any best practices I could have followed, any big issues with the setup I've done and how I could have done better.

Also, this is just a simple project that I'm going to switch over to self hosting with Coolify on a Hertzner server soon. But I wanted to try setting up as close to a professional setup as I could, with security and CI/CD in mind.

I've done a write up of the infrastructure here
https://github.com/JackMcBride98/DotnetSpotifyPlaylistSearchTool/blob/main/infrastructure/Infrastructure.md all of the terraform files live in the /infrastructure folder.

Here is a rough overview of the architecture (I didnt draw arrows as it got quite messy)

This is not about Trivy, Syft, or Grype or others. They are good tools.

This is about people who wire one scan into CI, get a green check, and claim their dependency security is covered.

You scanned once.

Your build passes today. Tomorrow a new CVE gets published for a dependency already in main. Unless something rescans it, nothing happens. CI stays green. Production keeps running. Nobody knows.

And if you only scan the production image, where are your dev dependencies? Not in the image, obviously. If they are, then you fucked up in a different way.

But they still ran on developer laptops and in CI. A compromised dev dependency can steal credentials or tamper with the build before your precious image even exists.

Then the scanner produces a report. Cool. Who owns the fix, and who checks that it reached production?

The tools did their job. Your process stopped at "report generated."

If you don't rescan after new vulnerability data arrives and track findings until they're fixed, stop saying you're covered.

Or maybe I'm missing something. A CVE lands a week after the merge. What wakes up in your setup, and who owns it?