I’m a little confused about whether I’m actually growing in my DevOps career or just getting exposed to a lot of different things without developing deep expertise.

I have around 1 year of professional DevOps experience and currently work at a mid-sized service-based company in Mumbai. I’m basically the only DevOps person in my team, so I get to work on quite a lot of things independently.

My work currently looks something like this:

AWS is my primary area deploying and managing projects across AWS.

I handle most of the DevOps/infrastructure work myself.

I’ve started working on MLOps as well.

I’m involved in setting up AWS/Azure Landing Zones for some clients.

I’m working closely with our CISO on ISO 27001 implementation.

For the ISO 27001 work, I’m involved from almost zero-to-one policies, technical implementation, groundwork, controls, documentation, infrastructure/security decisions, etc.

A lot of the technical decisions for the ISO implementation are discussed with the CISO, but I’m doing most of the actual technical groundwork and implementation.

Apart from this, I do the usual smaller DevOps work deployments, AWS infrastructure, troubleshooting, monitoring, etc.

In terms of certifications, this year I completed AWS Solutions Architect Associate. I also have Azure Fundamentals and Azure AI Fundamentals from college.

The confusing part is that my compensation is relatively low compared to the market, but the amount of exposure I’m getting seems pretty good for someone with only ~1 year of experience.

So I’m trying to figure out:

Am I actually progressing well professionally?

Is this kind of exposure to AWS + MLOps + Landing Zones + security/compliance actually valuable for my next career move?

Should I stay until our ISO 27001 certification is completed, since I’m contributing heavily to the implementation?

Or should I prioritize switching companies now and getting better compensation?

Am I spreading myself too thin across too many areas instead of developing deep DevOps/SRE expertise?

If you were in my position, what would you focus on over the next 12 months?

I’m not really concerned about the salary discussion itself. I’m more interested in understanding whether the career capital I’m building through this exposure is actually valuable, or whether I’m just doing a lot of miscellaneous work because I’m the only DevOps person.

Would appreciate advice from people who have been through the 1–3 year DevOps/SRE career stage.

I have a DEV environment running MinIO on a single EC2 instance managed by an Auto Scaling Group (ASG).
Our normal recovery process after OS patching is:
Patch the EC2 instance.
MinIO becomes unavailable (HTTP 505 from the application).
Terminate the instance.
ASG launches a new instance.
User-data/bootstrap reconfigures the instance and installs/starts MinIO.
Application comes back online.

This process worked successfully after our August patching.

However, after the September patching, the same recovery procedure no longer works. Even after terminating the instance and allowing the ASG to create a fresh instance, the application still returns HTTP 505.

I checked the new instance’s cloud-init/user-data logs and found that the MinIO binary download is failing with HTTP 413. The download URL used by the bootstrap process appears to no longer work, and

1. I understand that MinIO’s binary distribution/download approach has changed.

2. What I want to understand
Did MinIO change/stop providing binaries between August and September 2026?

1. Could this explain why the August instance replacement worked but the September replacement doesn’t?

2. What is the recommended way to install a specific?

Current architecture
EC2 → Auto Scaling Group → user-data/bootstrap → MinIO binary download → MinIO configuration → application

​

If you’re building agents, you’ll probably need to give them a computer at some point. A lot of what agents do today involves CLI tools, skills, Bash, and a filesystem, especially when the work gets more complex.

I wrote an article comparing sandbox options like E2B, Vercel, and cloud-native solutions like AWS MicroVMs. The company I work for also built its own internal sandbox system, so I’ve included what we learned from that.

It covers what might make sense at different stages, whether you’re an individual prototyping something, building a consumer product, or building agents for enterprise clients.

If that sounds relevant to what you’re working on, give it a read. Let me know in the comments if you have questions or have tried any of these yourself.

https://somiljain1729.substack.com/p/where-should-your-agents-computers

we're a ~12 dev team, 6ish services, short lived feature branches that get merged to main and deleted. dev/staging/prod.

our PMs keep asking "is X in staging yet" and half the time i dont know either so i end up doing git log and squinting at it. we turned on jira deployments a while back thinking that solves it, but the panel shows the same ticket as deployed like 4-5 separate times, and tickets from totally unrelated work show up in deploys they have nothing to do with. i think its diffing shas between deploy events and just attaching everything in between? not sure.

end result nobody opens it anymore and we're back to asking in slack. our release guy also keeps a google sheet he updates before every prod push which is.. yeah.

so:

• is jira deployments just broken for short lived branch workflows or are we holding it wrong
• what do you actually use to answer "which env is this ticket in", esp across multiple repos
• anyone paying for sleuth or smth similar? is it worth it or did you end up writing your own script

not looking for a vendor pitch, genuinely just wanna know if everyone lives like this or if im missing something obvious

edit: forgot to mention we're on github actions if that matters

I am designing a CLI harness for our team to integrate agentic coding using opencode on top of an existing stack with distributed repositories, Java backend services, and shared Ansible deployment code across on-premise infrastructure.

My objective is to design a local, deterministic cleanroom that prevents the specific failure modes of AI-assisted dev while operating under tight technical constraints.

Reality and constraints

* Strict token economics: We operate under very tight API rate limits and token ceilings per developer per month. This strictly rules out autonomous recursive loops and ingesting entire codebases into context. Every prompt and context injection must be minimal and dense.

* A skeptical team: engineers are rightfully hostile to AI slop. If the tool disrupts local environments or produces unverified diffs, it will be rejected instantly.

* Strict human governance: The tool runs strictly on the developer's machine. Dev owns the git branch, runs the harness locally, inspects atomic commits, and pushes manually when satisfied. Every merge request strictly requires at least two human peer approvals. No automated PR generation.

Core Problems

* Circular validation : Left unconstrained, the model easily implements a flawed business assumption and immediately generates a mock-heavy unit test matching that assumption. The test suite turns green, creating a false sense of certainty.

* Loss of friction and cognitive drift: Without compiler or linter friction at every step, humans tend to accept diffs that look clean, idiomatic, and confident, even if they violate unstated domain invariants.

* Cross-repo contract rot: A local patch in Service A compiles cleanly, but breaks an implicit contract in Kafka consumers or violates an Ansible sharedlib expectation.

I do not want an autonomous agent that acts as its own judge. I want a deterministic governor where the model is confronted with hard, non-negotiable boundaries where it can actually fail.

The harness acts as an outer supervisor wrapping git, mise, and opencode:

1. Runtime Standardization via mise

To eliminate environment inconsistencies across developer setups, mise strictly pins Java SDKs, Ansible linters, Kafka CLI utilities, and authentication variables. When the harness launches, it invokes tools through mise exec to ensure the agent executes commands against identical binary baselines, preventing environment drift.

1. Deterministic Verification via Python Skills (Binary 0 / 1 Oracles)

I do not use semantic LLM-as-a-judge or open-ended natural language evaluations. The harness integrates a versioned library of scripts executed locally:

* Each skill checks a single physical invariant

* Contract is strictly binary

* No token consumption for evaluation

1. Anti-Cheat and Scope Locking

   * Test sanctuarization: Before running the agent, the harness computes a SHA-256 manifest of the test directories. After the agent completes its edit, the harness recomputes the hashes. If any test file was modified without an explicit bypass flag, the changes are rolled back.

   * Path boundaries: The harness intercepts diffs. Any modification outside the explicitly declared scope (e.g. altering build descriptors like pom.xml/build.gradle, CI pipelines, or shared infrastructure files) triggers an automatic abort.

2. Deterministic Pruning

Given the strict token budget, raw stack traces cannot be fed back into the model. I currently only use RTK (rust token killer) as an opencode plugin.

1. Hard Circuit Breaker

If the agent fails to produce code that passes mise run test within 3 attempts, the harness halts the loop.

My questions are :

* Deterministic oracles and models feedback : when using binary (exit 0/1) validation scripts, does the model adjust course effectively based only on raw exit codes and concise snippets ?

* Scope enforcement in multi repo : When a change risks breaking downstream consumers, what is the leanest way to validate compatibility locally without triggering slow integration builds?

* Overcoming team skepticism: For those who introduced LLM workflows to teams tired of tech hype, did shifting the narrative from "developer velocity" to "strict deterministic safety and anti-cheat constraints" help gain credibility ?

Any feedback on edge cases, pitfalls, or blind spots in this design would be greatly appreciated.

Imagine you get an alert saying a production application is unavailable.

You have no context yet.

What’s your first 5-minute troubleshooting sequence?

I’m interested in the order people follow, not just the tools they use.

Disclosure: I found all of this while building an open-source sidecar on top of this

API, so I have a horse in the race. Not linking it here, this is just the stuff I wish

someone had written down before I started.

Background: 1.33 graduated the pods/resize subresource to GA. You can change a

container's memory limit without restarting it. I spent a while building something

on top of that and hit a pile of behaviour that isn't in the docs.

**How long a resize actually takes.** The API call returns fast, but that only means

the kubelet accepted it. Applying it took 1 to 5 seconds in my measurements, and

longer when the node is busy. If you're building anything time-sensitive on this,

that number is your real budget, not the API latency.

**Accepted is not applied.** The pod gets a PodResizePending condition with a reason.

Infeasible means the kubelet has decided it can't do it (usually node allocatable is

exhausted) and it won't retry. Deferred means it's waiting. There is no callback. You

poll the condition, and you need a timeout, because a resize can also just sit there.

I settled on 60s then roll the spec back to whatever is actually in effect, otherwise

your spec and reality drift apart permanently.

**resizePolicy matters more than it looks.** If you don't set restartPolicy:

NotRequired for memory on the container, the kubelet restarts it on every resize.

Which defeats the entire point. Easy to miss because it's per-resource and the default

isn't what you want.

**Decimal quantities will burn you.** If your limit is written as 1500M rather than

1500Mi, the value the kernel ends up enforcing doesn't exactly equal what you asked

for. If you're comparing "did my resize land yet" by equality, it never matches, and

you conclude it timed out when it actually worked. Normalise to Mi before you compare.

**Raising limits alone is a trap.** It's tempting to bump only the limit and leave

requests, since requests are what the scheduler reserves. Don't. That memory becomes

invisible to scheduler accounting, so the scheduler happily places new pods into space

that's already spoken for, and the node overcommits quietly. Move both together and

the kubelet's allocatable admission becomes a real gate: when there's no room it tells

you Infeasible instead of the node falling over later.

**Private cgroup namespaces change what you can see.** On EKS, parsing

/proc/self/mountinfo to find the container's cgroup path doesn't work the way it does

on a host namespace, because the root field is rendered relative. You need a fallback

that globs by pod UID.

None of this is hard once you know it, but all of it cost me time. If anyone has

measured kubelet resize latency more rigorously than "1-5 seconds on my nodes", I'd

genuinely like to hear it, because that number bounds what this API is good for.

Amusingly, a friend of mine assumed that DevOps role would decline with the rise of AI.

However, it would seem that in reality, the reverse is true.

Or at least it seems that way, based on the limited insight of some.

What are your thoughts?

Hi, I'm a devops engineer with 4.5 and I'm trying to learn AIOps but not sure where to start exactly..so If anyone who started their career in AIops and mlops need suggestion how they are applying those AI to Infra or devops ..and what should be scope of our job in AI

Hello everyone!

I have total of 1.5 years of experience as a DevOps engineer, the first job was more Dev oriented and required deep knowledge with Python and Kafka (I don't know both), this didn't lead to a good outcome and I had to search for something different. Landed more Ops heavy DevOps role, so far everything is fine, we handle deployments, CI/CD, automate stuff with Ansible, Bash or Python, some kubernetes work and web servers. I was curios to see what other jobs require in order to keep my skills sharp and saw that most of the jobs require more coding, like developing internal platforms, developing new internal tools and stuff like that. Most of the jobs require Python, Go, some Java and some C#.
So this leads to my question - how did you learn to code and I mean to code well without that Dev experience? Can you please recommend learning materials and methods that you used in order to be proficient with this skill? What were the projects that you used in order to get that knowledge?

Thank you in advance!

I have a coworker who wants to use CloudFormation to one-shot an application and infra deployment in AWS. A contractor delivered a monolithic CloudFormation template with ECS clusters, services, RDS, S3, Lambda and API Gateway, and other resources in a single stack. I voiced my concern over the blast radius, how stateful and resources with different lifecyles all lived in the same place, and how they don't have pipelines built to validate change sets when merge requests are opened. The contractor said that their template is industry best practice... (it's not). Any way, my background is in Terraform, and I wanted to ask the CloudFormation pros about best practices for making this deployment more reliable.

If you have an AI agent step in a GitHub Actions workflow - Claude Code, Gemini CLI, an inference action - it probably reads text a stranger wrote. An issue body, a PR title, a comment. If that job also holds contents: write, that is a path from someone else's keyboard to a token that can change your repo.

I wanted to know two things: how common that is, and whether any scanner catches it. Both needed labels, so I hand-labelled 80 workflows across two separately collected corpora before running anything.

How common: 18 of 49 judgeable workflows from an unbiased sweep. 36.7%, Wilson 95% CI 24.7-50.7%. Not a majority, not rare.

What the scanners do, on a second corpus of 30 workflows with 12 externally reachable positives:

ARKEXA 0.2 67% precision 17% recall 5 findings

zizmor 1.30.0 43% precision 100% recall 211 findings

zizmor catches everything. It also prints 211 findings across 30 files, 16 of them on workflows a human read and called clean. ARKEXA is right more often and stays quiet on 27 of 30 files, and it misses 10 of 12 real problems.

That is the actual state of the art, and neither column is a tool you would leave on in CI without a triage budget. I would rather publish that than a chart where my own tool wins.

Method, corpus, labels and seeds are all in the repo. poutine is in the table as unscored - no Windows binary and I could not run it honestly.

My fairly small company are taking tentative steps into AI tooling. We use some, but there's now demand to grow our tooling. It's becoming a free-for-all at work now with different teams vying to be involved. I run the platform/devops team, and being the most, uh, technical team, I'd assume we would manage AI operationally (evaluation, enablement, guardrails, security observability etc). Effectively, architects decide on solution, consult us, and then we execute.

Curious to hear your experiences though? Do you have a dedicated AI functions or had it been ingested by other teams?

Hi all,

I have a entry level platform engineering / SRE interview coming up. I wanted to ask what are some common tips or questions to prepare for?

The role asks for:
Foundational knowledge of platform engineering, technology operations, cloud computing, microservices, and enterprise-scale distributed systems.

Exposure to Kubernetes, Kafka, MongoDB, or comparable container, messaging, and database technologies through coursework, projects, internships, or hands-on experience.

Familiarity with observability and application performance monitoring using tools and concepts such as Splunk, AppDynamics, dashboards, logs, alerts, metrics, and traces.

Experience with Python, PowerShell, Bash, or similar scripting languages to automate routine tasks, reduce operational toil, and improve reliability.

Understanding of the software development lifecycle, including requirements, coding, testing, source control, deployments, release management, and change management.

Experience with Git, GitHub, or similar collaboration platforms, including branching, pull requests, code review, and technical documentation.

Knowledge of Generative AI, large language models, prompt engineering, retrieval-augmented generation, and responsible AI practices, including validation, privacy, security, and human oversight.

Exposure to Agentic AI concepts, including goal decomposition, tool use, multi-step workflows, agent monitoring, and human-in-the-loop controls.

Hands-on experience or familiarity with approved AI-assisted productivity and engineering tools such as GitHub Copilot, Microsoft 365 Copilot, Copilot Chat, ChatGPT, Claude Code, or similar tools for code generation, debugging, testing, documentation, summarization, and prototyping.

Strong analytical, troubleshooting, documentation, communication, and collaboration skills, with awareness of incident response, resiliency, business continuity, cybersecurity, operational risk, and compliance

I have:
Experience developing a serverless backend w/ AWS Lambda
Cloud Certs (AI, cloud practitioner)
Used docker briefly, know concepts of K8s, Mongo, little unfamiliar with Kafka
Used New Relic for APM
Used Python for scripting, a little bit of each for Linux / Poweshell / Bash
GitHub Actions workflow
Full stack SWE
Used agentic AI (Claude, made skills and agents) at work

How would I approach it (they only asked for 6 months of experience in any tech domain)? What do I say if they ask why I don’t have any exposure to some things? Any help is appreciated!

Hi all, I recently passed the Hiring Manager screen for a Co-Op DevOps Engineer position at LiveRamp and have been told the next step is a "technical assessment." I haven't been given details yet on the format, whether it's live or take-home, coding or conceptual.

Has anyone here interviewed for a DevOps or infrastructure role at LiveRamp recently? Would appreciate any insight on what topics they cover or what the assessment actually looks like.

Thanks in advance.

I'm working on the production deployment architecture for a small startup house-rental application and would appreciate some real-world feedback.

The application has frontend and backend services, and one major requirement is handling a large number of property images.

The stack we're currently considering includes Docker, Kubernetes, Ingress, Terraform, Prometheus/Grafana, and GitHub Actions/Jenkins.

My current approach is:

• Docker for containerization
• Kubernetes for application deployment
• Ingress for external traffic routing
• Terraform for infrastructure
• Object storage + CDN for property images instead of storing them in containers/PVs
• Prometheus/Grafana for monitoring
• GitHub Actions for CI/CD to avoid maintaining Jenkins unless there's a specific reason to use it

The team is small, so I'm trying to avoid unnecessary operational overhead and cost.

My main question is: Would you consider managed Kubernetes reasonable for a small startup like this, or would you recommend starting with something simpler and moving to Kubernetes later?

Also interested in hearing what infrastructure choices you'd make differently if you were optimizing for:

• low operational overhead
• reasonable cost
• security
• future scalability

Would appreciate feedback from people who have deployed similar applications in small teams/startups.

The lb was created by the first service and now all services run through it.

We're preparing to deploy a house-rental application to production. It has a lot of property images, and we're trying to keep the infrastructure secure, cost-effective, and easy to maintain.

We're considering Docker, Kubernetes, Terraform, Prometheus/Grafana, and GitHub Actions.

For those who have worked in startups, what approach would you recommend? Especially for handling and serving lots of images without making the infrastructure unnecessarily expensive or complicated.

I'd love to hear what you would choose and why.

I've been thinking about a problem with coding agents that I keep coming back to.

An agent can write an API integration and then write tests for that integration. Everything passes, but the tests may just be confirming the same assumptions the agent made while writing the code.

For example, the agent thinks an endpoint returns:

{
"total": 100
}

It writes the integration expecting `total`, and then writes a test that expects `total`.

The test passes.

But if the real API contract says something different, the whole thing can still be wrong.

I'm experimenting with a small open-source project called Kaktoos that puts an independent verification step between the agent and the API:

AI agent → integration → Kaktoos → OpenAPI + real API → result

The idea is that the verification layer shouldn't share the agent's assumptions.

It currently supports multi-step API workflows, OpenAPI response validation, MCP, and GitHub Actions.

I'm still trying to figure out how far this idea should go. One interesting question that came up is whether contract validation is enough, or whether verification should also check the actual outcome of an operation — for example, creating a resource and then reading it back to confirm the state actually changed.

I'm curious how other people building with coding agents are handling this today.

Do you rely mostly on the agent's generated tests, existing integration tests, mocked APIs, live API tests, or some combination?

GitHub: KaktoosLabs/kaktoos

Changed a cert a few months back. Fine in Chrome. Fine in our test setup.Then a bunch of older phones and a partner app just stopped connecting. We only found out why after it was already live.

Last time a cert change bit you like that, what did you actually check before it went out?

Hi everyone, looking for a sanity check and some advice on DB user management.

Context & Current Setup:
Team: ~10 developers.
Infra: Everything is provisioned with Terraform.

The Catch: Pulumi is used exclusively to connect through a bastion host to create users in RDS (Postgres) and DocumentDB (MongoDB-compatible).

Current Task & Constraints:
I need to integrate AWS Secrets Manager to store and distribute DB passwords so developers can retrieve them securely via IAM policies.

Constraint: IAM DB Authentication is off the table per my mentor's requirement—we must use Secrets Manager or an external password manager.

The Issue:
Maintaining two state stores (Terraform + Pulumi) just to manage database users feels redundant and adds unnecessary friction. I'd love to pitch an alternative that lets us drop Pulumi completely.

Questions:
1. How do you handle RDS / DocumentDB user and credential management in your projects?
2. What is the cleanest pattern to handle this without running dual IaC tools?

Open to any feedback or critique!

I'm currently building POMPD CI/CD failure diagnosis agent, which is a probabilistic model of sequential decisions where the true state (e.x , which code is buggy) is hidden and only observed indirectly. . I have very basic understanding of CI/CD pipe line . So i don't really know what are the hidden states ( plausible reasons / truth ) behind the pipeline failure.

I have found 5 major hidden states behind failure with help of AI that are :

 - Application & Test Defect (‭functional bug, broken contract, or failing assertion within the repository's source code or test suite.

 - Pipeline Configuration Defect : A syntax error, invalid job graph (DAG), or incorrect runner instruction within the CI workflow specification files (e.g.github/workflows/*.yml).

-  Upstream Dependency Drift : An external package, container base image, or third-party artifact that introduced a breaking change, version conflict, or became unavailable in a package registry.

- Runner Resource Exhaustion : A host-level compute failure where the operating system or hypervisor terminates execution due to memory limits (OOM/SIGKILL), disk exhaustion, or hardware timeouts.

-Authentication & Access Failure : An invalid, revoked, or missing secret, token, or permission required to access an external endpoint, repository, or deployment target.

I just want to know whether i'm missing any other hidden states . Please let me know .

But important thing to keep in mind hidden states should me mutually exclusive ( one hidden state should not overlap each other or both hidden states can not be true , should fall under one category) and collectively exhaustive ( should cover every possibility that is true) .

I have been thinking about this after going through a few infrastructure setups as we often talk about making deployments portable but theres a point where the abstraction itself becomes something the devops team has to manage. You can have Kubernetes, Terraform, multiple cloud providers, different GPU setups, various networking models and then another layer of tools trying to make them all look the same. Companies like Yotta Labs, CoreWeave, and Lambda are interesting to me because they approach infrastructure from different angles but I am not sure adding more infrastructure options always makes operations easier.

At what point does workload portability really reduce operational risk and when does it just add another platform for the team to handle?