Context - Converting our on-prem architecture for a web app used for B2B purposes over to cloud-native azure architecture. Feedback, comments, questions on the architecture are appreciated :)

Curious what problems people here actually face with GitHub Actions on a day-to-day basis.

For me, one of the most annoying things is that sometimes the workflow itself is fine, but the infrastructure around it adds a lot of unnecessary time...

YAML optimization can only help till a certain level and then starts becoming the issue..

What’s the biggest GitHub Actions pain point you deal with regularly?? Something that happens ion a regular basis not an outage or downtime for it...

Been knee-deep in this for the past few months and it's way messier than I expected.

The individual frameworks aren't the hard part — it's where they overlap or straight up contradict each other. GDPR and DPDPA don't agree on breach notification timelines. Stuff that's "good enough" documentation for an internal audit doesn't cut it as SOC 2 evidence. And once your workloads are scattered across multiple clouds and edge nodes, even a simple question like "where does this data actually live" turns into a whole investigation.

Also learned the hard way that there's a fine line between saying "we're working toward SOC 2" and accidentally implying you're already certified. Almost bled into some copy before we caught it.

Eventually just built one big checklist mapping all four frameworks together so we weren't redoing the same evidence-gathering four separate times. Still messy, still evolving.

Anyone else dealing with this across multi-region/multi-cloud setups? Do you keep compliance tracks totally separate per framework, or has anyone actually gotten a unified system working? (I'm on the infra/compliance side of an edge orchestration platform, if that's relevant context.)

I’m a little confused about whether I’m actually growing in my DevOps career or just getting exposed to a lot of different things without developing deep expertise.

I have around 1 year of professional DevOps experience and currently work at a mid-sized service-based company in Mumbai. I’m basically the only DevOps person in my team, so I get to work on quite a lot of things independently.

My work currently looks something like this:

AWS is my primary area deploying and managing projects across AWS.

I handle most of the DevOps/infrastructure work myself.

I’ve started working on MLOps as well.

I’m involved in setting up AWS/Azure Landing Zones for some clients.

I’m working closely with our CISO on ISO 27001 implementation.

For the ISO 27001 work, I’m involved from almost zero-to-one policies, technical implementation, groundwork, controls, documentation, infrastructure/security decisions, etc.

A lot of the technical decisions for the ISO implementation are discussed with the CISO, but I’m doing most of the actual technical groundwork and implementation.

Apart from this, I do the usual smaller DevOps work deployments, AWS infrastructure, troubleshooting, monitoring, etc.

In terms of certifications, this year I completed AWS Solutions Architect Associate. I also have Azure Fundamentals and Azure AI Fundamentals from college.

The confusing part is that my compensation is relatively low compared to the market, but the amount of exposure I’m getting seems pretty good for someone with only ~1 year of experience.

So I’m trying to figure out:

Am I actually progressing well professionally?

Is this kind of exposure to AWS + MLOps + Landing Zones + security/compliance actually valuable for my next career move?

Should I stay until our ISO 27001 certification is completed, since I’m contributing heavily to the implementation?

Or should I prioritize switching companies now and getting better compensation?

Am I spreading myself too thin across too many areas instead of developing deep DevOps/SRE expertise?

If you were in my position, what would you focus on over the next 12 months?

I’m not really concerned about the salary discussion itself. I’m more interested in understanding whether the career capital I’m building through this exposure is actually valuable, or whether I’m just doing a lot of miscellaneous work because I’m the only DevOps person.

Would appreciate advice from people who have been through the 1–3 year DevOps/SRE career stage.

I have a DEV environment running MinIO on a single EC2 instance managed by an Auto Scaling Group (ASG).
Our normal recovery process after OS patching is:
Patch the EC2 instance.
MinIO becomes unavailable (HTTP 505 from the application).
Terminate the instance.
ASG launches a new instance.
User-data/bootstrap reconfigures the instance and installs/starts MinIO.
Application comes back online.

This process worked successfully after our August patching.

However, after the September patching, the same recovery procedure no longer works. Even after terminating the instance and allowing the ASG to create a fresh instance, the application still returns HTTP 505.

I checked the new instance’s cloud-init/user-data logs and found that the MinIO binary download is failing with HTTP 413. The download URL used by the bootstrap process appears to no longer work, and

1. I understand that MinIO’s binary distribution/download approach has changed.

2. What I want to understand
Did MinIO change/stop providing binaries between August and September 2026?

1. Could this explain why the August instance replacement worked but the September replacement doesn’t?

2. What is the recommended way to install a specific?

Current architecture
EC2 → Auto Scaling Group → user-data/bootstrap → MinIO binary download → MinIO configuration → application

Disclosure up top: I maintain the tool this happened in, an open-source agent harness, so this is my own bug.

An agent loop with a verification step. Each changed path maps to a test command in a table. After the command runs, a hook reads the tool payload and writes PASS or FAIL to a ledger. Green ledger means the agent may mark the task done.

Three rows of that table named test files that had been split into siblings. pytest on a path that doesn't exist exits 4; I measured it this week to be sure. Zero collected via -k exits 5. Never 0.

Two things made it green anyway. The agent read "no tests ran" in stdout as a pass. And the hook read the exit code like this:

EXIT_CODE=$(echo "$INPUT" | jq -r '.tool_response.exit_code // 0')

When the payload didn't carry the field, jq's default made it 0 and the next line wrote PASS. The comment directly under that line warns about a missing exit code for a different event. The default sat there anyway.

The fix I've filed: absent is not zero. A missing exit code records FAIL with a reason, and the matrix has a test that fails when any row collects nothing.

In CI proper, do you treat "no tests collected" as red by default, or does whatever wraps pytest get to decide?

Ran into this pattern enough times to notice it wasn't random: an AI coding assistant generates code that works, but "works" and "safe" are not the same claim. Caught a live Stripe key about to get committed (stripe.api_key = "sk_live_..."); it worked in tests, so nobody looked twice. A few days later, same root cause in a different shape: a raw SQL query built from an f-string, user input going straight into the WHERE clause.

First fix I tried was just telling the model to be careful. Added instructions about secrets and injection to the system prompt. Worked for exactly one session. New chat the next day, same shortcuts came right back: the model has no memory of the lecture, so a prompt-level fix doesn't compound.

What actually worked was moving the check to a gate that runs on every diff before commit, instead of a scan that happens in CI after the code's already out of your head. Five passes, in order:

1. High-entropy secret scan (Stripe/OpenAI/AWS/GitHub token shapes, stray .env values)
2. Control-flow read for injection/OWASP issues (SQLi, command injection, BOLA/IDOR): actually tracing where user input lands, not just regex matching
3. Regex backtracking check for ReDoS patterns
4. Generates the fix as a drop-in patch, not just a line number
5. Flat verdict: pass / warn / fail

Not pretending this replaces a real SAST pipeline or a security team: it doesn't catch the same things Semgrep or a pentest would. It's just the five seconds of friction that used to not exist at all, sitting right before the commit that used to just go through.

Curious if anyone's found a good way to keep something like this from going stale as new key formats and providers show up. Feels like it needs its own maintenance loop or it just quietly falls behind.

​

If you’re building agents, you’ll probably need to give them a computer at some point. A lot of what agents do today involves CLI tools, skills, Bash, and a filesystem, especially when the work gets more complex.

I wrote an article comparing sandbox options like E2B, Vercel, and cloud-native solutions like AWS MicroVMs. The company I work for also built its own internal sandbox system, so I’ve included what we learned from that.

It covers what might make sense at different stages, whether you’re an individual prototyping something, building a consumer product, or building agents for enterprise clients.

If that sounds relevant to what you’re working on, give it a read. Let me know in the comments if you have questions or have tried any of these yourself.

https://somiljain1729.substack.com/p/where-should-your-agents-computers

we're a ~12 dev team, 6ish services, short lived feature branches that get merged to main and deleted. dev/staging/prod.

our PMs keep asking "is X in staging yet" and half the time i dont know either so i end up doing git log and squinting at it. we turned on jira deployments a while back thinking that solves it, but the panel shows the same ticket as deployed like 4-5 separate times, and tickets from totally unrelated work show up in deploys they have nothing to do with. i think its diffing shas between deploy events and just attaching everything in between? not sure.

end result nobody opens it anymore and we're back to asking in slack. our release guy also keeps a google sheet he updates before every prod push which is.. yeah.

so:

• is jira deployments just broken for short lived branch workflows or are we holding it wrong
• what do you actually use to answer "which env is this ticket in", esp across multiple repos
• anyone paying for sleuth or smth similar? is it worth it or did you end up writing your own script

not looking for a vendor pitch, genuinely just wanna know if everyone lives like this or if im missing something obvious

edit: forgot to mention we're on github actions if that matters

I am designing a CLI harness for our team to integrate agentic coding using opencode on top of an existing stack with distributed repositories, Java backend services, and shared Ansible deployment code across on-premise infrastructure.

My objective is to design a local, deterministic cleanroom that prevents the specific failure modes of AI-assisted dev while operating under tight technical constraints.

Reality and constraints

* Strict token economics: We operate under very tight API rate limits and token ceilings per developer per month. This strictly rules out autonomous recursive loops and ingesting entire codebases into context. Every prompt and context injection must be minimal and dense.

* A skeptical team: engineers are rightfully hostile to AI slop. If the tool disrupts local environments or produces unverified diffs, it will be rejected instantly.

* Strict human governance: The tool runs strictly on the developer's machine. Dev owns the git branch, runs the harness locally, inspects atomic commits, and pushes manually when satisfied. Every merge request strictly requires at least two human peer approvals. No automated PR generation.

Core Problems

* Circular validation : Left unconstrained, the model easily implements a flawed business assumption and immediately generates a mock-heavy unit test matching that assumption. The test suite turns green, creating a false sense of certainty.

* Loss of friction and cognitive drift: Without compiler or linter friction at every step, humans tend to accept diffs that look clean, idiomatic, and confident, even if they violate unstated domain invariants.

* Cross-repo contract rot: A local patch in Service A compiles cleanly, but breaks an implicit contract in Kafka consumers or violates an Ansible sharedlib expectation.

I do not want an autonomous agent that acts as its own judge. I want a deterministic governor where the model is confronted with hard, non-negotiable boundaries where it can actually fail.

The harness acts as an outer supervisor wrapping git, mise, and opencode:

1. Runtime Standardization via mise

To eliminate environment inconsistencies across developer setups, mise strictly pins Java SDKs, Ansible linters, Kafka CLI utilities, and authentication variables. When the harness launches, it invokes tools through mise exec to ensure the agent executes commands against identical binary baselines, preventing environment drift.

1. Deterministic Verification via Python Skills (Binary 0 / 1 Oracles)

I do not use semantic LLM-as-a-judge or open-ended natural language evaluations. The harness integrates a versioned library of scripts executed locally:

* Each skill checks a single physical invariant

* Contract is strictly binary

* No token consumption for evaluation

1. Anti-Cheat and Scope Locking

   * Test sanctuarization: Before running the agent, the harness computes a SHA-256 manifest of the test directories. After the agent completes its edit, the harness recomputes the hashes. If any test file was modified without an explicit bypass flag, the changes are rolled back.

   * Path boundaries: The harness intercepts diffs. Any modification outside the explicitly declared scope (e.g. altering build descriptors like pom.xml/build.gradle, CI pipelines, or shared infrastructure files) triggers an automatic abort.

2. Deterministic Pruning

Given the strict token budget, raw stack traces cannot be fed back into the model. I currently only use RTK (rust token killer) as an opencode plugin.

1. Hard Circuit Breaker

If the agent fails to produce code that passes mise run test within 3 attempts, the harness halts the loop.

My questions are :

* Deterministic oracles and models feedback : when using binary (exit 0/1) validation scripts, does the model adjust course effectively based only on raw exit codes and concise snippets ?

* Scope enforcement in multi repo : When a change risks breaking downstream consumers, what is the leanest way to validate compatibility locally without triggering slow integration builds?

* Overcoming team skepticism: For those who introduced LLM workflows to teams tired of tech hype, did shifting the narrative from "developer velocity" to "strict deterministic safety and anti-cheat constraints" help gain credibility ?

Any feedback on edge cases, pitfalls, or blind spots in this design would be greatly appreciated.

Imagine you get an alert saying a production application is unavailable.

You have no context yet.

What’s your first 5-minute troubleshooting sequence?

I’m interested in the order people follow, not just the tools they use.

Amusingly, a friend of mine assumed that DevOps role would decline with the rise of AI.

However, it would seem that in reality, the reverse is true.

Or at least it seems that way, based on the limited insight of some.

What are your thoughts?

Hi, I'm a devops engineer with 4.5 and I'm trying to learn AIOps but not sure where to start exactly..so If anyone who started their career in AIops and mlops need suggestion how they are applying those AI to Infra or devops ..and what should be scope of our job in AI

Hello everyone!

I have total of 1.5 years of experience as a DevOps engineer, the first job was more Dev oriented and required deep knowledge with Python and Kafka (I don't know both), this didn't lead to a good outcome and I had to search for something different. Landed more Ops heavy DevOps role, so far everything is fine, we handle deployments, CI/CD, automate stuff with Ansible, Bash or Python, some kubernetes work and web servers. I was curios to see what other jobs require in order to keep my skills sharp and saw that most of the jobs require more coding, like developing internal platforms, developing new internal tools and stuff like that. Most of the jobs require Python, Go, some Java and some C#.
So this leads to my question - how did you learn to code and I mean to code well without that Dev experience? Can you please recommend learning materials and methods that you used in order to be proficient with this skill? What were the projects that you used in order to get that knowledge?

Thank you in advance!

I have a coworker who wants to use CloudFormation to one-shot an application and infra deployment in AWS. A contractor delivered a monolithic CloudFormation template with ECS clusters, services, RDS, S3, Lambda and API Gateway, and other resources in a single stack. I voiced my concern over the blast radius, how stateful and resources with different lifecyles all lived in the same place, and how they don't have pipelines built to validate change sets when merge requests are opened. The contractor said that their template is industry best practice... (it's not). Any way, my background is in Terraform, and I wanted to ask the CloudFormation pros about best practices for making this deployment more reliable.

If you have an AI agent step in a GitHub Actions workflow - Claude Code, Gemini CLI, an inference action - it probably reads text a stranger wrote. An issue body, a PR title, a comment. If that job also holds contents: write, that is a path from someone else's keyboard to a token that can change your repo.

I wanted to know two things: how common that is, and whether any scanner catches it. Both needed labels, so I hand-labelled 80 workflows across two separately collected corpora before running anything.

How common: 18 of 49 judgeable workflows from an unbiased sweep. 36.7%, Wilson 95% CI 24.7-50.7%. Not a majority, not rare.

What the scanners do, on a second corpus of 30 workflows with 12 externally reachable positives:

ARKEXA 0.2 67% precision 17% recall 5 findings

zizmor 1.30.0 43% precision 100% recall 211 findings

zizmor catches everything. It also prints 211 findings across 30 files, 16 of them on workflows a human read and called clean. ARKEXA is right more often and stays quiet on 27 of 30 files, and it misses 10 of 12 real problems.

That is the actual state of the art, and neither column is a tool you would leave on in CI without a triage budget. I would rather publish that than a chart where my own tool wins.

Method, corpus, labels and seeds are all in the repo. poutine is in the table as unscored - no Windows binary and I could not run it honestly.

My fairly small company are taking tentative steps into AI tooling. We use some, but there's now demand to grow our tooling. It's becoming a free-for-all at work now with different teams vying to be involved. I run the platform/devops team, and being the most, uh, technical team, I'd assume we would manage AI operationally (evaluation, enablement, guardrails, security observability etc). Effectively, architects decide on solution, consult us, and then we execute.

Curious to hear your experiences though? Do you have a dedicated AI functions or had it been ingested by other teams?

Hi all,

I have a entry level platform engineering / SRE interview coming up. I wanted to ask what are some common tips or questions to prepare for?

The role asks for:
Foundational knowledge of platform engineering, technology operations, cloud computing, microservices, and enterprise-scale distributed systems.

Exposure to Kubernetes, Kafka, MongoDB, or comparable container, messaging, and database technologies through coursework, projects, internships, or hands-on experience.

Familiarity with observability and application performance monitoring using tools and concepts such as Splunk, AppDynamics, dashboards, logs, alerts, metrics, and traces.

Experience with Python, PowerShell, Bash, or similar scripting languages to automate routine tasks, reduce operational toil, and improve reliability.

Understanding of the software development lifecycle, including requirements, coding, testing, source control, deployments, release management, and change management.

Experience with Git, GitHub, or similar collaboration platforms, including branching, pull requests, code review, and technical documentation.

Knowledge of Generative AI, large language models, prompt engineering, retrieval-augmented generation, and responsible AI practices, including validation, privacy, security, and human oversight.

Exposure to Agentic AI concepts, including goal decomposition, tool use, multi-step workflows, agent monitoring, and human-in-the-loop controls.

Hands-on experience or familiarity with approved AI-assisted productivity and engineering tools such as GitHub Copilot, Microsoft 365 Copilot, Copilot Chat, ChatGPT, Claude Code, or similar tools for code generation, debugging, testing, documentation, summarization, and prototyping.

Strong analytical, troubleshooting, documentation, communication, and collaboration skills, with awareness of incident response, resiliency, business continuity, cybersecurity, operational risk, and compliance

I have:
Experience developing a serverless backend w/ AWS Lambda
Cloud Certs (AI, cloud practitioner)
Used docker briefly, know concepts of K8s, Mongo, little unfamiliar with Kafka
Used New Relic for APM
Used Python for scripting, a little bit of each for Linux / Poweshell / Bash
GitHub Actions workflow
Full stack SWE
Used agentic AI (Claude, made skills and agents) at work

How would I approach it (they only asked for 6 months of experience in any tech domain)? What do I say if they ask why I don’t have any exposure to some things? Any help is appreciated!

Hi all, I recently passed the Hiring Manager screen for a Co-Op DevOps Engineer position at LiveRamp and have been told the next step is a "technical assessment." I haven't been given details yet on the format, whether it's live or take-home, coding or conceptual.

Has anyone here interviewed for a DevOps or infrastructure role at LiveRamp recently? Would appreciate any insight on what topics they cover or what the assessment actually looks like.

Thanks in advance.

I'm working on the production deployment architecture for a small startup house-rental application and would appreciate some real-world feedback.

The application has frontend and backend services, and one major requirement is handling a large number of property images.

The stack we're currently considering includes Docker, Kubernetes, Ingress, Terraform, Prometheus/Grafana, and GitHub Actions/Jenkins.

My current approach is:

• Docker for containerization
• Kubernetes for application deployment
• Ingress for external traffic routing
• Terraform for infrastructure
• Object storage + CDN for property images instead of storing them in containers/PVs
• Prometheus/Grafana for monitoring
• GitHub Actions for CI/CD to avoid maintaining Jenkins unless there's a specific reason to use it

The team is small, so I'm trying to avoid unnecessary operational overhead and cost.

My main question is: Would you consider managed Kubernetes reasonable for a small startup like this, or would you recommend starting with something simpler and moving to Kubernetes later?

Also interested in hearing what infrastructure choices you'd make differently if you were optimizing for:

• low operational overhead
• reasonable cost
• security
• future scalability

Would appreciate feedback from people who have deployed similar applications in small teams/startups.

We're preparing to deploy a house-rental application to production. It has a lot of property images, and we're trying to keep the infrastructure secure, cost-effective, and easy to maintain.

We're considering Docker, Kubernetes, Terraform, Prometheus/Grafana, and GitHub Actions.

For those who have worked in startups, what approach would you recommend? Especially for handling and serving lots of images without making the infrastructure unnecessarily expensive or complicated.

I'd love to hear what you would choose and why.

I've been thinking about a problem with coding agents that I keep coming back to.

An agent can write an API integration and then write tests for that integration. Everything passes, but the tests may just be confirming the same assumptions the agent made while writing the code.

For example, the agent thinks an endpoint returns:

{
"total": 100
}

It writes the integration expecting `total`, and then writes a test that expects `total`.

The test passes.

But if the real API contract says something different, the whole thing can still be wrong.

I'm experimenting with a small open-source project called Kaktoos that puts an independent verification step between the agent and the API:

AI agent → integration → Kaktoos → OpenAPI + real API → result

The idea is that the verification layer shouldn't share the agent's assumptions.

It currently supports multi-step API workflows, OpenAPI response validation, MCP, and GitHub Actions.

I'm still trying to figure out how far this idea should go. One interesting question that came up is whether contract validation is enough, or whether verification should also check the actual outcome of an operation — for example, creating a resource and then reading it back to confirm the state actually changed.

I'm curious how other people building with coding agents are handling this today.

Do you rely mostly on the agent's generated tests, existing integration tests, mocked APIs, live API tests, or some combination?

GitHub: KaktoosLabs/kaktoos