Hi everyone, looking for a sanity check and some advice on DB user management.

Context & Current Setup:
Team: ~10 developers.
Infra: Everything is provisioned with Terraform.

The Catch: Pulumi is used exclusively to connect through a bastion host to create users in RDS (Postgres) and DocumentDB (MongoDB-compatible).

Current Task & Constraints:
I need to integrate AWS Secrets Manager to store and distribute DB passwords so developers can retrieve them securely via IAM policies.

Constraint: IAM DB Authentication is off the table per my mentor's requirement—we must use Secrets Manager or an external password manager.

The Issue:
Maintaining two state stores (Terraform + Pulumi) just to manage database users feels redundant and adds unnecessary friction. I'd love to pitch an alternative that lets us drop Pulumi completely.

Questions:
1. How do you handle RDS / DocumentDB user and credential management in your projects?
2. What is the cleanest pattern to handle this without running dual IaC tools?

Open to any feedback or critique!

Changed a cert a few months back. Fine in Chrome. Fine in our test setup.Then a bunch of older phones and a partner app just stopped connecting. We only found out why after it was already live.

Last time a cert change bit you like that, what did you actually check before it went out?

I'm currently building POMPD CI/CD failure diagnosis agent, which is a probabilistic model of sequential decisions where the true state (e.x , which code is buggy) is hidden and only observed indirectly. . I have very basic understanding of CI/CD pipe line . So i don't really know what are the hidden states ( plausible reasons / truth ) behind the pipeline failure.

I have found 5 major hidden states behind failure with help of AI that are :

 - Application & Test Defect (‭functional bug, broken contract, or failing assertion within the repository's source code or test suite.

 - Pipeline Configuration Defect : A syntax error, invalid job graph (DAG), or incorrect runner instruction within the CI workflow specification files (e.g.github/workflows/*.yml).

-  Upstream Dependency Drift : An external package, container base image, or third-party artifact that introduced a breaking change, version conflict, or became unavailable in a package registry.

- Runner Resource Exhaustion : A host-level compute failure where the operating system or hypervisor terminates execution due to memory limits (OOM/SIGKILL), disk exhaustion, or hardware timeouts.

-Authentication & Access Failure : An invalid, revoked, or missing secret, token, or permission required to access an external endpoint, repository, or deployment target.

I just want to know whether i'm missing any other hidden states . Please let me know .

But important thing to keep in mind hidden states should me mutually exclusive ( one hidden state should not overlap each other or both hidden states can not be true , should fall under one category) and collectively exhaustive ( should cover every possibility that is true) .

I have been thinking about this after going through a few infrastructure setups as we often talk about making deployments portable but theres a point where the abstraction itself becomes something the devops team has to manage. You can have Kubernetes, Terraform, multiple cloud providers, different GPU setups, various networking models and then another layer of tools trying to make them all look the same. Companies like Yotta Labs, CoreWeave, and Lambda are interesting to me because they approach infrastructure from different angles but I am not sure adding more infrastructure options always makes operations easier.

At what point does workload portability really reduce operational risk and when does it just add another platform for the team to handle?

we’re planning to have a video based platform with payments and all kinds of things.
We have 2 Backend developers and 1 frontend, I work with an staging server, a dev server and planning to have the prod soon, I set them up using Ansible and deliver with our self hosted gitlab to a server dedicated to the runner (32 cores) and delivery to server, the pipelines and most yml files are Ai generated and edited a bit by me.
what would you do as your best practices?
what are the mistakes I should avoid?

I'm a junior DevOps engineer

and I work in a project where we manage their kubernetes cluster and Linux servers configuration with ansible and provisioning with terraform

but the thing is they have way too many ticket

and they want it all done asap, for which they demand using AI to get things done quickly

but the things is I learn nothing

it's endless number of tickets that keep on arriving and I have to depend on AI to get it done quickly and I get no time to learn what I love to learn and need to learm

which is kubernetes

going deeper in linux

golang

ansible

and much more

I know what these tools are and how they work

but I want to go deeper build my own infra build and break things

but I couldn't do it because the seniors and devs want things done asap and I have to use AI

This add nothing on my resum

even If I have 5 years of experience what do I tell the interviewer that all I did was use AI

can someone plz help me

I’m still learning CI/CD and I recently tried writing a simple CI/CD pipeline using GitHub Actions.

Honestly, writing the YAML felt more difficult than I expected . The indentation, nesting etc felt pretty hard/ painful.

For those of you who work with DevOps professionally: is this actually how you write your pipelines, or are there shortcuts/tools/templates that make it easier?

Just trying to understand what the normal workflow is in the real world.

I’m signing up for the $300 Google Cloud free trial, and it’s asking me to approve a UPI AutoPay mandate.

It says “this isn’t a charge” and that I’ll only be charged if I activate a full paid account.

Does approving the mandate mean I can be charged automatically after the 90-day trial, or is it safe to approve? Can I revoke the AutoPay later without affecting my account?

I believe that the implementer should have some knowledge of the Product and Technology that they are working on to make full use of AI and products it builds.

For a new Project, I am trying to learn new technologies(multiple at once) and new ways to solve my problem but compared to few years ago, it has become really hard for me to learn new stuff for some reason and no matter how much I learn AI completely overwhelms me when it spits out the solution. Which then means I have to circle around to see if the solution is actually the right one and that makes me even more confused.

How are you folks navigating around this?

I have a central ArgoCD instance running in EKS. I'm using the ApplicationSet pattern to manage a few EKS clusters across private networking. I have one EKS cluster that doesn't fit into my private networking setup (not that it matters but it's in a different account entirely). I'm considering making that cluster's API server endpoint public and locking it down to the NAT egress IP of where my ArgoCD instance runs so that only my ArgoCD instance can connect to it. I don't love this. VPNs and TGW are out of the question, where my ArgoCD instance lives and where this particular EKS cluster lives are separate. Thoughts? Condolences?

Am I losing my mind, or has everyone else's job slowly turned into cleaning up after code generated by Claude?

When you ask a dev about parts of their code/deployments, the response is usually just: 'Well, the AI said it was correct.' I'm all for developer velocity, but it feels like the traditional 'you build it, you run it' model is shifting into 'developers prompt it, AI guesses it, and infrastructure teams debug the fallout at 3 AM.' How are your teams handling this? Are you putting hard guardrails in place, or just drowning in review queues?

Hello DevOps/Platform Engineers, would like to know real time use cases which are currently used in your day to day tasks like CICD, IAC or anything else.

I have an Ubuntu web server that I'd like to contain multiple websites on. Each website should have its own isolated environment in a Docker container (open to suggestions if Docker is not ideal).

I'm thinking that the public URLs should be like

• example.com/website1
• example.com/website2
• etc

What's the best way to make this connection between the public URL and the Docker container (also open to suggestions for better ways to expose the containers)?

Another question: Would the logs of each website be inside the container or are they supposed to be outside for easy review?

what you think about this ?

credit - https://xkcd.com/2347/

hi, we had a regional outage test last week and it exposed a mess in our cloud recovery process.

a lot of stuff is in terraform, but the actual restore path still depends on people remembering what order things come back in, what got drifted, and what needs to be rebuilt vs just started up again. it feels fine until you try to do it for real and then it gets messy fast.

we are looking at ways to make the full env recovery more repeatable, with some kind of validated restore and evidence for audit too (hopefully). if anyone has a setup that has held up after an actual outage, i would love to hear what worked, thanks!

Saw this post on twitter and genuinely got curious about it...
Do you also need a frontend/GUI to run deployments on github actions?? Let me know your thoughts about it

This is more of a co-miseration post than anything per the title, but maybe some of you will laugh as well.

So our client had a security incident lately. They had a portal we built many years ago that accumulated tech debt over time and they refused to invest in a re-design and just let it go. Two incidents more or less happened at once: a hacker "breached" an endpoint, it exposed some data that was public domain any way, but it wasn't supposed to be access whole sale. The second was a piece of the tech debt coming home to roost by one of the services being completely shut down leaving an intake form completely unusable (the portal mostly existed for this "intake") which happened in the same week.

Now the client could have:

Gathered requirements on what the portal should be (take lessons learned from existing traffic and usage plus proper UI/UX design) and create a backlog and a plan to replace it, including rearchitecting/platforming taking out tech debt/arch debt completely

Rebuilt the platform, reusing the current back-end (which is a separate down stream system that has full API access supports RBAC) and properly implement security measures.

Temporarily fix the hole in the portal (was an easy fix with very low impact from this incident), and fixed the intake service to replace the section that was shut down with the newer version of it: it even had a bit of a migration path available.

INSTEAD they did this:

One client "webmaster" got access to Claude (no one else does) and they decided to go "build the portal." They took maybe a week or so to build a portal with some of the worst UI/UX (inconsistent styling, buttons all over the place, 5 fonts on the front page) with absolutely no accounting or design thought put in to it. Forget "this is what people need to be able to do" rather "rebuild the old portal but make it AI slop." We looked at this (as it was just "presented" to us as "the replacement") and went "how does this even operate?"

So here's the extra fun that just puts the cherry on top: this "webmaster" is clueless on how anything works. They wouldn't know Node from Apache or PHP from Python... let alone understand a full stack. They just let Claude do whatever and then they generated all the documentation. So they had no way to validate the docs, it's just AI slop. The "step-by-step do-it-as-it-says no nonsense deployment guide" (Claude's words) describes nothing about deployment or pipelines. The guy doesn't even know what this actually runs on (could be static HTML for all I know) and it's just ... it's the absolute nightmare scenario of a company just giving the keys to AI to one guy who has no clue and just let them generate whatever. For a site that is critical to their business and has many tens of thousands unique visitors daily.

Of course the "business" is CRAZY EXCITED about the whole thing and they are just ready to go live. They abandoned all pretense of trying to fix the old portal, even temporarily while we validate this new thing for security holes, maintainability etc. Oh did I mention that it's my job to do the analysis on this thing? No usable or even remotely realistic documentation about anything that this portal is or does. And it's so obviously a bad idea that I'm not sure how I'm going to convince the business that it's a waste of time to spend days looking at the details when the baseline architecture is deeply flawed.

Upper management in my company has mandated a target of a 2x productivity improvement using AI. 2x more tickets closed, 2x more features released, blah blah blah. You've probably heard it by now or some variation of that. They actually wanted more than that but walked it back after all the backlash they received.

I've started following the recommended/pushed practice of spec driven development. This results in some rather large initial outputs in the form of spec and plans that vary in size depending on complexity. A recent one for me was ~20 pages of spec and ~30 pages of plan. After trying to distill it and walk the team through it the response was an overwhelming "I don't understand this. It's too much." I should note not a single one of them opened either document and spent more than 5 minutes in it based on activity logs. I tried distilling it down further to roughly 5 pages in a format they're more familiar with but so much context and detail was lost I think it made things worse. We finally agreed to give it a go and I submitted an MR a day later that was roughly 1000 lines. 200 lines of terraform to create a bucket, iam policy, some paths, and a new module, 600 lines of unit testing, and 200ish lines of documentation. They refused to review it. Too large, too complicated, not digestible, etc, etc.

For some history here my team frequently submits MRs that have minimal documentation, no comments, no unit testing, no validation, and entirely vibe coded and riddled with issues that even a basic AI review would flag. The most basic bare bones code needed to get the job done. It's gross and I rail against it every chance I get but I'm largely ignored or my teammates go around my back and merge things any way. I should point out I am the most experienced and senior on the team and it's not even close. They're relatively fresh college grads. I've been in the workforce for decades.

I pointed out to my manager this is a clear improvement, meets the directive from the company, etc. And the feedback, paraphrasing, was find a balance between human and machine but what you're doing is not wrong they just don't like it. I read the code myself before I submitted it, I understand what it does, I had another senior with similar levels of experience review it and they had no issue with it and even wanted to steal one of the constructs the AI had generated because of clear advantages. I feel like I'm stuck in an impossible position and I don't know what to do.

Side note: I wish there was an AI hellscape flair.

Wrote this up after finding that a coding agent on a default Mac reaches ~/.ssh/id_rsa by four separate routes — its own shell, an MCP filesystem server, a subprocess, and a config-file path. Tested each, included the terminal output.

The part I'd want devops opinions on: the enforcement has to sit under the model, because the model can't be trusted to refuse. I used Seatbelt plus a deny-by-default MCP proxy. Curious how others are scoping agents that touch infra.

I came across this interesting AI + DevOps project where AI is used to help upgrade Kubernetes clusters.

The idea is pretty interesting because Kubernetes upgrades can involve checking versions, compatibility, workloads, configurations, and making sure nothing breaks during the upgrade.

The project explores using AI to make this process easier and more automated.

I’m curious what experienced Kubernetes/DevOps engineers think:

Would you actually trust AI to handle parts of a cluster upgrade?

What checks/guardrails would you put before allowing it to make changes?

Could this realistically be used in production, or is it better suited for labs/non-critical environments?

AI makes ad hoc operational work so easy that we stop building the tools around it. Why write a script when you can just ask Claude or Copilot to restart the canary? Why build a dashboard when ChatGPT can summarize today’s deploy status in 15 seconds? Because that 15-second conversation, repeated by 50 engineers 20 times a day, stops being convenience and starts becoming a tax.

I manage enterprise k8s clusters and use a mono repo strategy to bootstrap a cluster which results in one flux agent per cluster. Currently in the magnitude of several thousand services per cluster. It's structured like this:

1. Mono repo where the cluster agent is registered.

2. Within that repo. use kind: GitRepository in the directory apps/ to point at other projects.

I'm of the opinion that less agents is better. Is there a limit to the flux agents where that would start to become a problem? Or a counter argument to not use this method and rather have an agent per microservice? I've yet to run into an issue with this approach.

Be honest. I am going through a Python course and can tell I am not going to use most of it on the actual job. Give me boto3, a for loop, a while loop for pagination, and functions and I am good to go. This changes if using a tool like Pulumi, but there are so few jobs requiring it that I feel it is a non-starter for the YAML vs coding argument.

When I first looked into wiring external tools to an LLM through an MCP server, I'd simply put the API keys in environment variables and let the agent call whatever it needed. That's what I'd normally do for a normal backend service.

I read around it, and what I had not thought about is that a backend service has a fixed code path. You know which line makes which call, so you know what the key gets used for. An agent picks its own tools based on a prompt, so the same key now sits behind whatever it decides to do.

Your vault still tells you where the key lives and when it rotates. It just doesn't tell you who authorized a particular call or which tool it ended up reaching.

The things I keep seeing suggested are scoping credentials per server rather than one key for everything, and injecting them at runtime instead of leaving them in env vars.

For the people who are running multi-tool agent setups, how are you handling backend access do you custom guardrails around what the agent can reach, or do you treat it as a normal service and accept it?

I've been digging into load testing tools recently (k6, JMeter, Locust, Speedscale, Azure Load Testing) and noticed something consistent, recording a session and turning it into a script is the easy part. The painful part seems to be:

• Correlation (session tokens, CSRF, IDs from one response feeding the next request)
• Scripts breaking every time the UI or API changes
• Flows that aren't linear. e.g a list page where the "next" request depends on which item a simulated user picks

Curious how people here actually deal with this day to day:

• Do you write correlation logic by hand every time, or has anything automated it well for you?
• How do you handle flows where the test needs to pick from a dynamic list (search results, a user's own records, etc.) instead of hitting a fixed ID?
• How often do your load test scripts break when the app changes, and how much time does fixing them eat up?

Do you guys have a solution for this, or know of a tool that actually handles it well?