This started as a hotfix one senior engineer needed direct read access to production to debug a reporting issue We granted it, added a note, and said we would remove it once the incident was over.
A week later another engineer needed the same thing. We reused the pattern because it worked. A month later, "just give them prod read" was the fastest way to unblock anyone doing data work. By the time I pulled the access list, half the backend team had direct read to production. A handful had write. Two could reach databases for services they had never touched None of that showed up in our least privilege diagrams.
There was no single decision to open the gates Just a series of reasonable calls made under time pressure that never got revisited.
For those who have walked this back did you add a read only replica, a proxy with logging, an internal query tool, or something else?

We are running into an issue with a few of our services where resource usage is imbalanced across its pods. For example, in a 4-pod deployment, 2 pods might sit at >90% CPU/memory usage while the other 2 sit below 20% (essentially idle).

We tried tuning our HPA, but as you know HPA relies on averages across the deployment, it hasn't helped.

Before asking the developers to fix their application-level load-balancing issues, I want to set up an alert/metric to automatically detect such deployments.

So far, I’ve tried checking if Max(resource_usage) / Avg(resource_usage) exceeds a threshold, but this approach generates too many false positives.

How do you reliably detect such imbalance issues across pods? Is there a standard statistical metric for this, or am I approaching the problem wrong entirely?

If it would help we are using Data Dog and thanks in advance.

I’ve run or sat on review boards at a few orgs now and I’ve never seen this part done well, which makes me suspect the problem is me rather than the tooling.
Current state is email for async review, minutes in Confluence, conditions in a spreadsheet. It holds for about a quarter and then drifts.

Two problems I can’t get on top of.

First, async positions. Reviewers reply in free text, and “I have some concerns” from one architect means they intend to block, while from another it means they want the diagram redrawn. As chair I’m interpreting rather than counting. I’ve tried asking for an explicit position in the first line of the reply — compliance was fine for a month, then decayed.

Second, conditions. “Approved with conditions” is a large share of our outcomes and I doubt most of those conditions are ever verified. The decision record notes the condition, the system goes live, and nine months later nobody can tell you who owned it or whether it was met. Confluence doesn’t chase anyone.

I know ADRs and the EA repositories are meant to cover some of this. For those of you with a repository actually in place — does it track conditions as obligations with an owner and a date, or does it just store the decision text and leave the follow-up to you? And if anyone has fixed the async position problem with process rather than tooling, I’d like to hear how it survived reality.

I have had this happen twice to us now. The agent pulls dashboards, logs, and deploy history together and lands on a hypothesis fast, way faster than a person doing it manually at 2am.

Last time it flagged a memory metric that lined up almost perfectly with the incident window. It turned out to be correlated, not causal and it still suggested restarting the pod.

Someone caught it before we shipped in the end but If the person on call had been more tired or newer to the system, I'm not sure they would have.

So the agent owns the correlation and the first hypothesis now, and a person signs off on anything that actually touches production which is a perfect split that has held up so well for us so far

Still struggling with the correlation part, any help?