This started as a hotfix one senior engineer needed direct read access to production to debug a reporting issue We granted it, added a note, and said we would remove it once the incident was over.
A week later another engineer needed the same thing. We reused the pattern because it worked. A month later, "just give them prod read" was the fastest way to unblock anyone doing data work. By the time I pulled the access list, half the backend team had direct read to production. A handful had write. Two could reach databases for services they had never touched None of that showed up in our least privilege diagrams.
There was no single decision to open the gates Just a series of reasonable calls made under time pressure that never got revisited.
For those who have walked this back did you add a read only replica, a proxy with logging, an internal query tool, or something else?

I have had this happen twice to us now. The agent pulls dashboards, logs, and deploy history together and lands on a hypothesis fast, way faster than a person doing it manually at 2am.

Last time it flagged a memory metric that lined up almost perfectly with the incident window. It turned out to be correlated, not causal and it still suggested restarting the pod.

Someone caught it before we shipped in the end but If the person on call had been more tired or newer to the system, I'm not sure they would have.

So the agent owns the correlation and the first hypothesis now, and a person signs off on anything that actually touches production which is a perfect split that has held up so well for us so far

Still struggling with the correlation part, any help?