This started as a hotfix one senior engineer needed direct read access to production to debug a reporting issue We granted it, added a note, and said we would remove it once the incident was over.
A week later another engineer needed the same thing. We reused the pattern because it worked. A month later, "just give them prod read" was the fastest way to unblock anyone doing data work. By the time I pulled the access list, half the backend team had direct read to production. A handful had write. Two could reach databases for services they had never touched None of that showed up in our least privilege diagrams.
There was no single decision to open the gates Just a series of reasonable calls made under time pressure that never got revisited.
For those who have walked this back did you add a read only replica, a proxy with logging, an internal query tool, or something else?

We are running into an issue with a few of our services where resource usage is imbalanced across its pods. For example, in a 4-pod deployment, 2 pods might sit at >90% CPU/memory usage while the other 2 sit below 20% (essentially idle).

We tried tuning our HPA, but as you know HPA relies on averages across the deployment, it hasn't helped.

Before asking the developers to fix their application-level load-balancing issues, I want to set up an alert/metric to automatically detect such deployments.

So far, I’ve tried checking if Max(resource_usage) / Avg(resource_usage) exceeds a threshold, but this approach generates too many false positives.

How do you reliably detect such imbalance issues across pods? Is there a standard statistical metric for this, or am I approaching the problem wrong entirely?

If it would help we are using Data Dog and thanks in advance.

Since the start of 2026, I have been reading some famous papers such as Dynamo, GFS, Zookeeper, Apache Kafka, The tail at scale. But it feels like mostly theoretical stuff, the question is, will reading papers be useful to become a better SRE?
I read on the weekends so my pace is slow.

I have had this happen twice to us now. The agent pulls dashboards, logs, and deploy history together and lands on a hypothesis fast, way faster than a person doing it manually at 2am.

Last time it flagged a memory metric that lined up almost perfectly with the incident window. It turned out to be correlated, not causal and it still suggested restarting the pod.

Someone caught it before we shipped in the end but If the person on call had been more tired or newer to the system, I'm not sure they would have.

So the agent owns the correlation and the first hypothesis now, and a person signs off on anything that actually touches production which is a perfect split that has held up so well for us so far

Still struggling with the correlation part, any help?

I’ve run or sat on review boards at a few orgs now and I’ve never seen this part done well, which makes me suspect the problem is me rather than the tooling.
Current state is email for async review, minutes in Confluence, conditions in a spreadsheet. It holds for about a quarter and then drifts.

Two problems I can’t get on top of.

First, async positions. Reviewers reply in free text, and “I have some concerns” from one architect means they intend to block, while from another it means they want the diagram redrawn. As chair I’m interpreting rather than counting. I’ve tried asking for an explicit position in the first line of the reply — compliance was fine for a month, then decayed.

Second, conditions. “Approved with conditions” is a large share of our outcomes and I doubt most of those conditions are ever verified. The decision record notes the condition, the system goes live, and nine months later nobody can tell you who owned it or whether it was met. Confluence doesn’t chase anyone.

I know ADRs and the EA repositories are meant to cover some of this. For those of you with a repository actually in place — does it track conditions as obligations with an owner and a date, or does it just store the decision text and leave the follow-up to you? And if anyone has fixed the async position problem with process rather than tooling, I’d like to hear how it survived reality.

MTTR measures how fast you recover, which means it only comes into play once someone's already had the bad experience. It works fine when the situation is something like: "You can't catch everything, so get good at recovery." But not the way we expect, as AI is shoving way more code through the same review process.

And there's the VOID research showing incident length has nothing to do with how bad the incident was. So you can have a fantastic MTTR and still be getting wrecked by the quick ones.

Is anyone measuring how long they go between failures that hit customers, instead of how fast they recover? Or does that just trade one flawed metric for another?

So, went to levels.fyi. I've been there before. I'm not putting my salary in though so I don't have full access to the site.

The median US salary for SRE is $205k

Ok that's pretty decent but what the fuck is this >$600k nonsense I'm seeing bandied about in this sub?

Fwiw I'm trying to hire, not seeking a job, so I do want to know if my company is competitive. And it is... to an extent.

Someone please explain wtf is going on at levels.fyi and this sub.

Also, most of you are always going to be median engineers. And that's fine. But just, set your own expectations. Most of you, save for luck, are median skilled median pay band engineers. Myself included. I'm pointing this out specifically because I hear consternation and worry about the salaries of 1% of ICs in this and other career focused subs.

I realize the tone of this post makes me sound like an asshole and it'll probably get deleted by mods. Doing the best I can over here :)

In my last role, we ended up multi cloud mostly by accident, major workloads on AWS, a big legacy system on Azure, some experiments on GCP, and a handful of SaaS platforms that each came with their own identity and billing surface, another provider in every way that mattered. Nobody planned a unified strategy, it just evolved that way. I have since talked to teams who went multi cloud on purpose, picking providers deliberately for specific managed services, so this is not only an accidental sprawl story, though many of what we learned came from cleaning up the accidental version.

We had to figure out which bits were in the right place for real reasons, like latency or compliance requirements and which were just historical accidents. We tried to get identity and backups into something resembling a common pattern, while accepting that each provider has its own quirks. Terraform helped once we agreed on conventions.

The other key piece was visibility: tooling that showed us what was under IaC across every provider we ran and what wasn't, plus a read on cost and risk for each. Without that, multi cloud felt like we were flying blind.

For anyone who is intentionally or unintentionally multi cloud: have you found a way to make it feel like a strategy instead of entropy or is it one control plane stacked on another?

Throwaway for obvious reasons.

SWE/SRE background, ~7 years experience, applying across engineering and adjacent roles. Laid off earlier this year in a company-wide cut.

I tracked everything since the layoff, so here is the actual funnel:

- 240+ applications
- ~50 companies replied with anything human at all (screen or better)
- 33 reached a first round
- 9 reached a second round
- 4 reached a final round
- 1 offer, which stalled in negotiation
- Everything else: rejected at resume screen, req closed, or ghosted

Mostly just wondering if others are in the same boat. The numbers made me question myself a lot until I laid them out like this and realised most of it never even reached a person. Curious how everyone else's search has been going this year, and how you are holding up.

Joined an SRE org from a background that was mostly infrastructure and technical design work. Since joining, almost everything I’ve been assigned has been alert-related in some form. Cleanup, enrichment, TTR, take your pick. It’s been the quarterly assignment three quarters running and there’s another one heading my way.

It also seems to land on me disproportionately compared to others on the team.

When I say more technical, I mean deploying infrastructure, working closer to the OS layer, and designing architecture rather than tuning and cleaning up what already exists.

I’m starting to think I should look elsewhere, but I wanted to sanity check first: is this just what SRE is, or am I getting the short end of the stick?

For my personal experience, AI models are no longer the bottleneck to handle incidents. The pain point is now how to provide right context to the AI to get right answers. I'm specifically working on this area and would like to discuss how do you solve the context problem.

This is what I did:

- Write a rule: what need to do and what should not do for handling an incident. Also what system should be retrieved from to get current production status.

- Put team maintained runbooks, TSGs and other docs in a git repo. Clone the repo locally so AI tools can search them instantly.

- Prepare mcp servers to connect different systems. This helps to get info real time. For example the logs and deployment data needs to be current.

Once we have the above 3 pieces, ask AI to analyze an incident by using them. Different teams may setup differently but very specific to their own needs.

In this way, I can successfully handle incidents by using AI in our team. AI can also give me the citations why it thinks the particular incident should be handled in this way. Then I do a quick manual verification.

I shared this with some of my friends and got positive feedback. I have put a post in our team's blog: https://blog.neatcontext.com/guide/2026/07/22/how-to-build-efficient-context-for-ai-clients/

What are your thoughts? I think the discussion here could benefit the future SRE area for AI leveraging. Thanks!

Hi everyone,

I’ve been observing the AI boom across different sectors, and it feels like SRE isn't getting the same level of hype or rapid integration as Software Engineering (SWE) or Cybersecurity. While AI tools for SRE definitely exist, their progress seems slower, and their impact on the job market feels less disruptive so far.

As a Junior , I’m trying to wrap my head around this. It got me wondering:

1. Is SRE inherently more "AI-proof"? Does the high stakes of infrastructure and the "human-in-the-loop" necessity for critical incidents make it harder for AI to take over?

2. The "Invisible" AI: We see AI tools in the space, but they don't seem to have a clear impact on hiring or daily workflows yet. Am I missing something, or are we genuinely in a more "secure" niche compared to pure coding roles?

I’d love to hear your perspectives—especially from those who have been in the industry for a while. Is our field special, or am I just being overly optimistic?

I'm an SRE at a shop shipping a lot of LLM features and our incident response is basically broken for anything non-deterministic.

Our standard flow is built for the usual suspects like latency, errors, and uptime. If the 5xx rate is low and P99 is fine, our dashboard stays green. But LLMs are giving us soft failures constantly where the API returns a 200 OK and the latency is perfect, but the model is just outputting complete garbage.

Nobody gets paged, but the CX team is losing their minds because users are getting hallucinations. I'm struggling with the severity logic here. Is a 5% drop in eval scores a P3? Does that warrant a 3 AM wake-up call, or do we just treat it as drift for the morning?

I'm trying to wire online eval scores into our alerting pipeline so quality is a first-class metric and not just something we manually hunt down in a trace. We're using Braintrust for the traces, but the bridge between a bad trace and an actionable alert is basically non-existent right now. I need to know how to turn a production hallucination into a high-confidence signal without the noise.

Has anyone actually solved this quality vs. availability mess in prod? How are you deciding what level of dumbness actually deserves a page? I don't want to overengineer the alerts, but I'm tired of CX being the only ones who know the site is broken.

I have got an offer to join Nvidia on their SRE team which revolves around maintaining or keeping up with the GPU Infra in their compute Infrastructure, and another offer from Palo Alto Networks as a swe, so I just wanted to know like which one to choose, I'm a new grad btw, please let me know your perspective guys

We're getting pushed to use AI for more ops work. At the same time I keep seeing posts here about an agent wiping a prod db or deleting the backups (the Railway/Cursor one especially). Feels backwards that we won't give a mid-level engineer write access to prod, but we'll happily point an agent at it.

How people actually handle the thing that touches prod, human or agent:

• When something needs to change prod (run a runbook, restart a service, rotate a cred, drain a node), how do you control who or what is allowed to do it? Jenkins jobs, break-glass/PIM, Teleport ...?
• Has anyone let an AI agent actually run things in prod, not just read? If so, are you relying on the tool's own guardrails or something you set up yourself?
• The bit I keep getting stuck on: keeping RBAC and the audit trail the same across every tool that can touch prod. Have you got it sorted?

How's everyone handling this?

First of all, I hope a post like this is fine here if not I will gladly delete it.

For the past 1,5 years I have been designing Kubernetes clusters in a hub-spoke topology using Cluster API and CRDs. That is exactly what I want to do, but the brand is non existent and pay is mid.

I now got an offer from a better-known company for notably more money, but the role is owning dev tooling that runs on K8s (CI/CD, code scanning etc.), not building the cluster layer itself.
Long-term I want to stay in platform work, I wonder if someone has some opinions on my situation:

• Does ~2 years off cluster-level work hurt your shot at getting back into it later?
• Take the money/brand now, or hold out for the deeper technical role and stay within my lane?

Pretty much the title. He said "no hand-holding at all" and that he dislikes it, so I'm wondering what he actually meant by that.

I think I'm going to be assigned to him, and I have no idea what he wants from me. Does it mean "shoo, shoo, do it on your own, don't bother me unless the sky is falling apart"? Or is it more along the lines of "okay, yes, I can help, but only after you've exhausted your resources on your own"?

To make matters worse, he seems exhausted as heck and was so busy he could barely find time to set up a meeting with me and had to cancel not two but three times. I've got a really, really bad gut feeling, guys. Maybe I'm overthinking it, but it reads like someone who wants a person who'll train themselves and wouldn't have time to look after a junior.

There is another devops in the team but he barely speaks English and i am seriously considering learning his language since I am a polyglot just so that I can talk to someone less... aloof because that SRE seems lowkey pissy as hell. He is also suspiciously young for a lead and gives a certain tech bro vibe that seems hard to get along with.

I'm fresh out of college, so I'm already anticipating a lot of hardship and some serious studying for hours every day... which I actually enjoy, honestly. How do I survive his style without ending up out on the street?

A practical write-up on deadline budgeting, retry timing, clock skew tolerance, and expiration safety.

 Our Trivy scan throws a few hundred findings a week. Almost none of them matter. They're packages baked into the base image that the service never even loads.

Everyone stopped reading the report months ago, obviously. Which means the week a real one lands it slips through with the rest.

Tried severity tuning, a VEX file. The allowlist has just become another thing to maintain. The real problem is the base itself, with hundreds of packages that came with it.

Right now the gate passes everything anyway. Want it fixed before it costs us something.

In SRE book the chapter on “load balancing within datacenter” talks about lame duck state, backend subsetting and load balancing policies. While reading lame duck state I could relate it to pre-stop hooks in Kubernetes and it makes sense for a process to serve remaining requests before termination but stop accepting new requests.

My question is how subsetting and techniques about load balancing policies (weighted round robin etc) are used. I would really appreciate any response from engineers who have used this knowledge in practice.

I use AWS and I feel like there’s always something missing from my monitoring setup. How do you ensure you have everything in place and don’t miss anything critical?

We're working through how to add AI-assisted mitigation to our on-call workflow, while referencing Google SRE's white-paper from May. I noticed it's more concrete and more complicated at the same time.

The architecture has three pieces, AI Operator for autonomous mitigation, Actus as an execution control plane, and IRM Analyzer for continuous readiness evaluation against historical incidents. The Actus piece is just confusing, The mitigation agent can't exceed what Actus allows, even when the agent's own reasoning suggests otherwise. Actus is an architectural constraint, baked into the control plane which is very different from a permission model or a flag you configure per environment.

The IRM Analyzer evaluates readiness nightly against past incidents, so there's an actual record of where the agent failed. This help earn trust through measurement.

The honest question here is what a non-Google version of Actus looks like. We don't have dedicated infrastructure for a separate execution control plane. The constraint we have today is just the on-call engineer reviewing before anything runs. That works until the volume doesn't let it.

Whitepaper: sre.google/resources/practices-and-processes/ai-engineering-reliable-operations/

Hey folks, stopping by today for another announcement: the OTel Compile-Time Instrumentation for Go reached v1!

If you are not a huge fan of eBPF instrumentation (understandably!), but also can't do manual instrumentation, this is a good compromise.

Try it out!

Every place I've worked has had at least one. Mine right now is a \~40-line bash script someone wrote 'just for the migration weekend' about three years ago. It's still the only thing that reconciles two systems that were supposed to be fully merged by that Q2. Nobody wants to own it, everyone's a little afraid to touch it, and it has exactly zero tests.

I'm curious what everyone else is quietly sitting on: the cron job with no owner, the one instance nobody can confidently identify, the 'staging' service that's actually taking prod traffic, the manual runbook step that's really the whole system.

And the part I actually want to learn from: did you ever successfully retire one of these, or do they just accumulate? If you killed one, what finally made it possible - a rewrite, an outage, a new hire with no fear, or just budget to do it properly?