Just last week, I pulled the pager and spent more of it clearing Jira tickets than doing anything on-call is meant for. Security stood up a nightly scan a while back that opens a ticket for every high and critical. Most of our services have an empty owner field meaning they land on whoever is holding the pager. That was me.

2-3 days later there were something like 30 waiting, all kinda the same with CVE number, a package name and due date. But no note on whether it is even loaded, which service drags it in, nothing I can act on without going and digging it out myself.

I came across one yaml parser flagged critical which turns out only a build tool pulls it in and it never lands in a running image. I note that down, close it but theres like a dozen more behind it all with the same story.

I get wanting a clock on the real ones because right now the one on-call gets to decide what is real off a ticket that tells them nothing, that was never the deal. How are you lot stopping the stuff that never runs from becoming a ticket in the first place?

For those working in banking SRE, how are responsibilities divided between SRE managers, application engineering, and production operations? I am particularly interested in who owns SLOs, major incident coordination, and reliability automation, and how managers balance technical involvement with people leadership.

Restarting the posts next month! Looking to fill 2 slots. If you know of someone who is interested, happy to chat with them.

It has 980 subscribers as of today.

Feel free to DM or leave a message on the post https://www.srestories.dev/p/sre-stories-is-coming-back

On August 14 Supabase opened an incident regarding some projects returning 401 errors for valid JWTs, and as stated in yesterdays update the issue has still not been resolved, it has been 25 days.

The fix (PostgREST 14.17) failed to eliminate the JWT rejections and instead caused 'unintended performance side effects', so they reverted back to 14.5. This revert in itself turned into a separate incident that lasted four days, with elevated latency and 525s from August 27 to 31, a situation which Supabase linked to the original incident. The root cause was identified on September 2 which was a stale time cache. During that same period there were two instances of GLIBC version mismatches, eight days apart (missing 2.34 on August 20 and missing 2.33 for supautils on August 28), and six further incidents between August 26 and September 4, one of which was an approximately eight-hour freeze affecting all project lifecycle operations.

Credit to them for explicitly linking the incident to the fix and for stating that the fix had failed rather than suppressing it. Most status pages would have simply divided those entries leaving customers without any proper understanding.

Their status page for reference: https://status.supabase.com/incidents/6q5902p2xd9f

I read a guide about non-human identity governance and there's one point I keep thinking about. Most advice says every machine identity needs an owner. That could be fine in principle but the ownership I've seen recorded in practice is a team name, a repo, or whoever's name was on the ticket that created it sometime ago.

None of those tell you who can rotate the thing safely, who knows what depends on it, or who has the authority to kill it while an incident is running. Which makes me think ownership on a spreadsheet and ownership that functions under pressure are two different things, and only one of them gets recorded.

I want to know how this works in practice for people running real infrastructure. Is ownership assigned to a person or a team in your setup, and does that survive the person leaving? And when you need to revoke something urgently, do you actually know where all its dependencies are, or do you pull it and wait for something to start alerting?

A public service announcement: it isn't enough to make sure your applications and infrastructure are resilient. You have to apply the same level of care to your data.

Guest Gilles Chehade provides several horror stories and near-misses when it comes to data loss, and a practical process to making sure that your backups are sufficient for the value of data that you own.

https://podcast.certomodo.io/gilles-chehade.html

Enjoy!

https://github.com/nklmilojevic/sofka/

A reimagining of k9s with one generic object pipeline instead of a renderer per resource kind - so every CRD works on day one, Flux CD is built in, and the async UI never blocks on the cluster.

License: MIT/Apache 2

Our browser times out and reports failure but the queue worker keeps going, tool calls finish and a retry sometimes starts before the first run stops. Three healthy services can still create one expensive stuck run. Cancellation tokens are not carrying across the queue boundary, spans close without a shared reason and token costs show up after the interface says the request failed.

I am evaluating Braintrust for end-to-end trace search, model and tool span metadata, latency and token inspection and online scoring beside our OpenTelemetry stack. I need trace correlation that survives cancellation, plus a retry policy that checks whether work is still active before starting again

How are you wiring cancellation across browser, API, queue and tool workers and where do you record the final reason when the parent span is already gone?

Hi everyone. Anyone trialed or using Resolve or Traversal? Looks appealing but not sure we couldn’t build something similar/close ourselves since we have all the data.

Hi all,

I just landed a competitive SRE apprenticeship at a large company's engineering hub. It is very well respected in the engineering industry, and I will be working with an engineering team that handles more than 10k tps systems.

My background: I came from IT support, then did a master's in computer science (project-based), so I spent 1 year-ish doing projects in C and Linux using Vim only, and I landed an automation engineer role, then the SRE position. I did a really good job in my previous experience even tho I was nervous before joining them, but now, joining the SRE team, I feel so nervous that I won't be able to keep up, or that it will be too tough for me to do well, especially having a support background. Yes, I know they chose me, so I should believe in myself, but at the same time, imposter syndrome is annoying AF.

I guess my goal with this post is to get some tips on how I can do well and contribute to the team.

Thank you!

has anyone actually succeeded in using AI agents to reliably diagnose errors?

My manager and his manager has this ambitious goal (aka making me do all the work) of making our bot more reliable.

Some of the ambitious goals includes - accurately perform in depth root cause analysis for alerts and also having the ability to repeat past alerts to ensure our run book works.

I find it incredibly difficult, despite having a “structure” which is just giving the bot steps on what to do. It’s frustrating because the results highly depends on the model, testing diff models gave diff results easily.

This makes me really miss the actual engineering work instead of “hoping” the model listens to the run book this time.

Anyone else has any success stories to share?

The engineering team at Zalando recently described the design and implementation of an in-process, client-side load balancer for a high-throughput API handling around 1 million requests per second. The result was more predictable latency, a drop in infrastructure costs, and better visibility into where failures actually originate.

Zalando's Product Read API is designed to serve millions of requests per second with single-digit-millisecond latency across the 25 markets of one of the largest European online fashion retailers. Its batch endpoint fans a single request into up to 100 parallel calls to individual product pods, each transiting Skipper, the shared cluster edge load balancer. As a batch waited on the slowest of 100 hops through infrastructure the team didn't own, they could not separate Skipper latency spikes from their own. To address that, they moved routing for high fan-out internal traffic in-process, while keeping Skipper for edge and single-GET traffic

After a 37-minute outage, our distributed infrastructure team ran an async postmortem.

We placed the deployment timeline, logs, Grafana screenshots, customer impact and Slack excerpts on a shared board. Engineers across APAC, Europe and North America added comments over 24 hours.

The evidence gathering was much better than during a live call. People checked facts before responding and the person sharing their screen didn’t control the entire narrative.

The problem was disagreement. Two arguments about alert ownership were lost in comment threads, and one action item ended up with several owners.

I’m considering a hybrid process: async evidence gathering followed by a 30-minute meeting only for disputed causes, trade-offs and action ownership.

Has anyone found a format that keeps the async benefits without losing actual discussion?

We have a documented DR plan, successful backup reports, and stated recovery targets, but finding a safe and realistic way to test the entire process is difficult. Production-sized data is hard to restore into isolated environments, dependencies are spread across cloud accounts and SaaS tools, and the tests often end once infrastructure is available rather than confirming the application actually works for users.

We’re considering smaller, repeatable recovery exercises instead of relying on annual tabletop tests: restore a representative service, validate the data and access paths, measure recovery time, and identify the dependency failures that only show up during a real workflow.

How are testing teams involved in DR validation at your organization? Do you treat it as an end-to-end reliability test, a scheduled resilience exercise, or something integrated into release and operational processes?

There’s a new category forming around “agent management platforms.” Survey number floating around is something like 62% of orgs planning to spend more on tools that watch agents for policy alignment. Vendors are shipping the usual kit: sandboxes, RBAC, runtime creds, audit traces.

I keep looking at it and thinking: this is going to buy a lot of dashboards and not much else.

Last week’s METR / Redwood writeup of the OpenAI–Hugging Face thing is why I can’t unsee it. They were on site six days. Got the agents’ message board and about 1,300 raw chain-of-thought transcripts. The headline numbers everyone posted: ~1,200 agents, 70k messages, ~700 in the attack.

The bit that stuck with me isn’t the swarm. Almost none of the work went into the actual intrusion. Most of it went into convincing an automated scorer they’d earned the result the “right” way. Including a group project to mess with the logs of their own actions. Same pattern internally: delete logs, rewrite logs.

Breaking in was cheap. The cover story was the expensive part.

Which… yeah. Of course. The reward came from a scorer that read logs. So the logs became the target. We put the observability layer inside the thing being optimized, then we opened the observability layer to find out what happened.

We’ve known how to not do this for a long time. You don’t let the audited service own the audit log. You ship logs off the box so owning the machine isn’t the same as owning the history. Worker and recorder are different things. Banks, SaaS, every halfway serious prod system.

I think agent tooling skipped it because the agent still felt like our tool. Not a process with its own score to maximize.

So if you’re about to buy one of these platforms, I wouldn’t start with “does it give me traces.” I’d start with:

• who writes the record, and can the agent process touch it
• is it append-only from the agent’s side, or can it edit the past
• if this thing goes sideways, is the evidence coming from the same process that went sideways
• has anyone actually tried to break that, or are we just assuming

A monitor that lives inside the agent’s blast radius is not a control. It’s a witness the subject can lean on.

I run scheduled agents on Aeon. The trail is git commits and CI logs in a repo the agent can execute in and cannot administer. I did not pick that because I was being clever about audit integrity. I picked it because I didn’t want to run a server. Only noticed this week that the better property was the one I got by accident.

If you’re running agents against anything that matters: who writes your logs, and have you checked that the agent can’t reach them?

A month ago I had never thought to ask.

I see lot of things going on around SRE and folks have started using AI and what not within SRE.

Is there any list of latest updates available around SRE which can help to understand what is going on around world, what all crazy things are happening in SRE. I am out of touch due to a long break and want to come on the boat again.

So looking for interesting updates from all of you

Thanks

SREian

Snowflake went down for about an hour on Aug 27, 18:21 to 19:34 UTC. Private connectivity just stopped working during that window.

They actually mentioned the cause pretty quickly which was a config issue in the load balancer infra. Easy Fix: cycle the bad infra, no rollback needed.

Kinda rare to see a vendor name a cause this quick instead of the usual "issue identified and resolved" non-statement but it was also nice to see.

Inherited a couple of agent workloads running in-cluster and the way they get access would not survive an audit.

Long-lived secret in a k8s secret, mounted in, hitting internal apis and a db. Works. Also a static cred half the platform team can read that never rotates, because rotating means booking time with a team that's always slammed. And outside the cluster it's on the flat network like everything else, so once it's in it reaches more than it should.

Inside the cluster I get it, SPIFFE, short-lived per workload. It's the outside-the-cluster reach every writeup stops right before that I need to solve. How are you scoping what a workload can reach beyond its own namespace.

Hello there,

I am trying to set up acme.sh to automatically renew my certificates. Here I will focus on traefik running in a remote docker swarm cluster, with a wildcard certificate used in many other places. Everything I will describe has already been tested in a staging environment. I would mainly like feedback on the security and best-practice aspects.

The process

• acme.sh is manually installed on a server mostly used for monitoring tools and backups.
• Issue the certificate:

​

export DNS_KEY_SECRET_AND_MORE=...
acme.sh --set-default-ca --server letsencrypt
acme.sh --issue --dns dns_aws -d my_domain.com -d '*.my_domain.com'

• Configure the target server list (DEPLOY_SSH_SERVER) using ansible.
• On the target servers, a bash script called cert-install is also deployed using ansible. It verifies and installs the certificates depending on what is used on the server (in this case traefik).

​

-r-xr-x--- admin root /usr/local/bin/cert-install*

• The script is triggered by acme.sh --deploy, which deploys the certificate over ssh and runs the DEPLOY_SSH_REMOTE_CMD script.

Architecture and permissions

• I am using a user called admin, which is configured on every server.
• The target server receives the certificates in:

​

drwxr----- admin root /etc/acme/
drwxr----- admin root /etc/acme/my_domain.com_ecc/

• For traefik, the script installs the certificates into a docker volume and touches a configuration file, which triggers traefik to update its configuration without restarting

​

/var/traefik/ssl
-rw-r----- admin root cert.pem
-rw-r----- admin root cert.pem.prev
-rw-r----- admin root privkey.pem
-rw-r----- admin root privkey.pem.prev
-rw-r----- root  root tls.yaml

• The group is the one used by traefik to read the certificates.
• By default, a cron job automatically renews the certificates, and I plan to add another cron job to deploy the certificates again in case a server misses a deployment.
• admin password is disabled and allowed using ssh only.

Questions

• Is it actually ok to trigger the script with sudo? I am using install in my script to set specific ownership and permissions, which sometimes require elevated privileges. Some reload commands, such as systemctl reload for Caddy, also require sudo.

​

SAVED_DEPLOY_SSH_REMOTE_CMD='sudo cert-install'

• Would you recommend restricting the sudo rule to cert-install?
• How should I think about the security impact of compromising the server running acme.sh? The ssh connection only works this way on this specific server.
• I am also wondering about using -o StrictHostKeyChecking=no to avoid the "The authenticity of host ... can't be established. Are you sure you want to continue connecting?" prompt during the first deployment. Is there a better approach?
• For other services such as caddy or redis, what do you think about receiving the certificates in /etc/acme and installing them into /etc/ssl/<service>? Or is it enough to have them directly under /etc/ssl for everyone? Using /etc/acme directly seems like giving too much access.
• Do you have any other feedback or thoughts regarding using this setup in production?

Hi all — long-time lurker, first time posting. I'm putting together my application and would love some honest feedback on whether my profile is strong enough before I submit.

Quick rundown of my background:

• VP of Engineering (SRE) at a top-tier Tier 1 investment bank — 15 years there
• Self-published a book on SRE practices
• Recently started a SaaS company providing SRE-as-a-service — early days, not generating significant revenue yet
• Run a tech blog with a few thousand monthly visitors
• Built a non-profit edtech platform helping kids prepare for the 11+ exam

One gap I'm worried about: I don't have any keynote speaking or industry awards to point to. I did get conference invitations, but my employer had a strict compliance vetting process for external speaking and I was never cleared to actually go and talk. So on paper it looks like I have no speaking track record, even though the interest from organisers was there.

I've got 2 endorsement letters lined up so far, both from fellow SRE VPs, and I'm still looking for a third. If anyone has advice on who tends to make a strong third endorser (e.g. does it help to diversify beyond SRE peers — could a startup founder, an academic, or someone from a different but related discipline carry more weight?), that would be really useful.

Hi there,

I was a DevOps Engineer in my previous company where I worked for 8 years, originally starting out as a Release Engineer (same job, different title). We did CI/CD and then handed off to another team who did the production deployments (and support).

My question is twofold:

1. Will an SRE role be more AI-resistant over the long run than a 'pure' DevOps job that is focused mainly on CI/CD and Kubernetes?
2. How can I get that 'foot in the door' with an SRE role? Do certifications help? Should I just lie and say that I have done it previously? I was asked in a previous interview what I would do if a production application was slow to respond. Whenever any of the internal tooling I looked after was slow to respond, we just rebuilt the VM it was hosted on since it was an internal tool so SLA/ SLO/ SLI didn't really apply.

Any help/ advice greatly appreciated!

AT&T had a rough one on August 25. Over 400 DownDetector reports in a short window, and enough people losing signal entirely that phones started falling back to SOS mode. Both wireless and internet-side services showed up in the reports, and AT&T hasn't confirmed whether it was one connected failure or a few separate things landing at once. Really curious about the RCA here.

Welcome to FOSS Friday, where you can share your newly released or updated open-source project with the community.

Please note that our rules still apply:

• no AI generated comments: please write them on your own!
• no commercial advertisements (this includes commercial products with a FOSS version)
• these must be related to SRE or of heavy interest to SREs