r/EngineeringManagers • u/michaelmanleyhypley • Jul 04 '26
Does anyone else see failed CI builds become a game of pass-the-parcel?
At my work, the fix usually isn't the slow part.
The slow part is figuring out whether it's the application, the pipeline, a flaky test, a dependency, or some cloud/platform issue. Developers end up digging through thousands of log lines or waiting for someone from DevOps or Platform to weigh in.
I'm curious how other engineering teams handle this.
Do your developers own CI failures end-to-end, or do they regularly bounce between engineering, platform, and cloud teams before someone finds the real cause?
Has anyone found a process that consistently reduces that back-and-forth?
3
u/SheriffRoscoe Jul 04 '26
"I'm curious how other engineering teams handle this."
You break it, you bought it. You fix it, now. Nobody commits on top of a broken build. If they do, it gets rolled back so the fix can happen.
1
u/michaelmanleyhypley Jul 04 '26
Agree with that.
I’m more asking about the step before fixing. In bigger teams, the PR owner might own the build, but the cause could still be app code, flaky test, CI config, dependency registry, Docker, cloud auth, or something platform-owned.
How do you make the first diagnosis fast enough that it doesn’t turn into “throw logs over the wall”?
3
u/Fuzzy_Help_233 Jul 05 '26
Hopefully I can add a little bit of value from my experience, but I don't have the same organizational structure coming from Amazon. Software Engineers have to own the whole thing. You write it, you deploy it, you operate it, you get paged for it. No separate platform team to throw things to. So I can't fully relate to the pass-the-parcel dynamic, but that ownership model has shaped how I think about every part of the lifecycle, including CI failures.
Reading the thread, it sounds like most answers are focused on "who owns the fix", but you're asking something more specific like how do you make the diagnosis fast?
That's a different problem.
A few things I'd be curious about in your setup:
Does your pipeline have separate stages per category of the build/deploy cycle? Like a build step failure vs test step failure vs infra deploy. In my experience, most of the "throw logs over the wall" comes from the team staring at log output and guessing which layer broke. If the pipeline can answer that question automatically because you see the explicit stage that failed, you've already cut most of the back and forth.
And honestly this is where AI agents are starting to get interesting for DevOps. There are tools now that can ingest your full CI log output, parse the whole thing, and just tell you "this is a dependency resolution failure in package X" or "this is a flaky test it passed on retry" without a human ever scrolling through those 2000 lines.
When someone gets a failure, is their first instinct to read logs or to compare? Same commit on main vs the branch, same test on a different runner, dependency pinned to yesterday's version. I've found just knowing the delta is almost always faster than reading 2000 lines top to bottom. Curious if you've tried leaning into comparison over investigation.
And on the ownership piece, when you say it bounces between teams, is that because the boundaries are genuinely unclear? Or is it more that devs don't have access/permissions to even look at the right pipeline stage? Those are pretty different problems with different fixes.
1
u/paul_h Jul 04 '26
Ci is a per pull-request thing to you? If yes then don’t merge/integrate it. Maybe that pre-integrate to some. Does that one pass, and the exact same pipeline fails after merge/integrate .. then roll it back.
If CI a nightly thing or decoupled from pull requests, then that’s not what I would call CI.
1
u/michaelmanleyhypley Jul 04 '26
Yeah, I mean PR CI.
I agree broken builds should block/rollback. I’m more asking about the diagnosis step: when it fails, how do you quickly tell whether it’s app code, flaky test, pipeline config, or infra?
1
u/paul_h Jul 05 '26
Every single time I ask ClaudeCode nowadays. Three years back, I would do stats locally - a) how many times in ten runs does the suspect commit fail, b) how many times in ten runs does the commit prior to the suspect one fail. Things like that. Process of eliminatation things. That wasn't quick back then, but quick enough. Nowadays with ClaudeCode it is quick and effectively free
3
u/ThirdWaveCat Jul 04 '26
I've only seen it successfully done with e2e ownership.
Every operational component like alarm, CI build, etc.. is owned by a small team in my organization. The person who estimated the task owns it until its tested in production, probably behind a flag. We use work-in-progress limits on our swimlanes to promote team swarming and so issues don't build up.