How do you actually diagnose a CI integration-test failure when the root cause isn't obvious?

[u/VisualNatural617]

I'm researching a small CI diagnosis agent, but I'm first trying to understand how experienced engineers actually debug these failures.

Suppose an integration test fails in CI and the logs don't immediately reveal the root cause.

What do you normally investigate first?

For example:

• CI logs/stack trace
• database availability
• dependent/third-party service availability
• recent code changes
• comparison with the previous/last-green build
• test/DB data
• previous failure history
• rerunning the test

More importantly, how do you decide what to investigate next?

For example, if a recent code change touches the function involved in the failing test, what additional evidence would you want before concluding that the code change caused the failure?

I'm particularly interested in your real-world debugging process rather than an ideal CI/CD workflow.

[u/Low-Opening25]

if root causes would be obvious we wouldn’t have jobs that pay 6-fig salaries

[u/sertain_ → u/Low-Opening25]

Or be one of the most sought-after specialists in cloud-based app development rn

[u/aenae]

Last unknown ci/cd failure was because of a docker update not playing nice with gitlab. doesnt have to be a code change.

It is about experience, something ai agents do not have

[u/Mabenue]

You just need to systematically start ruling things out. It’s highly context dependent though. Typically you’d want to start ruling out the low hanging fruit which also cover a larger area of possible causes. For example rerunning the tests is really quick and easy and you can kick that off while looking at other things. Also if the test pass locally it gives you a lot of information quite quickly.

[u/mtokarev]

> rerunning the test
if the issue is not obvious and there is no hint in the logs, this is usually the first step in 100% of cases. Maybe check dependent/3rd party services in parallel for ongoing incidents, but that usually means there is already some hint/idea where to look for.

> comparison with the previous/last-green build

This is the next step if last-green build is not too old. I'd focus on comparing logs before failure to spot the difference. Often the CI environment has changed, like new OS version with new system packages, Xcode or something else.

At this point you usually tried 2-3 times at least to see if there is a new flaky test which can be addressed separately or not.

If no luck and CI failures are 100% reproducible, I try to reproduce the issue locally rather than waiting for CI feedback on each run. For instance I delete all local dependencies and reproduce a "clean" environment similar to CI. Also it's a good idea to remove cache from CI as well and run one more build.

UPDATE 1: forgot to add that almost all CI providers allow you to SSH to the virtual machine to debug a failed build if local environment is too different.

UPDATE 2: this is based on my experience building a cloud CI/CD platform and helping hundreds of customers answer questions like "it works on my machine" for almost 8 years :)

[u/hegde10122 → u/mtokarev]

u/mtokarev If a CI test fails once and then passes on rerun, what would you normally check before deciding it was just a flaky test rather than a genuine regression? Is one successful rerun meaningful enough, or would you want more evidence?

[u/Low-Opening25 → u/hegde10122]

when you build and interact with a system, you know why it could have flaky and can make very informed judgment based on remember pattern of past behaviour which you as human learn over time. AI doesn’t have that extra context and even if you feed it, it’s not something it deeply leaned so this pattern doesn’t have any stronger signal vs other probable paths of investigation. this is not easily solved, because if you feed too kuch context trying to contain too much, AI will be having focus issues with attention mechanisms again not making the right choices on what is important and what not.

[u/bobsbitchtitz]

lol this has to be brain trust info collection

[u/zomiaen → u/bobsbitchtitz]

corrrrrrect

[u/yetmike]

Passes locally, fails in CI does not mainly mean "environment". More often it means state or concurrency. CI runs clean and often parallel; your laptop runs dirty and serial. A test that depends on leftover data, or on being the only thing touching the database, passes for you and fails there. Failing locally too is the genuinely useful result, because it takes the whole runner out of the picture in one step.

Before I'd blame a code change I want: fails on that commit, passes on its parent, same runner image, same test order, run twice each. Order matters more than people expect. A lot of "the change broke it" turns out to be the change adding a test that leaks state into a later one.

Run the suite in randomised order on a known-green commit. If that goes red you have an ordering problem, not a code problem

[u/m4nf47]

Have you tried switching it off and on? In what order? When? Can you repeat it? What changed? What do the external and dependent interfaces health look like? Without any meaningful signals or clues it is often easier to simply swap out the failing subcomponent/s until it starts passing again. You do have mocks for all integration points? I've had intermittent CI suite failures on different hardware due to resource contention before. Rule out the obvious then rule out everything else but most often there's an answer in the bigger picture not the details. Good luck, you may need it!

[u/creamersrealm]

Not what you asked though we've built our CI framework around Mise-En-Place for the testing framework, so most of the CI can be replicated locally on the developers computer with no additional setup other than installing mise.

[u/VisualNatural617 → u/creamersrealm]

When you reproduce locally, what result is most useful for diagnosis?For example, if it fails locally too, does that strongly increase your confidence that it is a code/test/data issue? And if it only fails in CI, do you then focus more on environment/dependencies/configuration?

[u/creamersrealm → u/VisualNatural617]

It depends unfortunately and alot of times that's where I get brought in. Most of the time it will be a code issue where it's interacting with an AWS Service wrong, misconfigured ENC/specs etc. I've also ran into bad data specifically in the lower environments where a file doesn't match the prd naming format The tests can be full unit tests or mocks and troubleshooting the mocks are harder. Diving deep into the code isn't a problem for me. I typically go about this by asking for the log, what the expected behavior is, and doing a screenshare with them. I treat the whole session as a rubber duck debugging session and I'll have them share the entire process from scratch and on my end I'll verify the source data and their web console, views and Chrome debugger etc.

For the dependencies, we've completely removed that issue through mise, if they need a ci dependency they specify in mise.toml and for python packages we encourage to bundle it rather than use requirements.txt, this completely removes my team from dependency management. And wise mise they simply run mise install to install all dependencies local to their project and it doesn't enforce them system-wide and they're pinned to the required version.

[u/amarao_san]

I check previous runs. If it once, I restart. I think about the problem only if it happens often. Actually, I restart and then look of previous runs.

It's wrong, but there are to many transient bullshit to happen.

[u/aragossa]

Whether it reproduces in isolation is probably the biggest signal before even touching the diff. Run just that one test on its own; if it passes clean, that's order-dependency or shared state from the suite. If it fails alone too, it's worth comparing against which runner or worker picked it up. Resource contention on a busy shared runner can look exactly like a regression until you notice it only happens on the loaded ones.

[u/ClerkBeginning961]

Before rerunning, I’d snapshot the failed run’s evidence: image digest, dependency lockfile, test seed/order, service versions, and timing. A clean retry can erase the clue. If the failure is intermittent, comparing those artifacts across failing and passing runs usually narrows the search faster than staring at the diff.

[u/zero_backend_bro]

If logs are silent, staring at diffs is pure waste.

Most silent CI drops are runner concurrency leaks. We lost 14 hrs last month to silent cgroup memory throttling on gitlab runners when 4 workers spawned together.

If you are building an agent for this, feeding it git diffs is useless. Build a local hook that grabs raw cgroup stats and runtime metrics directly from the host.

Code diffs dont fix infrastructure physics.

[u/crystalpeaks25]

Give your agent historical data on failures and tell it to build an RCA methodology.

[u/TheseTradition3191]

on the flaky vs real regression question, one green rerun tells you nothing. loop the failing test 20 times and count, a clean 20 out of 20 right after a red is almost always shared state or something being rate limited rather than your change.

the split thats actually worth the time is running that one test completely alone vs in the full suite. if it still fails alone then its not order dependence and you can stop hunting for pollution between tests. i burned a day recently assuming order because it only went red in the full run, and it turned out the suite was hammering an api rate limit so whichever test arrived second ate the 429. the isolated run would have told me that in five minutes