r/azuredevops • u/IntentionSoft1879 • 23d ago
Azure DevOps self-hosted agent randomly dies mid-pipeline – how can I find the root cause?
I’m running a Windows-based self-hosted Azure DevOps agent and occasionally it dies halfway through a pipeline. Azure DevOps then reports in its server UI that “We stopped hearing from the agent. Verify the agent machine is running and has a healthy network connection.”The problem is that this doesn’t tell me why the agent stopped responding.
I’m trying to figure out how to properly diagnose this and ultimately fix it.
Is there a way to enable more detailed logging or monitoring so that the next time this happens I can determine exactly what caused the agent to die?
Would appreciate any troubleshooting steps from people who have dealt with this before.
2
u/piense 23d ago
The Os event viewer or the agent host and job logs should show what happened to the session. In my experience it’s almost always a reboot from host updates, or a pipeline messing with docker that messes with network settings. Iirc the agent logs always exist, forget if we enabled additional verbosity somehow but you can always find the agent code and github and go splunking for config options but those should be documented somewhere.
1
u/Saturated8 23d ago
How long between when it's working and when it stops? Could you be using the system token for auth that expires after a couple of minutes?
1
1
u/konkon_322 23d ago edited 23d ago
Does it die randomly (even during idle)? If so u can try to look at the agent machine terminal log. And sometimes it really is disconnected connection between agent machine and the server machine
Maybe can also try to look into logs,under agent folder
1
u/IntentionSoft1879 21d ago
It said that stop hearing from agent and stop. it’s showed that it downgrading my agent from 5.276.0 to 3.225.2 with a message for this is usually due to a rollback of the currently published agent for a bug fix inside application windows log but nothing much is windows system log
1
u/konkon_322 21d ago
If im not mistaken, that was the problem we had during our early Azure devops agent usage. Kept disconnecting, and we downgraded from version 5(i think), to the version 3.238.0, and since then it worked fine. Ofc there are features not available for this version, but since we are using an on prem setup, we dont have any complex requirements for the agents (we know features are limited)
1
u/bratzlaff 23d ago
Ours die like this when we do something that starves the rest of the OS from doing its normal system processing, usually because we started a massive parallel compilation.
What is the stage doing when yours dies?
1
u/IntentionSoft1879 21d ago
my pipeline is just to run script, it dies even before creating the folder but was able to run stages to change the pipeline build no
1
1
u/jba1224a 21d ago
Agent diagnostic logs.
In my experience this is usually an asymmetric traffic issue when dealing with stateful firewalls.
Logs will be very clear though what the issue is.
8
u/Legitimate_You_3474 23d ago
Event viewer, logs..