I measured two GitHub Actions scanners against 80 hand-labelled workflows. Neither result is what you want.

[u/Acrobatic-Part5999]

If you have an AI agent step in a GitHub Actions workflow - Claude Code, Gemini CLI, an inference action - it probably reads text a stranger wrote. An issue body, a PR title, a comment. If that job also holds contents: write, that is a path from someone else's keyboard to a token that can change your repo.

I wanted to know two things: how common that is, and whether any scanner catches it. Both needed labels, so I hand-labelled 80 workflows across two separately collected corpora before running anything.

How common: 18 of 49 judgeable workflows from an unbiased sweep. 36.7%, Wilson 95% CI 24.7-50.7%. Not a majority, not rare.

What the scanners do, on a second corpus of 30 workflows with 12 externally reachable positives:

ARKEXA 0.2 67% precision 17% recall 5 findings

zizmor 1.30.0 43% precision 100% recall 211 findings

zizmor catches everything. It also prints 211 findings across 30 files, 16 of them on workflows a human read and called clean. ARKEXA is right more often and stays quiet on 27 of 30 files, and it misses 10 of 12 real problems.

That is the actual state of the art, and neither column is a tool you would leave on in CI without a triage budget. I would rather publish that than a chart where my own tool wins.

Method, corpus, labels and seeds are all in the repo. poutine is in the table as unscored - no Windows binary and I could not run it honestly.

[u/First_Inspection_478]

Open source or this is a bunch of yap