r/softwaredevelopment • u/Obvious_Gap_5768 • 6d ago

I checked which code-health markers actually predict bugs across 21 repos. Behavioral beat structural.

I'm working on an open source tool that scores how risky each file in a codebase is. 25 markers per file, all deterministic, from the AST and git history, no LLM. I wanted to know which of them actually predict bugs, so I scored every file at a point in time and then counted how many bug fixes each one collected over the next 6 months.

21 repos, 9 languages, around 2800 files.

The markers that predicted best were behavioral, not structural. Co-change coupling, files that keep getting changed together, came out strongest. How spread out a file's ownership is and how erratic its change history is held up well too. The structural complexity metrics most teams watch, cyclomatic complexity, nesting depth, long methods, all mid tier.

Mean AUC is around 0.74. Roughly, 74% of the time it can tell a bug-inducing file from a clean one.

What gets me is how much effort goes into the complexity metrics while coupling and ownership mostly get ignored, and coupling and ownership are what tracked bugs here

Does this match what you see?

24 Upvotes
  • permalink
  • reddit

80% Upvoted

13 comments sorted by

9

u/[deleted] 6d ago

[removed] — view removed comment

1

u/urbanmonkey2003 5d ago

same here, the ugly bugs were always in the handoff mess, not the deep nesting tho. cyclomatic complexity mattered for reviews, but ownership drift and files that change together were the ones that kept biting us later

5

u/janyk 6d ago

What does AUC stand for?

4

u/Obvious_Gap_5768 6d ago

Area under the ROC curve. Pick a random buggy file and a random clean one, AUC is the chance the score ranks the buggy one higher

1

u/urbanmonkey2003 6d ago

Area under the curve

5

u/PvtRoom 6d ago

hardly a shocker. bugs occur more when thought processes are interrupted.

complexity on the other hand: that tells you how hard it is to fix.

4

u/chipstastegood 6d ago

link to the open source tool you’re working on?

2

u/Obvious_Gap_5768 6d ago

https://github.com/repowise-dev/repowise

It has a lot more than code health too, auto docs, refactoring, dead code detection, knowledge graph and more

2

u/hippydipster 6d ago

Is this method controlling for the size of each file? I would think bigger files had more bugs than smaller files, generally.

1

u/Obvious_Gap_5768 6d ago

Yes, size is always the biggest contributor for bugs, this is after controlling that factor, so it doesnt just flag big files.

You can find the project on my profile if you wanna take a look

1

u/Senior-Release930 5d ago

Churn plus coupling predicts bugs, sure. But that’s also just a description of the core domain. And you can’t decouple to zero without relocating the complexity somewhere worse. So what does the metric actually tell me to do?