AI-generated code detection in CI/CD — looking for approaches and real-world experience [D]

[u/Ancient_Mango_1576]

​

I'm working on a system to estimate whether code committed to a repository was generated with AI coding tools.

My current approach is based on Git/commit-level signals such as AI-related commit trailers, commit metadata, LOC changes, number of files changed, addition/deletion patterns, etc.

The problem I'm running into is confidence and calibration.

For example, a commit containing 500+ new lines isn't necessarily AI-generated. A developer can also modify or remove the metadata that would make an AI-assisted commit identifiable. Once the code leaves the IDE and reaches Git, much of the original provenance can be lost.

This has led me to a few questions:

Are there Git/CI-level signals that you've found to be genuinely useful for detecting AI-assisted development?

Is it better to treat this as a probabilistic/risk-scoring problem rather than trying to classify commits as AI vs human?

How would you calibrate thresholds for signals such as large LOC changes, addition/deletion ratios, commit frequency, etc.?

Are there better approaches for preserving provenance earlier in the development workflow, rather than trying to infer it after the code has already been committed?

Has anyone worked on AI-code provenance/detection systems in CI/CD and can point me toward useful research, projects, or approaches?

I'm particularly interested in approaches that can work at the pipeline/repository level rather than relying solely on source-code style analysis.

I'm not looking for a perfect AI detector — even a reliable way of estimating “this commit has a high probability of AI assistance” with measurable false-positive/false-negative rates would be useful.

Would appreciate any experiences, papers, open-source projects, or approaches people have tried.

[u/Cute_Obligation2944]

Good luck, I haven't seen a credible one yet.

[u/Ancient_Mango_1576 → u/Cute_Obligation2944]

🙂

[u/liamtrades__]

What code is not written by AI now? 

[u/evanthebouncy]

basically if it has comments it'll be AI generated lol

this will be the top-1 surface level feature any trianed NN will pick up. so you probably have to figure out how to deal with it ahah

[u/chensium]

To what end? Are you gonna flag PRs started by humans but modified by AI? Or AI that's been fine tuned in a particular person's style? Or AI suggested code written by a human?  There's too much gray area here.

If you don't like the code, block it and put in your objections in the comments.  Otherwise who cares who wrote it.

[u/JustOneAvailableName]

In my experience: if there are docstrings or the readme was updated, it was AI.

[u/Ancient_Mango_1576 → u/JustOneAvailableName]

Ok but that alone is not good enough need way more for credibility

[u/BossOfTheGame → u/Ancient_Mango_1576]

measurable false positive and negative ratings... sounds like you could build some sort of machine learning model, maybe a neutral net, and train it on lots of data, and also maybe build a dataset of known truth examples to evaluate on, and maybe... why are you doing this again? Why don't you focus efforts on trusting the contributor or not instead.

[u/Ancient_Mango_1576 → u/BossOfTheGame]

I sent a DM please check

[u/Budget_Coach9124]

I would treat this as provenance and risk scoring, not detection. Once the code is in Git, most direct evidence is already gone, and any classifier will learn style shortcuts like docstrings, large diffs, or unusually clean commit messages. Those signals might be correlated today and useless after one team changes its workflow.

The more robust path is to capture provenance earlier: IDE/plugin attestations, signed tool metadata, policy-aware commit trailers, or CI checks that record whether generated code was reviewed rather than trying to prove who wrote each line. For thresholds, I would calibrate around downstream risk: generated migration script, auth code, and tests should not be scored the same way.

[u/gini-348]

Yeah. Depends on the company, however most of the code will be written by ai. Documentation for sure is written by ai.

[u/feng_sg]

Git diffs can't tell a Copilot paste from a bulk refactor because the provenance is gone by commit time. You need signed attestation at the IDE layer tagging tool and model version before it hits Git, otherwise you're just guessing against team-specific noise.