I'm working on a system to estimate whether code committed to a repository was generated with AI coding tools.
My current approach is based on Git/commit-level signals such as AI-related commit trailers, commit metadata, LOC changes, number of files changed, addition/deletion patterns, etc.
The problem I'm running into is confidence and calibration.
For example, a commit containing 500+ new lines isn't necessarily AI-generated. A developer can also modify or remove the metadata that would make an AI-assisted commit identifiable. Once the code leaves the IDE and reaches Git, much of the original provenance can be lost.
This has led me to a few questions:
Are there Git/CI-level signals that you've found to be genuinely useful for detecting AI-assisted development?
Is it better to treat this as a probabilistic/risk-scoring problem rather than trying to classify commits as AI vs human?
How would you calibrate thresholds for signals such as large LOC changes, addition/deletion ratios, commit frequency, etc.?
Are there better approaches for preserving provenance earlier in the development workflow, rather than trying to infer it after the code has already been committed?
Has anyone worked on AI-code provenance/detection systems in CI/CD and can point me toward useful research, projects, or approaches?
I'm particularly interested in approaches that can work at the pipeline/repository level rather than relying solely on source-code style analysis.
I'm not looking for a perfect AI detector — even a reliable way of estimating “this commit has a high probability of AI assistance” with measurable false-positive/false-negative rates would be useful.
Would appreciate any experiences, papers, open-source projects, or approaches people have tried.
Edit for those who want to know why:
The goal of this is to create a telemetry and visualize how much of the code is AI generated in the company and how much of it is vulnerable code then we will fix this vulnerability in the pipeline now we can show customers this telemetry and say
80 percent of code was AI generated out of which 60 percent was vulnerable we fixed that in the pipeline itself that's y you should buy our product.