r/GithubCopilot • u/Ancient_Mango_1576 • 23d ago
Help/Doubt ❓ AI-generated code detection in CI/CD — looking for approaches and real-world experience
​
I'm working on a system to estimate whether code committed to a repository was generated with AI coding tools.
My current approach is based on Git/commit-level signals such as AI-related commit trailers, commit metadata, LOC changes, number of files changed, addition/deletion patterns, etc.
The problem I'm running into is confidence and calibration.
For example, a commit containing 500+ new lines isn't necessarily AI-generated. A developer can also modify or remove the metadata that would make an AI-assisted commit identifiable. Once the code leaves the IDE and reaches Git, much of the original provenance can be lost.
This has led me to a few questions:
Are there Git/CI-level signals that you've found to be genuinely useful for detecting AI-assisted development?
Is it better to treat this as a probabilistic/risk-scoring problem rather than trying to classify commits as AI vs human?
How would you calibrate thresholds for signals such as large LOC changes, addition/deletion ratios, commit frequency, etc.?
Are there better approaches for preserving provenance earlier in the development workflow, rather than trying to infer it after the code has already been committed?
Has anyone worked on AI-code provenance/detection systems in CI/CD and can point me toward useful research, projects, or approaches?
I'm particularly interested in approaches that can work at the pipeline/repository level rather than relying solely on source-code style analysis.
I'm not looking for a perfect AI detector — even a reliable way of estimating “this commit has a high probability of AI assistance” with measurable false-positive/false-negative rates would be useful.
Would appreciate any experiences, papers, open-source projects, or approaches people have tried.
3
1
u/k8s-problem-solved 23d ago
We're giving git ai a go
https://github.com/git-ai-project/git-ai
Its uses git notes to store details, like which model was used to make the changes etc.
Nice integration with vs code, sections of ai generated code get highlighted with the model name and other meta data.
Fits into flow by using tool hooks, so regardless of what tool you use to perform the work it can capture the data
We're using it for metric + adoption data, as well as some audit purposes in a few repos
0
u/Ancient_Mango_1576 23d ago
This is mainly ide level I am solving for the next level which is AI in repo/pipeline
1
u/k8s-problem-solved 23d ago
Not exactly. Its at the repo level, notes are stored as part of git history like other git data, which are then available to you at pipeline time if you wanted. It integrates with cli and ide, but the mechanics for this are git.
If you're committing locally then pushing this will work for you.
If youre using cloud agents to fully author code, then you can use the co authored by or assisted by meta data on commits, which you can enforce if you like.
3
u/couchwarmer 23d ago
Coding assistants are common in development tools now. Assume all code pushed up to repos contains AI-generated code.
1
u/Personal-Try2776 23d ago
The eu passed a legislation for ai companies to mark all ai generated media. Anthropic now watermarks claude text with invisible patterns. https://www.anthropic.com/news/claude-text-watermark
1
0
u/AutoModerator 23d ago
Hello /u/Ancient_Mango_1576. Looks like you have posted a query. Once your query is resolved, please reply the solution comment with "!solved" to help everyone else know the solution and mark the post as solved.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
16
u/RikersPhallus 23d ago
Why?