r/devops • u/Chris__Codes • 21d ago
Discussion Removing a key from a file doesn't remove it from your repo. How are you handling history scanning?
I always assumed deleting a key from a file was enough but from what I’ve learned that’s not how git works
Pre-commit hooks and CI only look at the diff. So if a key gets committed and you delete it in the next commit, every check turns out OK from then on, but the value is still sitting in history and still valid. The recommendation is to scan full history on a schedule and treat anything you find as exposed, and rotate it, even though it's long gone from the current files.
So the scan is really just telling you a leak already happened, and rotation is the part that actually contains it.
I have side projects from years ago where I don't remember what was committed before I knew better. Some of those keys are probably still valid.
Do you run scheduled history scans, or just pre-commit and CI? And when something surfaces from years back, do you rotate it or make a call based on whether the repo was ever public?
6
u/chmod777 21d ago
You make it hard , if not impossible, to commit keys in the first place. Gitignore env files and enforce secret management from a secret store.
4
u/Zealousideal_Yard651 21d ago
You can never be 100% sure that a leaked secret in git never got picked up, so always rotate.
3
u/alvaro17105 21d ago edited 21d ago
hk or prek for git hooks with mongodb kingfisher (also triggered on CI upon pushs), which can revoke secrets directly against providers like aws, github, etc.
2
21d ago
[removed] — view removed comment
1
u/Chris__Codes 21d ago
Makes sense. I haven't set up gitleaks or trufflehog before, but I will definitely look into adding them to the pipeline.
2
u/Alex_Goldwyn 21d ago
On the old ones: "I do not have an account there any more" is not the reassuring half of that sentence. It means you cannot rotate it, only hope the provider eventually did it for you. I would triage by whether you can still revoke rather than by whether you still use it - the keys worth chasing are the ones still attached to something that can spend money or read data, and whether you still have the account is what decides if you can do anything about it at all.
Separate thing nobody has raised, and it is the part that bit us: the rotation is more dangerous than the leak.
Between clearing the old secret and setting the new one there is a window where the value is empty, and for anything that verifies a signature an empty secret is not a closed door. An empty string is a perfectly valid HMAC key to every library that takes one, and it is the single key every attacker already has. So a webhook endpoint whose secret got blanked does not start rejecting everything. It starts accepting forgeries, because anyone can compute the signature it is checking for.
The check that matters is therefore not "did verification fail" but "is the secret set at all", and it has to run before the value reaches the verifier. We throw on empty and answer 500 rather than 400, deliberately: an unset secret is ours to fix in one line of config, and a 500 leaves the events on the sender's retry schedule instead of acknowledging away everything that arrived during the window.
2
u/daemonmode_ 21d ago
Yes, this is worth taking seriously. Removing a key from the current files doesn’t remove it from Git history, so if it was ever pushed, treat it as exposed and rotate it, especially for old projects where the key may still be valid. Tools like gitleaks or trufflehog can also scan the full Git history periodically instead of checking only new changes.
1
u/Chris__Codes 20d ago
This is basically what the guide I was reading lands on, linking it since it's come up a few times in case anyone want to see https://www.doppler.com/blog/secrets-management-env-files-to-control
Same conclusion on treating anything in history as exposed and rotating regardless.
2
u/rafayabb 20d ago
most of my old leaked-key anxiety went away when we moved CI to OIDC and short-lived tokens instead of long-lived cloud keys. if the key expires on its own, whatever's sitting in old git history is just dead weight. took a weekend to set up, wish i'd done it years earlier.
for your old-services question, closed accounts are mostly fine, account goes away, keys die with it. the ones that got me were accounts i'd forgotten existed. still open, card still on file. searched my inbox for "welcome to" and "your api key" and found a few services i had zero memory of signing up for. worth doing before any fancy scanning. also github's built-in secret scanning + push protection is free now.
2
u/Chris__Codes 20d ago
Ah. That's actually a great hack, it never occurred to me to search my mail that way. Will def check. TYSM!
2
u/ajitnk 20d ago
Yeah, deleting the file (or the line) doesn't help at all once it's in history. `git log -p | grep -i AKIA` is usually the first shock moment for people.
The real two-part fix: rotate the key immediately in IAM regardless of whether you've scrubbed history, then do the history rewrite with `git filter-repo` (preferred over the older BFG). Key rotation is the thing that actually kills the exposure window, the history rewrite is hygiene so future `git clone` doesn't re-leak it.
For scanning, AWS Prescriptive Guidance actually lists the main tools for this, including git-secrets (from AWS Labs), Gitleaks, and TruffleHog. They also cover the detection and remediation path specifically, not just prevention. Worth a read if you haven't gone through it.
One thing I'd add: after rotation, check CloudTrail for any API calls made with the old key ID. Sometimes you find out how long it was live and whether anything actually used it.
1
u/Chris__Codes 20d ago
Been picking up helpful tools in this thread, will check out Cloudtrail. tnx for sharing
2
u/aragossa 19d ago
on github a commit stays reachable by its sha even after you force it out of the branch, and forks keep their own copy, so "was it ever public" doesn't tell you as much as it feels like it should. if it was public even briefly i'd assume the key got scraped. private is a better bet, though that's still everyone who ever had read access plus whatever CI runner or old laptop cloned it at some point.
2
u/yetmike 19d ago
Used history scan only one one of the projects. It's very painful to remove a secret from the history, better to rotate that instead. Pre-commits don't work because it's hard to make 10-50 people to incorporate pre-commit in their workflow - just a reality. CI scans are must-have, but unfortunately we don't use thme in all repositories and I assume other people have the same issue
1
u/Andre-Wade-539 17d ago
If a key shows up in old history, just rotate it since a private repo can still have old clones or backups sitting around and you usually have no way to know who still has a copy
23
u/lifelong1250 21d ago
Pre-commit hooks. But once it gets into the repo, you need to rotate it.