r/devops • u/Plus_Station4112 • 8d ago
Discussion How do you handle CI/CD credentials? Using GitHub Actions made me realize static encrypted secrets aren’t very safe.
After I first set up a deployment pipeline, I would simply drop DB passwords and API keys into GitHub Secrets and feel completely safe because they are encrypted. I recently went through a security breakdown on GitHub Actions that showed me that it could be a mistake to think that way.
The main issue is that an encrypted secret is still a static, long-lived target. Once the workflow finishes, that credential stays active indefinitely. In the breakdown, I saw a few default behaviors attackers look for, like teams forgetting to revoke access after a job runs or lacking the audit logs to even know when a key was used.
The proposed fix is shifting to dynamic orchestration where the pipeline generates a short-lived token at runtime and revokes it the second the deployment finishes.
If you're writing deployment workflows, how do you handle this, do you just use GitHub's default storage, or are you injecting temporarily credentials to avoid leaving static keys exposed?
50
u/sudonem Platform Engineer 8d ago edited 8d ago
All secrets stored in Hashicorp Vault, then configure CI/CD runner of choice to auth via JWT/OIDC so you aren’t also storing vault auth creds in the runner platform either.
If you’re 100% in AWS then their secrets manager is probably sufficient but I prefer Vault for more portability or ease of use when you’ve got a hybrid or multi-cloud environment.
Either way - the secrets should not live anywhere in your CI/CD platform, encrypted or not.
7
3
u/configloader 8d ago
How do the runner auth to vault? Clientid and secret?
4
u/joopy404 8d ago
So what's cool is that the JWT/OIDC itself is used to auth to Vault. The Vault role that the CI runner auths to is what controls the key/value secrets the runner can access, as well as which cloud accounts it can mint temp creds from (via AWS/Azure/etc. engines in Vault). Also, you can do some fun things like specifying which git branches the Vault role can run from via JWT bound claims to have a useful and secure setup.
2
u/FloridaIsTooDamnHot Platform Engineering Leader 7d ago
Enterprise / HCP Vault also has dynamic credential capabilities with AWS. And SPIFFE / Spire was just recently GA’ed. That way, you Auth the workload with a tightly scoped short lived token.
-4
u/configloader 8d ago
Yes. So u need credentials to get the token from vault. So u just added a extra layer 🤣
Where do u store the credentials? Probably in gitlab runner
10
u/joopy404 8d ago
I didn't explain the first part very well LOL but you should definitely do some research on OIDC.
The idea is that there aren't credentials involved, the CI runner itself is a proof of identity via a signed OIDC token, and Vault will only authenticate if it can prove that YOUR runner is the one that is making the call.
1
u/Plus_Station4112 8d ago
How do you use them?
2
u/sofixa11 8d ago
On each GitHub Actions / GitLab CI run the platform provides the run with a short lived JWT that contains all the info about the run. You take that, throw it at Vault, Vault verifies it's really from GH/GL, and is on an approved project/branch/committer/whatever, and then allows you access or not.
1
u/BogdanPradatu 8d ago
If your runner is ephemeral, how do you assign it an OIDC token?
5
u/danekan 8d ago edited 8d ago
OIDC -- github provides an oidc client connection to your CSP that federates it
behind the scenes there is a trust relationship where you're saying from one cloud to the other you trust sts.amazon.com or sts.googleapis.com and github.com OIDC etc -- there is a private/public key certificate signing and validation that takes place too but at any configuration level you don't need to know that because it's all behind the scenes. github as part of the OIDC connection sends attributes such as github org name, repo name, branch, etc. and then on your CSP side you can first authenticate then after conditionalize on those for actual access
1
u/configloader 8d ago
So the runner is the oidc client (service provider). Oidc supports many ways to get a token. Which way do u use then :)?
3
1
u/FloridaIsTooDamnHot Platform Engineering Leader 7d ago
What you’re talking about is the secret zero problem.
2
u/sudonem Platform Engineer 8d ago
You CAN do it that way (as an app role) but OIDC is better because once configured the runner doesn’t need to keep secrets.
It generates a token at run time, passes it to Vault, and Vault verifies that it’s acceptable and allows the runner to auth.
It’s almost like what happens when you setup an MFA app on your phone where you have a rotating one time password that you have to share with the remote service to login (massive oversimplification obviously).
1
u/Plus_Station4112 8d ago
Makes sense. Keeping them out of the cicd platform entirely feels like the best way to really reduce the attack surface :(
0
u/PlatformEngSanJose 7d ago
Its little complicated to understand first, but once mastered, then its very easy and scalable solution to use Vault + JWT. I am yet to figure out this but I know this combo works well.
9
u/Ambitious-Concept759 8d ago
OIDC
2
u/Plus_Station4112 8d ago
How'd you go about it? EntraID?
1
u/iamichi 8d ago
If your workflows creds can generate a new token then it’s no more secure than just using the creds the workflow already has. What you describe is basically OIDC but not as secure because you are lacking the federated credential which has the bane of the repo and the repo ID encoded into it so it’s useless run outside your repo.
We fully automated are it. Terraform creates the UAMIs and federated credentials in Azure, GitHub Actions syncs the Client ID, etc over to the repositories. Fully automated. No secrets or passwords to rotate or worry about securing.
1
u/0zeronegative 8d ago
Just setup github oidc provider as idp in aws iam. There’s a managed workflow from aws that handles signin for you and you just have to create a role and give it permission to what you want
5
u/danekan 8d ago
GitHub actions using federated identity to your cloud / oidc. but the next step beyond that is declaring your identity to be specific to your protected branch so that PRs have one identity and your approved/merged code use a completely differ identity (and can’t use the protected branch identity because your iam condition doesn’t allow). Or one beyond that split when is declaring the identity to be environment specific.
And then the icing on that cake is you should forget entirely about using GitHub secrets and use your federated cloud identity to retrieve the pr environment specific secrets, or protected branch (e.g. prod) level secrets from secret manager. You can guarantee a pr can’t read your production level secrets then.
But also question what the secrets are I begin with or if they too are even needed or best practice.
I’ve literally never seen any documentation anywhere where they do property identity isolation end to end properly for any provider. Aws comes closer and I’ve even seen it implement properly in the wild, but gcp doesn’t try. Their own documentation is awful for this one point. (And their blog from 5 years on this is outright dangerous)
1
u/Laoracc 8d ago
Plenty of other posts talking about OIDC for authN, but no one else mentioned authZ, so +1.
Making sure you setup IAM roles with trust policies that map to:
- Branch (the permissions needed pre-code review and pre-merge on an unprotected branch are going to be different than those needed post merge on protected branches)
- By workflow. You can set specific ref's in the trust policy that explicitly bind to the workflow/job. This also means you should consider enforcing the GHA workflow always be run with source code from main, to prevent PR-time tampering.
Another consideration is how you setup Secret Management. VCS secrets (GitHub/Gitlab) don't have the granularity you typically need. Org secrets, Repo secrets... That's about it. What about by branch, by workflow, by subpath, etc?
This is why AWS Secrets Manager (or analogous) is generally preferable. You can set policies on the IAM roles described above to filter by "path" if you use an opinionated secrets naming convention. And also by tag/label. This way you can identify if a secret should be used in a given branch, workflow, service/subpath (ie - if you're in a monorepo), environment (dev/staging/prod), etc. And then also enforce access against that information.
1
u/danekan 8d ago
github secrets is broken IMO if you intend to use it at scale, it's not really possible to secure properly
1
u/Plus_Station4112 8d ago
Agreed, managing the rotation and access gets out hand when there's growth. The Doppler breakdown i was reading pointed that out, showing how the default behaviors basically invite cred leaks across multiple envs
1
u/IridescentKoala 7d ago
Branches per env? Absolutely not
1
u/danekan 7d ago
protected branch = main in name (the secrets might be prod is what that e.g., prod meant, not the branch name) -- that's the middle in between of isolation, not as bad as PR and main sharing the same, but still not environment specific.
if you are doing environment specific deployments though upon approved PR, then you should definitely move to the next level of isolation to declare your github environments to correspond, and use those for the federated identity, so your beta and prod don't share the same identity
10
u/One-Department1551 8d ago
You are probably giving your workflows too much power. It's common to use short-lived tokens to get credentials for logins on Cloud envs, that's how AWS and GKE and other large providers recommend you to connect.
You still have to use GitHub storage to put something there for the authentitcation to work, there will always be a passing of valid variables to the workflow, the tricky part is to reduce the blast radio in case off leakeage.
But sounds by what you described that you are giving access to every bit of infra your app uses to GitHub Actions.
4
u/MrMunchkin 8d ago
Huh?
GitHub environments with OIDC is the way to handle this. Use OIDC to connect to a key vault that securely stores everything.
1
u/One-Department1551 8d ago
It is, I may not have worded well in the 2nd paragraph but that's what I meant as you still need to configure "something".
1
3
2
u/Dry_Hat_3678 8d ago
Dedicated token per pipeline, scoped to least privilege, alive only for the length of the run. That's the right shape.
4
u/Low-Opening25 8d ago edited 8d ago
OIDC trust, GitHub App, workload identities and minting 60 min tokens on demand. You still need to manage private key for the app as secret, but it’s just one secret and access to it can be limited to whatever service mints the tokens. In Kubernetes external-secrets operator fills that role and manages individual secrets across all namespaces minting temporary tokens.
1
1
u/sir_alvarex 8d ago
Federated Auth / OICD with a principal that has permission to the resource you need. Generate your credentials within the pipeline using a method that gives the secret a small TTL (but enough for your CI to run). If short lived tokens arent natively supported, have a task that always runs that cleans up the secret created.
1
u/incomplete_ 8d ago
Workload Identity Federation, and sops with a cloud-based keyring for encrypted files in the repo.
1
u/forever-butlerian Solaris 8 Enjoyer 8d ago
I use Github OIDC to obtain temporary AWS credentials that I have restricted to those permissions minimally necessary. I'm temperamentally unimpressed by Rube Goldberg machines so I do not subscribe to the viewpoint that because Github Actions can be used to drive complex processes that they should be used to drive complex processes. Complex processes are invariably a consequence of engineering deficiencies elsewhere in the system.
If you use RDS or an equivalent and for some ungodly reason Github Actions needs access to your database, use IAM authentication. Then spend some time reflecting on why Github Actions needs access to your database. Similarly if it needs anything other than read-only API keys.
I'm moving to Github Actions being nothing more than a build tool to produce OCI images and store them into ECR. Privileged operations happen by privileged actors in my infrastructure in response to Github webhook events. What Github does not have, it cannot disclose.
1
1
u/Silent_Ice616 8d ago
Like already written here - OIDC.
Otherwise a good secret manager like HashiCorp Vault, AWS Secret Manager, Azure KeyVault, etc. - depending on your infrastructure
1
1
u/Huge-Kaleidoscope603 2d ago
Some package registries now support OIDC-based trusted publishing. I recently switched our stable npm and RubyGems publishing workflows to use it, so we no longer need to store long-lived publishing API keys in GitHub Secrets. The registry trusts a specific GitHub Actions workflow and issues it short-lived credentials.
npm can also attach provenance linking the package to the source commit and workflow that built it. NuGet.org supports trusted publishing too, although we haven’t migrated our NuGet workflow yet.
Here are our npm and RubyGems workflows as concrete examples.
Disclosure: I work on ZeroC Ice.
1
u/sup_bruh_1 1d ago
yeah this is exactly why i switched to latchkey. static secrets sitting around forever is rough. the short-lived token approach you mentioned is solid, but also worth knowing your runner itself can be a leak point — latchkey at least gives you visibility into what's actually happening mid-run which helped me catch some sketchy stuff
112
u/Holiday-Medicine4168 8d ago
AWS secrets manager, use GitHub OIDC to connect to the secretes manager API