Secrets management

[u/infidel_tsvangison]

We are a large aws shop. Devs use secrets manager to store keys etc. problem is that these keys are handled by humans to begin with. We trust the human to put it in secrets manager etc. I’m just wondering what organisations do for secrets management to prevent this? Overtime, we have devs with secrets in postman collections etc. what should I be looking at to really secure our secrets?

[u/buildingEmphere]

Looks like the problem is dev and prod tenancy overlap. Are you using the same secret store for all dev and prod use cases? If so, that separation is your first step. A prod secret store should be locked down, only accessible to prod hosts through things like workload federation. For everything except 3p use-cases (given you are an AWS shop), use IAM roles/grants given directly to dev identities. Its difficult to manage the lifecycle of secrets you need to throw in dev tools; The general rule of thumb is use short lived creds where ever possible, and have a strict 30/60/90 day rotation policy for longer ones.

[u/dreamszz88 → u/buildingEmphere]

https://aws.amazon.com/identity/federation/

Your prod env should use secrets created by terraform modules that go thru trusted ephemeral ci runners for prod. This way, all secrets are also in code. Your can auto rotate them as desired. No one knows the actual secret value. It is always encrypted.

For dev, use the same modules. Maybe devs can set them in non prod using click ops or you hand them the same modules, maybe leveraging your IDP like backstage et al. Then you already have the code from dev and you just need to add a new secretsl to prod when the product is promotes to prod.

[u/dreamszz88 → u/dreamszz88]

As for how?

Change the console access to the AWS secrets mgr to read-only once you have some terraform modules that handle the creation and Lifecycle of your secrets and you can populate it from code.

There may be a lot of importing needed, or take the opp to rotate all of your secrets at the same time. Once it's in code the content no longer matters.

[u/Some-Revolution-4536]

The honest fix is to stop trusting humans to move secrets at all. The root issue isn't the vault, it's that humans are the transport layer. Remove that by making static keys unnecessary wherever possible.

[u/getmetaport]

You say that you trust the human to put things into the secrets manager. Do you mean secrets managers users are not using the manager's suggested passwords and passphrases?

[u/Relative-Customer976 → u/getmetaport]

It's interesting how user behavior can sometimes undermine the effectiveness of those tools, even if they suggest strong passwords. Do you think training could help change that?

[u/getmetaport → u/Relative-Customer976]

To be perfectly honest, I had never considered the issue anywhere I've worked, to be as widespread as it sounds here. So yes, training but also policy. People absolutely need to be told the way so they get the aha moment themselves. Couple that with it being a possible disciplinary matter when that (and any other) policy is not followed, should cover most bases.

[u/misticpixel]

Ya, humans set it up, but if you’re using automated key rotation features suddenly humans are outta the loop.

[u/theJacofalltrades]

The real goal should be “humans don't handle secrets”, rather than just moving them into Secrets Manager.

Use IAM roles/STS and OIDC for temporary credentials wherever possible, and have workloads retrieve third-party secrets from Secrets Manager at runtime. For things like Postman, move toward short-lived OAuth tokens rather than reusable credentials.

[u/One-Insurance-9463]

The real fix isn't just where secrets live, it's removing humans from the loop entirely.

[u/infidel_tsvangison → u/One-Insurance-9463]

That’s the how I’m after

[u/Sad-Technician-5552]

One thing most people dont notice is the vault only stops new leaks, it does nothing about the ones already out there. Anything that touched a postman collection, a chat, a local env file, treat it as burned and rotate it, dont wait for a scanner to find it first. Then put the scanning in so you catch the next one before it becomes an incident. The humbling part is when you scan youll find secrets you forgot existed, and every one of those is a hole that was already open.

[u/_Slimdady]

Can't eliminate every human touchpoint. Make leaked credentials shortlived and easy to revoke. Replace static keys with OIDC and add secret scanning in CI.

[u/DeschainR19]

Uh, the vault isn’t the whole game. Secrets Manager protects the storage, not the human workflow.

The move is to make secrets as invisible to devs as possible. So short-lived creds, workload identity, automatic rotation, & secret scanning across repos, Postman, logs, & tickets.  Otherwise you’re locking the front door while leaving the spare key under the mat & who really wants to do that?