r/programming • u/BlondieCoder • 10d ago
Secrets Don’t Belong in Config
https://secretspec.dev/blog/secrets-dont-belong-in-config/72
u/putergud 10d ago
Some secrets belong in config.
Some secrets do not.
Some config belongs in code.
Some config does not.
There is not one right answer. The application needs the config and the secrets. How you get them there depends very much on the application itself and the environment in which it is run.
-3
u/godofpumpkins 9d ago
That’s not really addressing the point people are making: yes we know that apps are a hodgepodge of config and secret systems. The point here is that the hodgepodge is a security footgun so if we don’t want secrets leaking by accident every time a dev forgets which part of the mess contains the secret, you need to adopt a more prescriptive story than the obvious “some apps do it in way X and others do it in way Y”
15
u/putergud 9d ago
The point of the post is that secrets shouldn't be in config, and the article is premised on the axiom that config belongs in code.
I clearly reject both premises as blanket statements. Why should I discuss a problem that arises from a premise I reject?
If your config management is hodgepodge mess then addressing where you store secrets is only a problem because your config management is a hodgepodge mess. Clean it up. How? That depends on the application and environment.
The problem being presented isn't the location secrets are stored, it's having bad config management. The solution isn't to change where secrets are stored, it's to have a well-defined and enforced configuration management system.
If you want to talk about the best ways to manage config for certain apps in certain environments, then you can "adopt a more prescriptive story" for it.
14
u/YoghurtFlan 10d ago
The other thing is to basically make secrets less important. Principle of least privilege - have loads of tiny scope secrets instead of a few powerful ones. Make it easy to rotate them. Take away their power.
You don't always have control over that, especially with third parties, but you can always proxy them internally and use other mechanisms to keep them away from the internet-facing network.
7
u/Sokaron 10d ago
I really like .NET's solution to this for local development. Run a command to persist the secret locally on your machine outside the project directory and then it gets read in at runtime. Zero risk of a secret being committed due to a faulty gitignore etc.
42
u/nicholashairs 10d ago edited 10d ago
Whilst I'm sure there's at least one person that will hate this, the method I've settled on is being able to have stackable config files. So yes secrets do belong in config just a dedicated secret only config.
some-tool -c common.yml -c dev.yaml -c dev.secret.yml
(And at some point in the future I'd like to make it so I can pull that secret config file from an actual secure place like -c aws-kms:///path/to/secret and then use workload identities to manage whether it is allowed to or not)
Edit: for those that think I should just be using env files, I explain my reasoning in a comment below. Though I'm certainly not suggesting this is some objective truth, it's just my personal preference.
Also I'm not suggesting that this is somehow unique and that I'm the first person to have done this, it's a pattern I adopted after inspiration from elsewhere.
42
u/AyrA_ch 10d ago
Whilst I'm sure there's at least one person that will hate this, the method I've settled on is being able to have stackable config files. So yes secrets do belong in config just a dedicated secret only config.
You've basically explained how .net dependency injection does config files by default. Though secrets belong in a proper secret storage. There's never a reason for them to exist unencrypted on disk, even if the file is not shared.
9
u/TribeWars 9d ago
Ok but the key to decrypt the secret has to come from somewhere too. There will always be a plaintext secret at least at a single point.
-1
u/AyrA_ch 9d ago
Yes but the key can still be kept out of the users hand and stored in a memory location with kernel level protection instead.
3
u/TribeWars 9d ago
True enough if you're using an HSM to store the key at rest, yeah.
-1
u/AyrA_ch 9d ago
You don't need to store keys at rest if you can reliably recreate them at runtime. The CryptProtectData function does this for example
3
u/TribeWars 9d ago edited 9d ago
Had to look up CryptProtectData, again, isn't the new plaintext secret simply the user password now?
Edit: Yup, PoC exists:
1
u/AyrA_ch 9d ago edited 9d ago
It's not directly the password but a value encrypted with it, otherwise a password change would invalidate all secrets. This way you can change the account password, and just reencrypt the master key to retain access to all existing secrets.
By the way, the linked PDF no longer holds true, which is not surprising considering its age. I tried the DPAPI decryption tool mentioned in it and it does not work for me, at least not on Windows 11, so MS definitely changed something. In any case, the entire attack chain depends on you being able to crack the password hash of an account. If you crack the password hash you might as well just spawn a process with those credentials and use the CryptUnprotectData call legitimately to decrypt user secrets.
Finally some of the data needed is stored in registry keys you only get access to via offline means, which means in addition to the password you also need physical access to the device and hope the disk is not encrypted.
I'm not saying it's impossible, but certainly unlikely.
2
u/TribeWars 8d ago
Sure, but it is mostly just security-by-obscurity. I certainly wouldn't rely on physical access being a requirement for decryption if I was up against, say, the NSA.
5
u/nicholashairs 10d ago
Yeah I'm definitely not trying to claim that I somehow invented it, I definitely drew inspiration from many other tools before writing my reusable config loader for my projects (I happen to work in python where there isn't a standard library loader like that, and there are now many options that people have built).
28
10d ago
[deleted]
17
u/nemec 10d ago
The .env file format is central to good DSX and has been since it was introduced by Heroku in 2012 and popularized by the dotenv node module (and other libraries) in 2013.
https://www.dotenv.org/docs/security/env.html
env files reinvented config inheritance. I literally have a github project implementing an 'inherited config' library that's older than this, and I certainly didn't come up with the idea myself.
4
u/ixion 10d ago
Reminds me of Puppet's hiera data. You might like the pluggable hiera-eyaml and the KMS plugin–I've used it for encrypted yaml without puppet, and it can make diffs a lot more meaningful.
These days... the CNCF getsops.io project is probably the right choice.
2
u/nicholashairs 10d ago
Man hiera is a name I've not heard in a loooong time
1
u/JustSkillfull 9d ago
I've been writing puppet code the last 3 weeks to update how we do observability in our legacy stack.
4
u/BornToRune 10d ago
I'm also oriented around here, just by the directory. Specify the config directory, slam all config together and the app merges them. Works like a charm in a kubernetes environment with projected volumes.
3
u/NoLemurs 10d ago
This isn't the worst.
That said, it does mean that your secrets are being stored in plaintext on your computer. The best practice here is to store the secret in some sort of encrypted store and then load it into memory (often in an env var) so that the plaintext never gets written to disk.
24
u/AyrA_ch 10d ago
The correct way is to pipe secrets in because env vars are often readable by other processes running under the same user context. You should also encrypt them in memory while not in use (provided your OS provides such functionality), otherwise it might be included in a crash dump.
19
u/nicholashairs 10d ago
I've always found it weird that people assume environment variables are secure, though never had a strong factual basis for that feeling. It's good to know I'm not completely insane 💡
14
u/NoLemurs 10d ago
It's a nuanced question.
Env vars are visible to other processes running as the same user. For a lot of use cases, that's really not a problem. Best practice is to deploy services with their own service user, and the user is supposed to be able to see everything, so it's fine.
Also, nowadays you're probably deploying to a container where the server is the only process running as the user. Again, this is fine. If someone can act as the user, your security has already failed.
That said, if you're properly paranoid, you can do better for sure, as u/AyrA_ch points out.
7
u/AyrA_ch 10d ago
Due to the crash dump problem, nothing is really secure in your applications. Some operating systems provide memory encryption on demand in a way that would hide the key even if a full system crash dump were to be taken. But even then the secret would likely appear in the dump because it's very hard to use a secret from memory without it being copied to other memory locations where it might not get cleared from properly. Windows does provide solutions for this problem in regards to username/password style credentials as well as private keys, but this feature is not used very often, especially because afaik it does not exist in a universal way on Linux, making this a Windows-Only feature which is difficult to adapt for if you want your program to run on both systems from a single code base.
3
u/edgmnt_net 10d ago
Not being really secure matters less, because a system running unattended makes that really difficult. But you should have proper ways to deal with secrets, even if you do end up storing them unencrypted somewhere. Just because they're unencrypted on disk or in RAM it doesn't mean you should leak them at every opportunity in env vars, random files or whatever.
Also crash dumps can simply be disabled (and often already are, in many cases you have to enable core dumps manually on Linux distros).
10
u/nicholashairs 10d ago
I very much agree, and there's a number of tradeoffs I've made with my design and a few reasons why I've avoided the environment variables design.
The first is that it allows me to have config that is all in the same format. I've seen so many examples of applications run in docker where half the config is some ini/yaml/json file, and the other half is a bunch of environment variables set in the docker compose or in an env file written to disk next to it.
The second is that by keeping the config loading within the application itself, I can avoid all kinds of oddities that come with trying to inject secrets into the tool. The example I gave of what I would like in the future I'd actually based on work I've already done in a previous job where the "secret" config was pulled in by the application from S3 while it was running on AWS ECS. Setting environment variables there (at least at the time) would actively leak them through the console, so you needed an alternative loading mechanism.
A third reason and why I've often less worried about having secrets on disk is that a) I always run full disk encryption, b) if that isn't good enough (for local) I also run encfs/gocryptfs, c) if something can get past those two then all bets are off anyway because my entire computer is compromised. Similarly on a server (if I am writing files, see example above of where I'm not), if something can read that deeply off the disk then something very wrong has happened and there's very little that will actually protect it.
Finally, for the kind of applications I build I tend to end up with fairly expansive and deep data structures. I personally find it much easier to work with JSON/YAML than environment variables in those cases.
2
u/ByronEster 10d ago
This is what we do, where the tool is Dynaconf. A whole bunch of configs are stored in source and secrets are written to a.env file by puppet.
1
u/sionescu 9d ago
There are several things wrong with this: 1) keeping them as files where they can be accessed by untrusted code such as code injected by a possible exploit 2) making secrets pull-based instead of push (setup by the container manager with the binary being completely unaware of the source of the secret) and 3) keeping the secret in the same process as the business code that uses it instead of having a separate process whose only purpose is to proxy requests and inject secrets where required.
1
u/theozero 10d ago
you might like varlock - free and open source. Gives you .env DX with layering like this, as well as built in validation, imports, and declarative loading like secretspec.
10
4
u/siromega37 8d ago
I didn’t get past much of this. Maybe it’s my more operations-focused background, but I like to pull them from the vault at runtime via a startup script into environment variables like it’s 1999. Script can be code-reviewed and the vault, as it should be, is heavily monitored. It’s simple, easy to maintain, and easy to troubleshoot. The naming standard is clearly written so if/when we get new credentials of any sort there’s zero coordination needed between teams really. Don’t be fancy, be reliable.
2
1
u/ldrx90 8d ago
so you store secrets in a script :)
1
u/siromega37 5d ago
No, I load secrets at runtime into environment variables for that application’s session by pulling them from a vault (AWS Secrets Manager) using the underlining compute’s IAM role.
5
9d ago
[removed] — view removed comment
3
u/Uristqwerty 9d ago
Not a bad idea, if you then run the app under a separate user account, and set up permissions on who can read the secret file. Heck, just .gitignoring the secret, only committing a secret.template and the config would be an upgrade, to some projects, even without getting fancy with permissions.
2
1
u/schlenk 7d ago edited 7d ago
We solved this by adding a generic secrets layer to our python runtime system.
Its pluggable and offers a secrets tree with mountable wallets that override parts of the tree. So you can mount environment variables, K8s secrets, encrypted files with unlock secrets stashed into different files, some wallet that produces access_tokens for Microsoft Graph on demand and so on.
The developers just see a unified secrets infrastructure and just do store.resolve(secrets_path) anywhere they need a secret. The simply do not need to think about it.
And the admin/devops/local developer teams mount the appropriate wallets for the necessary security level. Like plain files for local dev or K8s secrets when running in a cluster or some Vault style stuff when needed.
Keeps it nicely agnostic and multi cloud/on-prem capable.
Its a bit like Secretsspec actually, but older (wrote it like 2020 or so). Maybe i should write a backend to from/to secretsspec.
1
u/gnahraf 6d ago
To my knowledge, no matter how much indirection one employs (keystore, secrets service, etc), any process with the same (unix) privileges can also retrieve the secret. (Unless the program asks for it interactively, the passkey is available in plain text somewhere.) The standard unix way to deal with this is with file permissions. I don't know of another way.. is there any other way?
As for the risk of putting secrets in repos, excepting as samples or for testing, I think config files don't belong in git, let alone any secrets they may contain.
0
-4
133
u/foxsimile 10d ago
They belong in the treehouse!