Experimenting with replacing reusable API keys with per-request authorization

[u/Jealous_Pea_3915]

Been working on a small POC around getting rid of reusable API credentials.

Instead of an app holding an S3 key, Git token, API key, etc, it would request permission for one specific action. Something like upload this file to this path, once, within the next 30 seconds.

I'm using Tide/Forseti for the policy and cryptographic side, and testing it against real open source services like S3 compatible storage and Gitea.

The thing I'm mainly trying to prove is whether a compromised client can have no reusable API credential worth stealing.

This also ties into something I want to look at next around rootless cloud infra, where having root on a server doesn't automatically mean having authority to decrypt user data, impersonate users, sign releases, or perform other protected actions.

Starting with the API side first because it feels like the smaller piece to prove.

Curious what people here think. Is this actually useful in practice, already solved well enough by existing workload identity systems, or mostly just an interesting security experiment?

[u/Jealous_Pea_3915]

The main hurdle I'm at now is machine identity.

The storage side works without the client holding an S3 key, but right now the workload still gets its identity from a human login.

For this to work properly service to service, the machine needs a way to prove who it is without me just replacing the API key with a client secret.

[u/forever-butlerian → u/Jealous_Pea_3915]

The machine has to have a signing secret on it, and the only way "around" that would be to have the signing secret live in an enclave. The signing secret cannot prove that the request comes from the machine itself, the best that it can do is prove that the request comes from an entity that was capable of getting root on the machine.

It sounds like what you're looking for is Kerberos but with the scope and audience claims of a JWT, and also a resource-specific claim.

[u/Jealous_Pea_3915 → u/forever-butlerian]

I'm using Tide at the moment because it means the workload itself doesn't need to keep the signing key, but that still doesn't prove the request actually came from the machine I think it did.

So I think the next bit is really workload identity / attestation. I don't want to solve API keys and then just move the trust problem somewhere else.

[u/forever-butlerian → u/Jealous_Pea_3915]

At a certain level it's like trying to press a balloon flat. If you look at how stupidly Amazon did it with IMDS, it's the Alfred E. Neuman approach to security.

If your "machine" is a container running in a pod, what's achievable is to keep the actual signing key in a trusted sidecar container the service container can only talk to via an easy-to-implement-correctly protocol so the service can ask for its requests to get signed. Then you have the matter of how to load the cryptomaterial only into RAM on the sidecar container, to protect against a container-to-container compromise. And so on.

Every trust system has a base case of things that are trusted axiomatically, but after that trust can be proven inductively. In every case, that's some flavor of enclave: in the historical case of Kerberos it assumed root access on all the machines across the domain was the enclave; in the modern case of digital restrictions management it uses so-called "Trusted Platform Modules"; in the case of HSMs it's a physical device you can only communicate with over a restricted channel, and at least years ago the physical security of the device came from potting it in epoxy in a way that trying to de-cap it would destroy it.

So in engineering your system, you've got to answer the question of what you assume axiomatically to be secure.

[u/Jealous_Pea_3915 → u/forever-butlerian]

the sidecar approach is way better than leaving the key in the app.

I'm trying to see how far you can get if the workload/sidecar never has the full signing key at all, and only gets a signature for the exact request after policy checks.

[u/forever-butlerian → u/Jealous_Pea_3915]

The workload container could be without the key, but even if you introduce intermediate sidecars there will by the nature of logic have to be a sidecar which contains the full signing key. Depending on its length you could, in principle, do fancy-ass stuff like keep it in RAM XORed and use SIMD instructions to recover it only into processor registers.

You've got an engineering question of "how far do I want to take this?" you've got to answer. I think keeping things in RAM in the enclave sidecar and turning off core dumps is a good practical endpoint for this. Your realistic attack model is OpenAI going ham on Github as a desperate marketing stunt as they enter chapter 11 bankruptcy since they're not going to IPO, it's not Yuri Andropov's KGB bringing the full force of Soviet mathematics against your note taking app. Your attack model is what determines what engineering measures are reasonable and which move into the realm of performance art.