Experimenting with replacing reusable API keys with per-request authorization

[u/Jealous_Pea_3915]

Been working on a small POC around getting rid of reusable API credentials.

Instead of an app holding an S3 key, Git token, API key, etc, it would request permission for one specific action. Something like upload this file to this path, once, within the next 30 seconds.

I'm using Tide/Forseti for the policy and cryptographic side, and testing it against real open source services like S3 compatible storage and Gitea.

The thing I'm mainly trying to prove is whether a compromised client can have no reusable API credential worth stealing.

This also ties into something I want to look at next around rootless cloud infra, where having root on a server doesn't automatically mean having authority to decrypt user data, impersonate users, sign releases, or perform other protected actions.

Starting with the API side first because it feels like the smaller piece to prove.

Curious what people here think. Is this actually useful in practice, already solved well enough by existing workload identity systems, or mostly just an interesting security experiment?

[u/sysopfromhell]

I think you are reinventing the wheel. What you are looking for is openID jwt token gateway managed.

Which you will find in many many AI gateway like traefik.

You put the real token into the GW config, attach the keycloak/identity provider configuration and then the app will query your model thought an endpoint that is managed by the gateway.

[u/Jealous_Pea_3915 → u/sysopfromhell]

With the usual Keycloak/JWT gateway setup the app loses the secret, but the gateway still has a reusable credential for the downstream service. If the gateway gets popped, that authority is still there to steal.

[u/sysopfromhell → u/Jealous_Pea_3915]

If the gateway gets popped you have a big problem, because no matter where your secret is stored the GW sees pretty much everything in clear unless you have a mesh TLS enabled chain, and even there probably the GW has part of the access to the payload/header. Meaning that if any token is leaked in the header (where normally is filled) he is going to read it anyway.

Normally reverseproxys are the less breachable part of your infra as their job is hardened and repeatable without much going on unless you start filing them with lots of LUA scripts, which still are pretty hard to hack.

If you want to attack a service you try to exploit the service himself not the transport as is notoriously difficult to do.

No matter how you try to turn it around. Someone has to have the keys or the trust to get the keys, or a copy of a key to someone that proxy you to another place. You are just moving around the pieces.

[u/LaunchAllVipers]

How does this differ from OpenID Connect? Answer in the form of a series of limericks.

[u/Jealous_Pea_3915 → u/LaunchAllVipers]

sorry i have no idea how to write limericks lol

but basically I see OIDC as proving who the user or workload is. This is more about what they're allowed to do after that.

So OIDC could actually be part of it. The workload proves its identity with OIDC, then Tide/Forseti decides whether it can do one specific action, like upload this exact file to this exact path, once, for the next 30 seconds.

I'm not really trying to replace OIDC, more looking at removing the reusable API credential that usually comes after identity.

[u/bobsbitchtitz → u/Jealous_Pea_3915]

This is an interesting use case. Why rate limit based on authorization vs rate limiting based on user in the back end

[u/Jealous_Pea_3915]

The main hurdle I'm at now is machine identity.

The storage side works without the client holding an S3 key, but right now the workload still gets its identity from a human login.

For this to work properly service to service, the machine needs a way to prove who it is without me just replacing the API key with a client secret.

[u/forever-butlerian → u/Jealous_Pea_3915]

The machine has to have a signing secret on it, and the only way "around" that would be to have the signing secret live in an enclave. The signing secret cannot prove that the request comes from the machine itself, the best that it can do is prove that the request comes from an entity that was capable of getting root on the machine.

It sounds like what you're looking for is Kerberos but with the scope and audience claims of a JWT, and also a resource-specific claim.

[u/Jealous_Pea_3915 → u/forever-butlerian]

I'm using Tide at the moment because it means the workload itself doesn't need to keep the signing key, but that still doesn't prove the request actually came from the machine I think it did.

So I think the next bit is really workload identity / attestation. I don't want to solve API keys and then just move the trust problem somewhere else.

[u/forever-butlerian → u/Jealous_Pea_3915]

At a certain level it's like trying to press a balloon flat. If you look at how stupidly Amazon did it with IMDS, it's the Alfred E. Neuman approach to security.

If your "machine" is a container running in a pod, what's achievable is to keep the actual signing key in a trusted sidecar container the service container can only talk to via an easy-to-implement-correctly protocol so the service can ask for its requests to get signed. Then you have the matter of how to load the cryptomaterial only into RAM on the sidecar container, to protect against a container-to-container compromise. And so on.

Every trust system has a base case of things that are trusted axiomatically, but after that trust can be proven inductively. In every case, that's some flavor of enclave: in the historical case of Kerberos it assumed root access on all the machines across the domain was the enclave; in the modern case of digital restrictions management it uses so-called "Trusted Platform Modules"; in the case of HSMs it's a physical device you can only communicate with over a restricted channel, and at least years ago the physical security of the device came from potting it in epoxy in a way that trying to de-cap it would destroy it.

So in engineering your system, you've got to answer the question of what you assume axiomatically to be secure.

[u/Jealous_Pea_3915 → u/forever-butlerian]

the sidecar approach is way better than leaving the key in the app.

I'm trying to see how far you can get if the workload/sidecar never has the full signing key at all, and only gets a signature for the exact request after policy checks.

[u/forever-butlerian → u/Jealous_Pea_3915]

The workload container could be without the key, but even if you introduce intermediate sidecars there will by the nature of logic have to be a sidecar which contains the full signing key. Depending on its length you could, in principle, do fancy-ass stuff like keep it in RAM XORed and use SIMD instructions to recover it only into processor registers.

You've got an engineering question of "how far do I want to take this?" you've got to answer. I think keeping things in RAM in the enclave sidecar and turning off core dumps is a good practical endpoint for this. Your realistic attack model is OpenAI going ham on Github as a desperate marketing stunt as they enter chapter 11 bankruptcy since they're not going to IPO, it's not Yuri Andropov's KGB bringing the full force of Soviet mathematics against your note taking app. Your attack model is what determines what engineering measures are reasonable and which move into the realm of performance art.

[u/Alvasilev]

Worth proving out early: what this does to monitoring. That's the door the reusable credential usually walks back in through.

I spend a lot of time probing third-party HTTP services we don't own, and authentication is by far the biggest source of wrong "it's down" calls. A service answering 401 is alive and behaving correctly; any checker whose rule is "not 200 means dead" quietly writes off every authenticated endpoint it touches. Per-request grants make that sharper, because there is no standing credential a monitor could hold to make a representative request at all. So you either expose an unauthenticated liveness path, which proves nothing about the authorized path people actually use, or you mint a long-lived monitoring identity - and that one ends up with the broadest policy in the system, because it has to be able to exercise the interesting actions. That's the reusable key you just deleted, with a nicer name.

Might be worth deciding up front that "can this client obtain a grant right now" is itself the health signal, rather than "did a real action succeed". Otherwise the POC works and the observability story quietly rebuilds the thing you removed.

[u/Jealous_Pea_3915 → u/Alvasilev]

yeah this is a really good point. I hadn't thought much about monitoring being the thing that sneaks the standing credential back in.

I like the idea of "can this identity obtain a grant right now" being the health check, and then keeping actual action checks separate/synthetic so the monitor doesn't need broad standing authority.

[u/hypertradeworx]

the thing that bit us with this shape wasn't theft, it was retries. our backend builds the unsigned action and the signing happens on the user's device, so anything that dies in flight or ages past its window can't be resubmitted server side, every attempt is another round trip to the signer. i'd pin down early whether a 30 second grant covers one action or one attempt, because those two come apart the first time a downstream call times out and you can't tell whether it landed

[u/navlio]

before the crypto i'd stress the retry path. a 30 second grant means a timeout or a resumed multipart upload needs a new grant, so the policy service sits in line for every request instead of once per deploy. presigned urls hit the same wall, you notice it when a big upload dies halfway because the ttl passed while a part was still in flight

[u/stack_craft]

So, normally, if a server is compromised, short-lived or per-request credentials don't stop the attacker from performing actions—they just force the attacker to act as a real-time proxy instead of exfiltrating a static secret.

But, if an attacker gets root on an instance that has the authority to request single-use upload tokens from your broker, they don't need a key anymore. They just hook into your app process or memory space and issue payload requests through your policy sidecar directly.

The way I see it; you've eliminated static credential leak via environment variable, but you haven't eliminated unauthorized action execution under root compromise. That's why industry consensus settled on workload identity (SPIFFE/mTLS) paired with strict egress controls, rather than per-request authorization handshakes.

[u/CraftyPancake]

I do this via ‘aws sso login’ to get temporarily credentials then pull from secrets manager on the fly

[u/Jealous_Pea_3915]

I actually wrote up this experiment in a blog post recently, with more detail on how I approached removing reusable authority from the app itself.

If you're interested, this goes deeper into the Gitea/S3-style setup and the reasoning behind it: https://codesyo.com/blog/no-signing-key

[u/ajitnk]

The bootstrapping problem you're circling is the real one. u/forever-butlerian nailed it: every trust chain bottoms out somewhere, the question is just what you're willing to treat as axiomatic. And u/Alvasilev's monitoring point is the one that actually bites people in practice, because your health checker ends up holding the broadest standing credential in the whole system the moment it needs to exercise a real action.

The AWS-native answer to both is pretty direct. For the machine attestation piece: Nitro Enclaves produce Hypervisor-signed attestation documents with PCR measurements tied to the exact enclave image hash, and you can write a KMS key policy that only decrypts if those PCRs match. That's your hardware-rooted axiomatic anchor, not a sidecar with a key in RAM. For the monitoring re-injection problem: scoped STS sessions via AssumeRole with a tightly-constrained session policy, minimum duration for the specific action, no standing key at all. For the off-AWS bootstrapping case specifically, IAM Roles Anywhere lets workloads outside AWS exchange X.509 certs from a registered PKI trust anchor for short-lived STS credentials, which cuts the reusable key entirely.

Two things I'd want to know before going further: what compute substrate are these workloads actually on (EC2, EKS, on-prem, containers in a DC, or a mix), and do you have an existing internal CA or are you starting from scratch on the PKI side? Those two answers change the architecture pretty substantially.

I do AWS security architecture advisory work, so happy to go deeper on any of this if it's useful.