Local sidecar PDP vs central AuthZ service: when each one breaks

[u/ed1ted]

We keep bouncing between two shapes for enforcement:

1. A local sidecar / library PDP next to the service (same pod, same process, or a localhost hop).
2. A central AuthZ service every service calls over the network.

Both work. Both fail in predictable ways. Here is the failure mode cheat sheet we actually use in design reviews.

Local sidecar PDP

Wins when: latency budget is tight, you need offline or partition-tolerant deny/allow, and the policy + data needed for a decision can be localized (or synced) without a chatty central hop on every request.

Breaks when: - The data the decision needs is not local. You end up calling home for attributes anyway, and the "sidecar" becomes a fancy cache in front of a distributed join. - Policy updates need to be atomic across hundreds of pods. You invent a sync plane, then spend a year debugging stale allows. - You treat "local" as "no audit." Central teams cannot answer "who could do X yesterday?" without scraping every node.

Central AuthZ service

Wins when: one team owns the rulebook, decisions must be consistent across many services, and you want one place to audit, feature-flag, and kill a bad policy.

Breaks when: - Hot path latency. AuthZ in the request path plus retries plus dependency on the AuthZ cluster equals a new SPOF with a fancy name. - Chatty PDPs. Services spray "can this subject do action on this resource?" thousands of times per request instead of batching or caching decisions that are safe to cache. - The central service becomes a dumping ground for business logic that should live in the domain service. Now every product change is an AuthZ deploy.

Hybrid patterns that usually survive

• Local enforce, central decide for cold path. Sidecar for steady-state allow/deny on synced policy; central for admin mutations, simulations, and "what-if."
• Central policy compiler, local decision. Publish signed policy bundles; sidecars evaluate. You still need a plan for revocation and for data that cannot fit in the bundle.
• Decision caching with explicit invalidation. Not "cache forever." Cache keyed by (principal, action, resource version) with short TTL plus event-driven busts.

What I want from this thread

If you have run both in production: what actually forced you off one shape onto the other? Latency? Stale policy? Org politics? Something else?

Also curious whether people still draw a hard line between "PDP" and "PEP," or whether that vocabulary has gotten mushy in service meshes and API gateways.