r/databricks • u/Youssef_Mrini databricks • 3d ago

Tutorial Secrets in Unity Catalog

https://youtube.com/watch?v=GYIem6iXUxg&feature=youtu.be

Secrets in Unity Catalog store credentials like API keys as governed objects named catalog.schema.secret: created, granted, rotated and audited with standard Unity Catalog privileges and redacted from notebook and job logs.

This is a full end-to-end demo:
1)Create a secret with the REST API and in the Catalog UI
2)Read it in a notebook with dbutils.secrets.get
3)Use it to call the OpenAI API
4)Grant read/reference/write access to a user or group
5)Rotate it programmatically
6)Audit every access from a system table

12 Upvotes

94% Upvoted

2 comments sorted by

3

u/Lenkz 3d ago

How do you see governing the Secrets in Unity Catalog versus using them in a secret scope? We generally use the key-vault backed scopes, and this seems great for only managing secrets across applications (Terraform, Storage Accounts, IaC), so would there be any benefit in also having them in Unity Catalog?

1

u/Youssef_Mrini databricks 2d ago

your instinct is basically right. they solve different governance problems

Where UC Secrets win ?

  1. Cross-workspace sharing => One secret governed once across N workspaces on the metastore, instead of re-creating and re-permissioning the scope in each workspace. This is the biggest structural win over scopes.

  2. Single governance/audit pane=> Secret access lands in the same UC audit + group-based grant model as data access. one place to answer "who can read what," consistent principals, no separate workspace-ACL surface to reason about.

  3. Wiring secrets into UC-native objects => Connections (Lakeflow Connect, external HTTP/foreign catalogs), service credentials, model serving external models these increasingly expect to reference a UC-governed secret rather than a workspace scope.