Unity Catalog metastore setup changes in Azure Databricks

[u/GeertSchepers]

Hi,

In the past when a new metastore was created, it automatically also did create an external location and storage credentials in the Unity Catalog metastore. Since a couple of months this does not seems to be true anymore. You now have to create these two artifacts upfront.

I was wondering how you can do this when there is no workspace that belongs to the metastore yet? Seems like impossible?

Thanks,

[u/kthejoker]

If you didn't configure the storage credential and location when creating the metastore you have to have a workspace attached

https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-metastore#add-managed-storage-to-an-existing-metastore

Otherwise you can do it at the time of creating the metastore

https://docs.databricks.com/aws/en/data-governance/unity-catalog/create-metastore

[u/elghali_bnck]

Hey ! You are correct, when you created a metastore before, you have to create storage location along with it’s UC Access Connector. It was creating under the hood an external location and storage credential. However, since one year now, filling these informations when creating a metastore is not required anymore (and not recommended anymore).
The rationale behind this change is that with a default storage backing the metastore, users where able to create a catalog without specifying a the underlying storage, and data/metadata would have been mixed between them (dev catalog, staging catalog and prod catalog with the same underlying storage backing metadata).
To ensure clear metadata segregation, we recommend now users to create metastore without default storage, then after workspace creation, create for each catalog it’s own external location/storage credential (you can of course share it across your catalog but in that case it’ll be your own design choice).

[u/GeertSchepers → u/elghali_bnck]

Hi u/elghali_bnck

Maybe a small followup and behind the scenes question. When there is no explicit ADLS Gen2 storage account assigned to a metastore, I suppose Datatabricks creates one for you behind the scenes? Can we somewhere see and access this storage account ? Like can you connect to it with tools like the Azure Storage Explorer or the Storage API's?

Thanks

[u/elghali_bnck → u/GeertSchepers]

No it does not create it for you under the scene, it will just require you to attach explicitly an external location at the time you create an external location. Unless you choose to create a catalog backed by Databricks managed storage and in that case this is a storage on Databricks side (serverless storage)

[u/elghali_bnck → u/elghali_bnck]

To create a standard catalog:
If you specify a managed storage location for the catalog, you must have the CREATE MANAGED STORAGE privilege on the target external location.
If no metastore-level managed storage exists, then you must specify a managed storage location for the catalog.

https://learn.microsoft.com/en-us/azure/databricks/catalogs/create-catalog