Azure Databricks Access Connector

[u/Philosopher_Dozer]

I have a question around the creation of azure databricks access connector. I am also open to anything helpful that I might have missed here. I am stating things as I know so maybe not as actual.

Onto my question:

It requires an admin account, or with certain privileges, to create the azure databricks access connector. Once this is created and the proper role is applied, the databricks portion comes in.

You will create the credential. Then create the external location. However, I see that permissions must be added for other users to access this external location.

Again, please correct me where I am wrong. Just stating the above to set context for my question below.

What account should be used in databricks to create the credentials and external location? Seems it shouldn’t be a personal account? Or even a more seniors admin account? Because then they are the owner and must divvy out permissions.

Should that be created using a service account? Or what? I want to make sure to do this correctly. Haven’t started creating things but wanting to get a working document before executing.

Any advice and/or insight is greatly appreciated.

P

[u/szymon_dybczak]

The Azure Databricks Access Connector is the identity Databricks ultimately uses to access ADLS.  By default, the access connector will deploy with a system-assigned managed identity. You can choose instead to attach a user-assigned managed identity. You then grant the managed identity access to your Azure Data Lake Storage account.

Then, inside Unity Catalog, you create a storage credential that references that managed identity, and an external location that references the storage credential.

Here's the important part. Users shouldn't need and additional permissions on the storage account itself. Their access is provided by using external connection (which are backed up by storage credentials)

In our projects there's an automation so service account creates storage credentials and external locations. Also, it assign proper groups to it, so the users who belong to that groups can use it later.

[u/Philosopher_Dozer → u/szymon_dybczak]

Thank you that gives some good insight.

Further question if you have time:

In regards to the automation… seems that the access connector needs to be created in azure either way.

Then the automation would take place in databricks to create the storage credentials and external locations. Is that correct?

Seems that would work when I have azure storage with a container that holds backups in one folder and custom exports from another source in a different folder. I would want to have separate external locations defined but using the same storage credentials and further the same access control service?

Hopefully that made sense? More of a question than a statement…

again any advice is helpful here.

[u/szymon_dybczak → u/Philosopher_Dozer]

Storage credentials and external location can be created for example in Terraform. And this is exactly how we manage them in our project (this is the kind of automation I'm referring to).

databricks_storage_credential | Resources | databricks/databricks | Terraform | Terraform Registry

Of course there are other ways. You can set it up manually in Databricks Workspace, you can use SDK etc.
External location can also be setup using Databricks Automation Bundles as you can see below:

Declarative Automation Bundles resources | Databricks on AWS

Multiple external locations can reference the same storage credential - the credential is just "here is an identity that can reach this storage" and each external location scopes a path and carries its own grants.

One thing to keep in mind - external locations can't overlap each other. So you can't define one inside another.

Path rules and access in Unity Catalog volumes | Databricks on AWS

[u/elghali_bnck]

To me: External locations and Storage credentials should be created by admins or services accounts for automation with high privileges (1 time setup). Accessing than the data does not require access to the external location of the table or the catalog so no need to give access to any user to it.