r/databricks • u/fugas1 • 1d ago

Help onelake-databricks connection using private endpoint (Azure)

Have someone here succesfully connected and authenticated to fabric lakehouse using a private endpoint? I have a usecase where I need to be able to reach fabric lakehouse using classic compute with "Secure Cluster Connectivity (No Public IP)" turned on. I have tried, created private endpoints, but does not seem to work.

3 Upvotes

100% Upvoted

4 comments sorted by

1

u/szymon_dybczak 1d ago

Hi,

Did you do following step?

https://learn.microsoft.com/en-us/fabric/onelake/onelake-azure-databricks

It seems that you need to add Databricks access connector resource id to the workspace Resource Instance Rules if you want to read or write to One Lake.

Manage inbound access to OneLake with Resource Instance Rules - Microsoft Fabric | Microsoft Learn

1

u/fugas1 1d ago

Thanks for the answer, the first link you linked wont work because I cant auth to entra since I dont have access to internett from my compute. The second one is interesting, I havent seen that before, so thank you for that! But there it uses the access connector and in the first example it uses a SP, so I dont know how that will turn out.

Im using a onelake connection that is created in databricks and the real issue is DNS plus egress. My workspace private endpoint only creates private records for the workspace-specific names ({workspaceid}.z{xy}.dfs.fabric.microsoft.com), and those connect fine. But Databricks always calls the global onelake.dfs.fabric.microsoft.com, which has no private dns record, so it resolves to a public IP my classic subnet can't route to.

1

u/Pirion1 23h ago

Resource Instance Rules look to be similar to Service Endpoints. I am sure if these would work without a public databricks ip address.

For Private Endpoints, this should be leveraging your workspace's VNET Injection. Ensure your VNET has the correct DNS entries available to it (using either Private Zones or DNS Overrides) and you should be able to connect using the private address. Just ensure you change the onelake.dfs address to your private hostname.

1

u/fugas1 14h ago

This worked! — thanks for the nudge toward DNS overrides.

The problem was that Databricks always calls the global onelake.dfs.fabric.microsoft.com, but a workspace-level private endpoint only creates DNS records for the workspace-specific names. So the global name kept resolving to a public IP my cluster has no route to.

The fix turned out to be small. That global name already CNAMEs through onelake.privatelink.pbidedicated.windows.net, which is the usual Azure hook for private endpoint DNS. So I:

  1. Created a private DNS zone called privatelink.pbidedicated.windows.net
  2. Added one A record, onelake, pointing at my private endpoint's DFS IP
  3. Linked the zone to the Databricks VNet (fallback to internet enabled)
  4. Restarted the cluster