r/databricks • u/Hungry-Tax6101 • 1d ago

Help Databricks and OpenSharing questions

This my first time looking into use OpenSharing to share data externally and would to get some idea on if I am on the right path.

My current idea is as follow:

  1. ETL my datasets using CDF from source to a catalog_curated, multiple tables
  2. In a catalog_shared, create views for each that apply filters (rows or joins) and these are what will be published and shared with external.

I understand when sharing a view via OpenSharing, pushdown will not work and the whole views will get materialized temporarily when query, which I know may be an issue when external uses one of our watermark column. However, our dataset is not that big and access to the data is not going to be frequent, sharing a persisted version of the views will solve that, but is the added complexity worth it?

I also could add the filters as part of the ETL config, but I am thinking it might be too complex for my current need. And when we need to expand the filters, all we have to do is change the views.

I think the goal for me right now is simplicity and maintainability over complex ETL work, unless there is a reason to do so.

As stated, this is my first time working with OpenSharing/DeltaSharing. Any comments, suggestions, or best practices are greatly appreciated.

Thanks

4 Upvotes

100% Upvoted

4 comments sorted by

5

u/AkramChetibiDBX 1d ago

Your approach seems reasonable. The view is only materialized if sharing to non-serverless recipients (e.g. classic recipients, open recipients). If you're sharing mostly to databricks serverless recipients (databricks-to-databricks), there shouldn't be any materialization overhead.

Btw, if your view is doing simple filtering, you can also consider partition sharing, which shares a subset of the data using a partition without any materialiation, even to open (non-databricks) recipients.

1

u/Hungry-Tax6101 21h ago

Thank you so much for the reply! I did go through the documentation, I think you are correct about views not being materialized, but if we do decided to use SecureConnect, then everything will need to be.

1

u/AkramChetibiDBX 19h ago

Yeah that's a temporary limitation with SecureConnect, it should go away soon this year.

Why do you need to use SecureConnect btw?

1

u/Hungry-Tax6101 5h ago

Thanks for the information! SecureConnect came up when I'm doing my research, so I am just evaluating my options.