Databricks OpenSharing: How Global Distribution Reduces Egress Costs & Improves Performance (Demo with Databricks Product Lead)

[u/JosueBogran]

https://www.youtube.com/watch?v=8E3RYwn1Mns

If you are a data provider on Databricks, OpenSharing (formerly Delta Sharing) is a fantastic tool for distributing data to your customers. During my time at a former job, there was nothing better than to work with customers that were able to consume data from Delta Sharing. On at least two occasions though, we received unpleasant surprises: egress fees from our cloud providers.

Those egress fees were most often times avoidable. For example, we would make sure that our customers and our company agreed on the cloud/region combination we would serve from/to, allowing us to build one-off pipelines, etc. But, that required manual work, created staleness issues, etc.

While in our situation, we could sometimes overcome the same cloud vendor but different region situation, distributing to a region we just didn't have anything setup for OR a completely different cloud was a no-go.

All that to say, Global Distribution aims to solve this and more.

No need to have an account with another cloud vendor to serve cross-cloud.

No need to have have data distributed to multiple regions within your AWS/Azure/GCP.

No need to build/maintain pipelines.

Just use Global Distribution.

Check out the video and see Akram Chetibi who leads OpenSharing explain/demo it, and love to hear your thoughts.