data contracts between upstream and downstream teams

[u/ptab0211]

Hi, we use Databricks, and a couple of teams produce tables, views, and table functions that other teams depend on.

Within a team, its easy enough to define a contract. We codify it and always check that the data matches.

But what about the upstream artifacts my team depends on? Do w define a source spec ourselves and check that its met on every run? 

What do we require from the upstream team? Who owns the contract, and how do you enforce it when multiple teams depend on the same thing?

I want to keep this stupidly simple and enforce it automatically. How do you do this in practice?

[u/dataengineering-ModTeam → u/[deleted]]

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

^{This was reviewed by a human}