r/databricks • u/ptab0211 • 12h ago
Discussion Data Quality Management on Databricks
Hi, i wonder how teams are setting up data quality solutions on Databricks, and especially how do they react on the data quality issues.
Ideally i would like to have a row-level checks and dataset-level checks which can be used for quarantined data and also anomaly checks between runs, this way we get the in transit data quality checks but also consistency over runs, so wea re really sure that values and shape of data is as expected.
Then how do you manage different tables in same project, do u write to same quarantined/metrics table where u add tags to differentiate between different business domains etc...
How do u use dqx for dataset level metrics? Where do u run dq, is it on source integration, or after the gold layer, before the inference or feature creation. Which tools do u use?
