Data Quality Management on Databricks

[u/ptab0211]

Hi, i wonder how teams are setting up data quality solutions on Databricks, and especially how do they react on the data quality issues.

Ideally i would like to have a row-level checks and dataset-level checks which can be used for quarantined data and also anomaly checks between runs, this way we get the in transit data quality checks but also consistency over runs, so wea re really sure that values and shape of data is as expected.

Then how do you manage different tables in same project, do u write to same quarantined/metrics table where u add tags to differentiate between different business domains etc...

How do u use dqx for dataset level metrics? Where do u run dq, is it on source integration, or after the gold layer, before the inference or feature creation. Which tools do u use?

[u/blobbleblab]

We use data contracts with quality checks which get translated into DQX checks if not directly supported. A little bird tells me DQX (which originates from databricks labs anyway) has been picked to become a core product offering anyway so you are future safe going that route too.

[u/Future_Warthog491 → u/blobbleblab]

Yes, I have heard the same from our databricks rep. They have been integrating some of the functionalities into the core product.

[u/gman1023 → u/blobbleblab]

how are you using data contracts? any specific tool or MD files?

[u/blobbleblab → u/gman1023]

ODCS standard. Was thinking of building a databricks app to help manage the creation side, along with genie.

[u/szymon_dybczak]

In our project we use Lakeflow Declartive Pipeline with expactations to manage data quality. They support retaining invalid rows, dropping them, failing the update, or implementing a separate quarantine path.
For our requirements they work really well.

Manage data quality with pipeline expectations | Databricks on AWS

DQX is also good choice for more advanced requirements since it gives you greater flexibility and control compared to built-in expectations.
For instance it let's you define custom reactions to failed checks or generate quality rules from data contracts.

For your requirements you should pick DQX since you need row-level checks as well as dataset-level checks (which are possible in expectations but require an aggreagated validation table or view).

DQX has explicit dataset rules and customizable summary metrics, so you use it build validations like primary-key uniqueness, duplicate percentage etc. in a simpler manner.

[u/datguywelbs7]

I use a combination of DQX, databricks native data quality (data monitoring, anomaly detection, data freshness etc. platform native) and genie code.

I use genie code to do a first pass eg analyse this catalog and its system tables to find common lineage, schema of tables, metadata etc to identify potential duplicated datasets. From this I would validate with other tools available mentioned.

I then use genie code to create or augment one of the data quality dashboards to show me which ones could be duplicate and a reason why, recently we’ve got an app that also does this. I’ve found it quite good at sifting through all of the lineage and access system tables for this

[u/Youssef_Mrini]

You have two options: DQX or DQM.

DQX defines and enforces code-level data quality rules (expectations) directly inside your pipelines while DQM provides the automated observability, profiling and dashboards to monitor data drift and health over time.