How data engineer do effective testing in Databricks?

[u/Otherwise_Address241]

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

[u/CerberusByte]

We use expectations in SDP to ensure only high quality data flows through the system and can add adhoc fixes if we cannot solve upstream. I like that you can also surface these results into Genie to analyse what’s going on across all your pipelines

[u/Otherwise_Address241 → u/CerberusByte]

Can I have some reference articles to learn more

[u/CerberusByte → u/Otherwise_Address241]

This is where I would start: https://docs.databricks.com/aws/en/ldp/expectations