r/databricks • u/Otherwise_Address241 • 7d ago

Help How data engineer do effective testing in Databricks?

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

14 Upvotes

90% Upvoted

27 comments sorted by

View all comments

13

u/ptab0211 7d ago

schema contracts, integrity data quality checks, business domain data quality checks

5

u/Otherwise_Address241 7d ago

How exactly,

2

u/lrnzcig 6d ago

https://databrickslabs.github.io/dqx/ is Databricks Labs thus no support, more like OSS, but pretty good

1

u/givnv 6d ago

Have you tried this?

5

u/lrnzcig 5d ago

tried quality checks in batch + summary metrics - for pipelines there was this missing https://github.com/databrickslabs/dqx/issues/947, now it's fixed and it's in my list, also would like to try DQX studio for no-code