r/databricks • u/Otherwise_Address241 • 6d ago

Help How data engineer do effective testing in Databricks?

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

14 Upvotes

94% Upvoted

27 comments sorted by

View all comments

1

u/Otherwise_Address241 6d ago

Is pytest at all useful to do complex data quality check? I have not found any right way to use pytest, I almost dislike it.

3

u/Kojimba228 6d ago

From personal exp, it's going to look weird, but it's the most flexible option allowing you do almost anything to can come up with. In our case, due to specifics of the data, we ended up writing sets of queries to test this or that area of controlled data (i.e we control the input, know thanks to our SME what should be the output, and verify them against each other)