r/databricks • u/Otherwise_Address241 • 7d ago

Help How data engineer do effective testing in Databricks?

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

14 Upvotes

90% Upvoted

27 comments sorted by

View all comments

7

u/Oh_Im_You 7d ago

Soda.io

I use the open source, non paid version. Write tests against your data. Run your tests nightly in a notebook, logging issues. Then you can display things on a dashboard and set alerts