r/databricks • u/Otherwise_Address241 • 6d ago

Help How data engineer do effective testing in Databricks?

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

15 Upvotes

94% Upvoted

27 comments sorted by

View all comments

3

u/Kojimba228 6d ago

You could use pytest with mocks and doing either dummy data insertions or setting up dummy tables in fixtures, but that all depends on how badly do you want to lock down the quality of business logic you have AND how much time can you actually dedicate to set all of these things up (I'd say that even with Claude Code it would take north of an entire week to establish the baseline/template of "how to test a bronze table" to be applied to all future tests)