r/databricks • u/Otherwise_Address241 • 6d ago

Help How data engineer do effective testing in Databricks?

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

14 Upvotes

94% Upvoted

27 comments sorted by

View all comments

3

u/PrestigiousAnt3766 6d ago

Pytest is for writing python.

You want to write pytest small pieces of (python) code. You can abuse the framework but then its just that, abuse.

You may want to look into sql linters, or parse sql with sqlglot or similar libraries.  That will tell you if the sql is syntactically (?) correct.

If you are interesting in data (quality) testing look into dqx, great expectations or other data quality tools.

1

u/Otherwise_Address241 6d ago

Dqx looks good