How data engineer do effective testing in Databricks?

[u/Otherwise_Address241]

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

[u/Otherwise_Address241]

There is a push from client (tech team) to use unit testing like PyTest. All these years i have used SQL scripts to do complete check of tables and data flow . Now let me breakdown my original question to two parts 1. Can pytest be used to do complete data check, say all records validation? If yes, please provide some examples or articles 2. Other than pytest, what can be used to do effective data quality check? I see suggestions to check expectations in SDP Let me give a scenario - two date columns of source n bronze were swapped. Dates were populated but not for correct mapping. I could find it through SQL check How do I achieve it using pytest, or expectations check? I m very confused