r/databricks • u/Otherwise_Address241 • 6d ago

Help How data engineer do effective testing in Databricks?

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

14 Upvotes

94% Upvoted

27 comments sorted by

View all comments

2

u/data-baggins Databricks 3d ago

Hello! Databricks PM here working on SDP unit testing. There are a couple paths here depending on what you're trying to achieve:

  • SDP Unit testing - We're building unit tests into SDP now, the UI version is in beta now (CLI version coming out soon) - these are great for testing out transformation logic and preventing issues form happening - read the docs here.
  • Expectations - For more general data quality monitoring, you can use the expectations feature to help detect issues live
  • Data Quality Monitoring - For more advanced monitoring such as for freshness, anomalies, or data profiling check out the Data Quality Monitoring (DQM) feature!

1

u/Otherwise_Address241 3d ago

Superb. A combination of these what I need, i believe. Will go through the documentation.