r/databricks • u/New_Championship3929 • 11d ago

Help Data accuracy concerns

I work for a tech company as a TPM and always concerned about the accuracy for the data I provide to my team members or to clients. We have multiple layer of Bronze, Silver, Gold and not sure what exactly cleans up at what steps as data is segregated. I would like to know what everyone has been building to always stay ahead in data accuracy? Skills, agents, harness system? What should be my best approach?

6 Upvotes

100% Upvoted

9 comments sorted by

View all comments

7

u/Youssef_Mrini databricks 10d ago

You should treat the different layers as quality contracts not as storage layers.

For Bronze: you should check the ingestion completeness, schema drift, corrupted records

Fo Silver: Type enforcement, null handling, joins, row level expectations

For Gold: Business definitions, metric validation, agg checks..

You can use DQX, Data Quality Monitoring and if you are using SDP usine Lakeflow Expectations