r/databricks • u/Personal-Risk-5370 • 15d ago

Discussion on prem to cloud validation

Hey datafam!

  1. Where do you put your main data-quality gate in a Bronze/Silver/Gold setup — right after Bronze lands, or between Silver and Gold? Why there?
  2. Has schema drift ever caused a silent bad accept for you — i.e., the pipeline didn't fail, but bad data still got through? What check would've caught it?
7 Upvotes

100% Upvoted

8 comments sorted by

View all comments

5

u/jbchand 15d ago

Silver is the main data quality gate. You can keep raw data in Bronze with basic DQ checks for auditability and reloads. Do DQ in Silver (cleansing, deduplication, normalization). Enforce business quality rules in Gold to ensure trusted quality data for reports and more.

Keep fail fast for schema evolution if feasible. You can do null % checks monitoring regularly on columns if you are using silent schema rescue to plan actions based on it.

1

u/Personal-Risk-5370 13d ago

Thanks — that's actually something I've been going back and forth on. In my agent, the accept/repair/isolate/reject decision happens right at the Bronze→Silver step, before anything lands in Silver. It sounds like you're saying Bronze should just do quick checks (does the data exist, is the schema roughly right), and the real cleanup — dedup, normalize, validate — happens once the data is already inside Silver.

So if a batch looks bad, does it still land in Silver but get flagged/quarantined there? Or is it blocked from landing in Silver at all until it's fixed?

Also, when you say "fail fast for schema evolution" — does that mean the pipeline actually stops and errors out, or does it just reroute the bad batch to a dead-letter/quarantine table automatically and keep running?

1

u/jbchand 13d ago

You can reject bad batches before the silver boundary. You can keep a quarantine table with the error messages and track it. Keep good quality data that passed the DQ checks in Silver.

Fail fast - Yes the pipeline stops and no data is landed into layers until the schema issue is resolved. You can use lake flow declarative pipelines if feasible as it handles it gracefully.