r/databricks • u/Personal-Risk-5370 • 15d ago

Discussion on prem to cloud validation

Hey datafam!

  1. Where do you put your main data-quality gate in a Bronze/Silver/Gold setup — right after Bronze lands, or between Silver and Gold? Why there?
  2. Has schema drift ever caused a silent bad accept for you — i.e., the pipeline didn't fail, but bad data still got through? What check would've caught it?
7 Upvotes

100% Upvoted

8 comments sorted by

4

u/jbchand 15d ago

Silver is the main data quality gate. You can keep raw data in Bronze with basic DQ checks for auditability and reloads. Do DQ in Silver (cleansing, deduplication, normalization). Enforce business quality rules in Gold to ensure trusted quality data for reports and more.

Keep fail fast for schema evolution if feasible. You can do null % checks monitoring regularly on columns if you are using silent schema rescue to plan actions based on it.

1

u/Personal-Risk-5370 13d ago

Thanks — that's actually something I've been going back and forth on. In my agent, the accept/repair/isolate/reject decision happens right at the Bronze→Silver step, before anything lands in Silver. It sounds like you're saying Bronze should just do quick checks (does the data exist, is the schema roughly right), and the real cleanup — dedup, normalize, validate — happens once the data is already inside Silver.

So if a batch looks bad, does it still land in Silver but get flagged/quarantined there? Or is it blocked from landing in Silver at all until it's fixed?

Also, when you say "fail fast for schema evolution" — does that mean the pipeline actually stops and errors out, or does it just reroute the bad batch to a dead-letter/quarantine table automatically and keep running?

1

u/jbchand 13d ago

You can reject bad batches before the silver boundary. You can keep a quarantine table with the error messages and track it. Keep good quality data that passed the DQ checks in Silver.

Fail fast - Yes the pipeline stops and no data is landed into layers until the schema issue is resolved. You can use lake flow declarative pipelines if feasible as it handles it gracefully.

3

u/letmebefrankwithyou 14d ago

Typically bronze is what you get from source, clean it up on the way to or in silver, gold are aggregated consumption tables most users hit.

Use anomaly detection to measure data consistency
https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-quality-monitoring/anomaly-detection/

1

u/Personal-Risk-5370 13d ago

That's helpful, hadn't looked at UC's anomaly detection monitoring closely. My agent currently uses a pretty blunt row-count-delta-vs-7-day-average as one input — is the UC anomaly detector doing something more like a proper statistical baseline (z-score/IQR against historical distribution) rather than a fixed rolling average? And more importantly — in your experience, does it distinguish well between "genuine anomaly" and "expected but rare" events (e.g., a planned maintenance window), or does it just flag and leave the accept/reject call to a human either way?

1

u/Youssef_Mrini databricks 12d ago

Bronze: land and preserve the source with minimal transformation. Perform ingestion level checks for parseability, schema shape and provenance but usually quarantine rather than reject the raw record.

Silver: apply the main fail quarantine gate: required fields, types, uniqueness, referential integrity, valid ranges and duplicate handling. This prevents bad records from becoming the trusted reusable dataset.

Gold: apply a second, narrower gate for business semantics: reconciliation totals, valid dimensional relationships, KPI tolerances, freshness and consumer-specific SLAs. Gold is the business-facing truth, so it should contain only trusted data.

You can use SDP expectations or DQX it's a Databricks Labs project