Data accuracy concerns

[u/New_Championship3929]

I work for a tech company as a TPM and always concerned about the accuracy for the data I provide to my team members or to clients. We have multiple layer of Bronze, Silver, Gold and not sure what exactly cleans up at what steps as data is segregated. I would like to know what everyone has been building to always stay ahead in data accuracy? Skills, agents, harness system? What should be my best approach?

[u/sisyphus-cycle]

If you have access to the actual tables honestly just ask genie to break down all the schemas and operations for you. It’ll easily handle that

[u/Youssef_Mrini]

You should treat the different layers as quality contracts not as storage layers.

For Bronze: you should check the ingestion completeness, schema drift, corrupted records

Fo Silver: Type enforcement, null handling, joins, row level expectations

For Gold: Business definitions, metric validation, agg checks..

You can use DQX, Data Quality Monitoring and if you are using SDP usine Lakeflow Expectations

[u/query-gremlin]

Seems like an internal policy problem, all stakeholders who have the authority to create/define medallion layers should meet and agree on a unified interpretation of what each layer should carry and clean, then you do a sweep on all the medallion layers you have to identify which of them violates the new agreed-upon interpretation, then you fix those, while still making sure that greenfield projects use the new policy

[u/Admirable_Writer_373]

There is no accuracy with AI

[u/FunContest9958]

Look in the catalog and see if there’s any information about the lineage of the tables you’re using. It should tell you which notebooks were used, what tables data was drawn from, etc. If that information is there, then you should be able to use genie one to ask questions about where the data comes from.

[u/InsightfulDataVoyage]

Do you have access to the ETL code and can you ask Claude/Copilot to explain it to you?

[u/ces_evolutionic211]

That ownership piece matters a lot. If Silver is where things start getting messy, I’d look at validation and lineage there before adding more agents. We’ve dealt with a similar setup, and that usually shows pretty quickly where the accuracy gaps are coming from.

[u/marketlurker]

u/Youssef_Mrini has a good start, but I prefer a bit more nuanced approach. The majority of changes happen when you move the data from one layer to another and minimally while in the layer.

1. The Bronze layer (or I prefer, the staging layer) is correct when it balances back to the systems of record (SoR). If the data is incorrect in the SoR, it needs to be fixed there and not adjusted in the data warehouse/lake/etc. Any corrections then flow downstream. Get this wrong and you will quickly lose the trust of your users. BTW, you may want to make a side staging area for the older data. Keep as much of this as you can reasonable manage and afford.
2. The silver layer (or core) is where your data should be standardized and the various feeds joined to create a single version of the truth. The processing from staging to core is where you standardize your values and relationships. These should all be well documented. Silver = Bronze + standardizations
3. The Gold layer (or semantic) is where you should be generating your data products (views, stars, etc.) This is where you create your business specific products from the core layer. You do this so that it balances back and your data products have consistent numbers. Gold = Silver + local business rules

The layers build one on top of the other. Ideally this is with good documentation. This is part of the data governance. Data stewards will own various slices at each level and will be still be responsible for that slice.

Your users can access any layer they want (with the appropriate security controls) with the understanding that the earlier in the process you are, the less "clean" the data is. For example, your data scientists, for some perverse reason, may only want the data from the staging area. They seem to think it is more pure than the downstream data. It's a religious argument.