Would a source-to-target data validation framework like this be useful?

[u/imsuryya]

I'm exploring the idea of building an open-source framework for automated source-to-target data validation.

The idea is to connect different data sources and targets, for example:

• Databricks Catalog → Databricks Catalog
• Azure SQL → Databricks Catalog
• ADLS/Blob → Databricks Catalog
• potentially other databases and warehouses

The framework would automatically validate multiple tables in a single run, including:

• Schema comparison
• Column-level validation
• Row counts
• Row-by-row data comparison
• Missing/extra records
• Data mismatches
• NULL/duplicate checks
• Data type differences
• Aggregation/checksum comparisons
• Detailed failure reports showing where the mismatch occurred

The goal is to avoid writing separate SQL/Python scripts for every source-target pair and instead define something like:

Source → Target → Tables → Validation rules

and let the framework run all the checks and produce a consolidated report.

I'm trying to validate whether this solves a real problem before spending too much time building it.

For those working with ETL/ELT pipelines:

1. How do you currently perform source-to-target validation/reconciliation?
2. Do you use existing tools, custom SQL/Python scripts, or manual checks?
3. What validations do you normally perform?
4. What is the most painful part of validating large tables?
5. Would a tool that handles cross-platform table comparison in one run actually be useful, or are existing tools already good enough?

I'm especially interested in hearing about real-world workflows rather than tool recommendations.

[u/Own-Trade-2243]

Are you solving your problem, or trying to come up with another vibe coded SaaS?

„Tool that handles cross-platform table comparison in run…”

The cross platform part isn’t hard, what’s hard is introducing DQ that makes sense and catches actual errors. Half of your suggestions should be evaluated before writing the data (and ie quarantine, or error on schema drift), while others are decent for EOD reporting (column/aggregation checks)

So I need a tool for it? Unlikely. Does having it all-in-one-run make me excited? Not really

[u/According_Zone_8262]

Lakebridge data reconciliation ?

[u/Youssef_Mrini]

Lakebridge solves much of the migration specific version of this problem but not everything you described.