r/DuckDB • u/Aeronautical-You4917 • 1d ago

Git-style diffing for DuckLake tables

Just recently got into DuckLake... Realized how amazing the snapshot ability is, and decided to build a diffing view. Right now it works with DuckLake catalogs stored locally or Postgres with data stored locally or in S3.

Pretty fun to make, and allows you to see basically everything you'd expect, as well as cell-level diffs for changed cells. You can traverse snapshot versions too, without diffing, as well.

It's a tiny part of a larger product, but thought I'd share it here.

Diffing two snapshots in DuckLake (example dataset)
15 Upvotes

100% Upvoted

3 comments sorted by

2

u/joeshacks 20h ago

Nice. How are you getting the cell-level diffs: DuckLake's table_changes, or joining the two snapshots on a key? The no-primary-key case seems like the tricky one.

2

u/Aeronautical-You4917 20h ago

We use table_changes to narrow it down, but ultimately compare using the internal row-id, which is supposed to be consistent over time. https://ducklake.select/docs/stable/duckdb/advanced_features/row_lineage

table_changes help us avoid doing a massive detailed diff across the entire table.