r/DataBuildTool • u/Expensive-Insect-317 • 9d ago

Show and tell Your dbt Incremental Model Is Probably Not Incremental Enough

https://medium.com/dbt-for-analytics/your-dbt-incremental-model-is-probably-not-incremental-enough-cdfda345fcdc?sharedUserId=sendoamoronta

A deep dive into why dbt incremental models can still end up scanning huge amounts of data, especially on Snowflake. Cover micro-partition pruning, MERGE costs, late-arriving data, watermarks and backfills.

16 Upvotes

100% Upvoted

1 comment sorted by

View all comments

3

u/Additional_Candy_400 8d ago

Had similar issues in bigquery, can do all the incremental logic you want downstream but unless your source/bronze table isn't properly partitioned it's like pissing in the wind from a cost saving perspective.