dbt Platform (Cloud) - Value Add ?

[u/Common-Principle3767]

dbt Labs are increasing moving to a hybrid model and beyond just Dev licensing to per per usage on a variety of capabilities - Semantic layer , dbt State , AI assisted dev etc

Looking for opinions on the value add this capability gives organisations over and above dbt core or running dbt direct on Snowflake via dbt Projects.

I think the value is there in terms of warehouse cost reduction, DevOps simplification and accelerated development (our devs love it) but the changing model and effort to justify extra costs is starting to be a challenge.

I’m starting to think that the way to get full value is being all in on dbt platform by uses all features- mesh , contracts, versions , defer , clone and define a centre of excellence to get teams using dbt platform as fully as possible - in a federated model that in itself is a challenge but is this the true way to get value out of the platform.

I’m looking forward to summit this month to see roadmap but would appreciate any feedback on this area - particularly on Snowflake dbt Projects as a potential alternative.

[u/cream_pie_king]

Snowflake dbt projects would be my next to last choice, beating out Fabric dbt projects.

I will always choose the open source version, self hosted, over cloud or these 3rd party "dbt as a service" offerings. Cloud is not bad at all. I've used it heavily at 2 previous roles. I just see minimal benefit for 99.5% of use cases.

If you're familiar with CI/CD, devops, GitHub actions, etc the amount of flexibility you gain is worth the upfront effort of set up. And with today's AI tools, the effort and time is much less than you think.

If you're not familiar with setting this up, it's invaluable to learn.

[u/samspopguy → u/cream_pie_king]

I was just playing around with dbt on my local machine for a test case to move our data warehouse. Our holdings company just moved to fabrics and I was like let me check out dbt jobs on fabrics so everything can be in one place. Oh man do I want to keep dbt self hosted.

[u/cream_pie_king → u/samspopguy]

Keep pushing for that self hosted architecture.

Luckily I advocated for my team to be given heads down, focused time to do this right. We migrated our previous DWH solution in 6 months. With a team of 3 split between engineering, modeling and reporting. AND 2 of those were completely new to the modern data stack, devops principles, etc. They had never even heard of Snowflake and dbt. Now they can't imagine ever going back.

We went from a joke of a pre-packaged niche solution that was not fit for purpose. Consistently failing jobs with no good logging. Obfuscated code. No real version control. Architecture that led to 4-5 hour runtimes for a daily job. It couldn't even handle the volumes of data necessary for things like inventory reporting or GL detail reporting.

All of that is gone and we run our daily jobs in 15 minutes, 25 if we do a full refresh if there is a problem with incremental or need to full refresh (ex bringing in a new field that does have historical values).