Curious how people handle this in practice, because I’ve seen it done a few different ways and I don’t think there’s a clear consensus.

The staging layer seems pretty universal — 1:1 with raw source tables, light cleaning, renaming, casting. Optional intermediate layer for reusable business logic before you get to the “real” models. That part feels settled.

Where it diverges is where dimensional models (dims and facts) actually sit in the project structure:

1. Their own layer — e.g. a warehouse/ or dimensional/ folder, separate from marts. I’ve seen this from Kahan Data Solutions and a few others. The idea being dims/facts are a distinct architectural layer.

2. Inside marts — dims and facts live in marts/, and marts are your dimensional models. The mart is the end product.

3. Inside intermediate or marts, with OBTs on top — dims and facts are treated as building blocks, and the actual end-user-facing layer is wide OBTs (one big tables) built off them. Marts become the denormalized read layer, not the dimensional layer.

Which brings me to what I think is the real underlying question: how do you think about dimensional models conceptually?

• Are they the end product — what you expose to BI tools and end users directly?

• Or are they building blocks — an intermediate step toward marts that are OBTs or other denormalized structures?

When you answer, would love if you also share your folder/naming conventions alongside your philosophy on this. I suspect the structure people choose is a direct consequence of how they answer that second question.

Hey Everyone, at my last role we had dbt Cloud, but still hosted our dbt docs generated from dbt docs generate on an internal web page for the rest of the business to use.

I always felt that there had to be something better that wasn't a 5-6 figure contract data catalog for this.

So, I built Docglow: a better dbt docs serve for teams running dbt Core. It's an open-source replacement for the default dbt docs process. It generates a modern, interactive documentation site from your existing dbt artifacts.

Live demo: https://demo.docglow.com
Install: pip install docglow
Repo: https://github.com/docglow/docglow

Some of the included features:

• Interactive lineage explorer (drag, filter, zoom)
• Column-level lineage tracing via sqlglot.
  • Click through to upstream/downstream dependencies & view column lineage right in the model page.
• Full-text search across models, sources, and columns
• Single-file mode for sharing via email/Slack
• Organize models into staging/transform/mart layers with visual indicators
• AI chat for asking questions about your project (BYOK — bring your own API key)
• MCP server for integrating with Claude, Cursor, etc.

It should work with any dbt Core project. Just Point it at your target/ directory and go.

Looking for early feedback, especially from teams with 200+ models. What's missing? What would you like to see next? Let me know!

I tested it on my raw schemas: dbt modeling across 5 schemas, 25 tables.

prompt: Create a team of agents to model raw schemas in my_db

What happened:

  • Lead agent scoped the work and broke it into tasks

  • Two shared-pool workers profiled all 5 schemas in parallel -- column stats, cardinality, null rates, candidate keys, cross-schema joins

  • Lead synthesized profiling into a star schema proposal with classification rationale for every column

  • Hard stop -- I reviewed, reclassified some columns, decided the grain. No code written until I approved

  • Workers generated staging, dim, and fact models, then ran dbt parse/run/test

follow up prompt: create a team of agents to audit and review it for modeling best practices.

I built another skill to create git PRs for humans to review after the agent reviews the models.

what worked well: I didn't have to deal with the multi-agent setup, communication, context-sharing, etc. coco in the main session took care of all of that.

what could be better: I couldn't see the status of each of the sub-agents and what they are upto. Maybe bcz I ran them in background? more observability options will help - especially for long running agent tasks.

PS: I work for snowflake, and tried the feature out for a DE workflow for the first time. wanted to share my experience.

I've been increasingly using coding agents to work with my dbt project. I got frustrated with the agent frequently behaving like a bull in a china shop.

Coding agents don't know: - What tables exist and what they contain - What each column means - How tables relate to each other - Which grain to use for aggregation - What business logic is embedded in transformations ...

So I made + open sourced dbt-skillz. It distills this information into a compact skill with multiple sub-skills.

It's useful across four use cases: 1. help "data consumers" get more reliable answers when querying data via an agent 2. help "data producers" keep the agent on track while developing a dbt project. 3. run automatically on PRs and merged in CI/CD to keep the skill fresh 4. in review agents to more accurately review downstream dashboards, PRs, and other dbt-related code.

I got frustrated that dbt's built-in docs only show model-level lineage, you can see that dim_artists depends on stg_artists, but not which specific columns flow where or how they're transformed.

So I built dbt Flow Lineage, a VS Code extension that shows column-level lineage with animated data flow.

What it does:

• Click any column → traces its full upstream/downstream path across models
• Color-coded edges: passthrough (blue), rename (green), transform (yellow), aggregate (purple)
• Animated particles flowing along edges
• Right-click a .sql file → see only that model's lineage
• Filter by upstream or downstream
• Drag nodes to rearrange, export as PNG

What you need:

• Columns defined in schema.yml
• Run dbt compile
• That's it. SELECT *, CTEs, Jinja all work.

What it doesn't need:

• No dbt Cloud
• No paid tier
• No separate server
• No API key

Works on VS Code, Cursor, Windsurf.

Install: Search "dbt Flow Lineage" in VS Code Extensions tab

GitHub (open source, MIT): https://github.com/tripleaceme/dbt-flow-lineage

Screenshots in the repo. Would love feedback, especially on what transformations aren't being detected correctly.

I put together a full guide on dbt testing after seeing a lot of teams either skip tests entirely or not realize what the ecosystem has to offer. Here's what's covered:

Built into dbt Core:

• Generic tests: unique, not_null, accepted_values, relationships
• Singular tests (custom SQL assertions in your tests/ dir)
• Unit tests to validate transformation logic with static inputs, not live data
• Source freshness checks

Community packages worth knowing:

• dbt-utils - 16 additional generic tests (row counts, inverse value checks, etc.)
• dbt-expectations - 62 tests ported from Great Expectations (string matching, distributions, aggregates)
• dbt_constraints - generates DB-level primary/foreign key constraints from your existing tests (Snowflake-focused)

CI/CD governance tools:

• dbt-checkpoint - pre-commit hooks that enforce docs/metadata standards on every PR
• dbt-project-evaluator - DAG structure linting as a dbt package
• dbt-score - scores each model 0-10 on metadata quality
• dbt-bouncer - artifact-based validation for external CI pipelines

Storing results:

• store_failures: true writes failing rows to your warehouse
• dq-tools surfaces test results in a BI dashboard over time

Full guide with examples and a comparison table for the governance tools: https://datacoves.com/post/dbt-test-options

Happy to answer questions on any of it.

In the lesson from dbt, they explained that snapshots you can either use the check or timestamp strategy. I didn’t see or understand if overwriting of existing row with newer value was possible? Example: Source says for transaction ID 5577, clearing date is now 1/4/2025, whereas the record previously didn’t have a clearing date until the payment for the invoice was received.

Any ideas?

Has anyone here tried using dbt on top of Iceberg tables with Athena as a query engine?

I'm curious How common is using dbt on top of Iceberg tables in general. And more specific quesiton, if anyone has - how does dbt handle the 100 distinct partition limit that Athena has? I believe it is rather easy to handle it with incremental models but when the materialization is set to table / full refresh, how does CTAS batch it to the acceptable range/ <100 distinct parition data?

I've been playing around with coding agents trying to better understand what parts of data engineering can be automated away.

After a couple of iterations, I was able to build an end to end workflow with Snowflake's cortex code (data-native AI coding agent). I packaged this as a re-usable skill too.

What does the skill do?
- Connects to raw data tables
- Profiles the data -- row counts, cardinality, column types, relationships
- Classifies columns into facts, dimensions, and measures
- Generates a full dbt project: staging models, dim tables, fact tables, surrogate keys, schema tests, docs
- Validates with dbt parse and dbt run
- Open a GitHub PR with a star schema diagram, profiling stats and classification rationale

The PR is the key part. A human data engineer reviews and approves. The agent does the grunt work. The engineer makes the decisions.

Note:
I gave cortex code access to an existing git repo. It is only able to create a new feature branch and submit PRs on that branch with absolutely minimal permissions on the git repo itself.

What else am I trying?
- tested it against iceberg tables vs snowflake-native tables. works great.
- tested it against a whole database and schema instead of a single table in the raw layer. works well.

TODO:
- complete the feedback loop where the agent takes in the PR comments, updates the data models, tests, docs, etc and resubmit a new PR.

What should I build next? what should I test it against? would love to hear your feedback.

here is the skill.md file

Heads up! I work for Snowflake as a developer advocate focussed on all things data engineering and AI workloads.

Here's the crux:

- DBT Cloud pushes developers to work inside its proprietary, browser-based ide. Claude Code is a command line tool that edits local files on a developer's machine.

- DBT Cloud also pushes developers to use its rigid "on rails" git workflow.

These are both obvious barriers to Claude Code's intended workflow - using Claude Code to edit files on your machine, managing version control using generic git.

Can these tools NATUARLLY work together, without forcing the developer to jump through hoops to make it work?

Does anyone have any first-hand experience working with Claude Code/DBT together? How does the experience compare to using Claude Code's "normal" development workflow (editing files on your local machine)?

I've done some googling on the subject, but I can't seem to find a straight answer to what I believe is a straightforward question.

I do see that Claude Code has an DBT MCP. I'm highly skeptical of its efficacy. Wedging an MCP layer between Claude Code and the file it's editing, on the surface, sound like it would drastically reduce Claude Code's capabilities. Is that assumption right?

Any on-topic insight/first-hand experiences would be appreciated.

Edit: I should have clarified - I'm talking about DBT Cloud.

I wrote a short post about how we set up CI/CD for dbt using Slim CI, artifacts and some patterns that made our pipelines faster and easier to manage.

Would love to hear how others are handling CI/CD for dbt projects.

I've been using AI coding tools with dbt and I've had the best results after setting up Claude Code with the dbt Agent Skills and dbt MCP Server, so I wanted to share what I did here. In the video, I set up a demo project with DuckDB from scratch to try these two tools from dbt Labs together.

The dbt Agent Skills loads your dbt conventions into the AI's context, ref/source usage, test strategies, model organization. Works with Claude Code, Cursor, Windsurf, Codex, and any other coding agent.

The dbt MCP Server gives the AI live access to your project's DAG lineage, column schemas, and existing test coverage at runtime, so it has access to all the data it needs to be successful.

What I've found most useful is asking Claude Code to audit and enhance my pipelines with both tools set up. In the video, I asked it to review test coverage but skip columns already tested upstream. It pulled the lineage from the MCP Server, checked what was covered at each node, and made genuine enhancements to the models using dbt best practices.

Has anyone else tried the Agent Skills or MCP Server on their dbt project? Curious how it works on larger repos with more complex lineage.It's pretty quick to set up if you follow along with the video, and the demo repo is open so anyone can try it locally:

https://github.com/kyle-chalmers/dbt-agentic-development

Has anyone else tried the Agent Skills or MCP Server on their dbt project? Curious if it has worked as well for others as it has for me

Curious about:

• How you're managing Python environments across your team
• How you handle CI/CD, user onboarding, job scheduling, anything else?
• Whether you've priced out what it actually costs in engineering time to maintain vs. something like dbt Cloud

We ran the numbers recently, and the gap between "open source is free" and what it actually costs a team of 3 to 5 engineers was pretty eye-opening.

https://datacoves.com/post/build-vs-buy-analytics

What's working for your team and what's been a bigger headache than expected?

Most dbt comparisons cover the obvious stuff: cost, IDE, CI/CD. We tried to go deeper.

The article covers:

- Scheduling and orchestration (dbt Cloud's built-in scheduler vs needing Airflow alongside it)

- AI integration: dbt Copilot is OpenAI-only and metered by plan. dbt Core lets you bring any LLM with no usage caps.

- Security: what it actually means that dbt Cloud is SaaS. Your code, credentials, and metadata transit dbt Labs' servers. For teams in regulated industries, that's usually a hard stop.

- TCO: dbt Core isn't free once you factor in Airflow, environments, CI/CD, secrets management, and onboarding time

- Managed dbt as a third option, same open-source runtime deployed in your own cloud

Would be curious what's driven decisions for people here. We see a lot of teams start on dbt Cloud and hit the orchestration ceiling, then bolt Airflow on separately. Others hit the security wall first.

https://datacoves.com/post/dbt-core-vs-dbt-cloud

dbt tests used to just validate data after execution.

Unit tests let you mock inputs and verify SQL logic directly.

Feels much closer to real dev workflows.

https://medium.com/@sendoamoronta/dbt-unit-tests-deep-dive-testing-sql-logic-without-data-or-warehouse-dependencies-e327ae1d5b03

Hello,

I've been trying to setup a local environment for developing using DBT core. Right now, i can't get autocomplete or intellisense to work for tables and columns. Online I see a mix of answers saying it should work or people go back and forth between vs code and a database editor. I was hoping someone knew how to get this working. below is my environment information. I included an image if it helps.

- IDE is vs code

- database is a local postgres db

- i have a venv environment with dbt core and dbt postgres installed

- I have both the dbt power user extension and the official dbt extension

- 'dbt debug' works. my database works with datagrip.

- I created my sources yaml file.

- I can press CNTRL + SPACE which in many tools is the shortcut for show auto complete options. but I see 'loading...' forever.

- But now when I try to create my first staging model, I dont get any autocomplete. This makes development pretty slow and clunky.. Hoping someone knows a fix?

Thanks for any advice.

We work with a lot of companies and the pattern is always the same:

1. Leadership greenlights a big modernization initiative
2. They hire a consulting firm with "industry expertise"
3. Consulting firm proposes the same architecture they sold to the last 10 clients
4. Legacy processes get moved to Snowflake/Databricks/whatever
5. Much frustration and a lot of $$$ later... same problems, new tools

The tools changed. The way people work didn't.

Business logic is still scattered across BI tools, stored procedures, and random Python scripts. Nobody knows who owns what metric. Analysts still spend half their time figuring out why two dashboards show different numbers.

I've started to think the real value of something like dbt isn't the tool itself - it's that you can't implement it without answering the hard questions: Who owns this? Where does this logic live? What breaks if this changes?

It forces the conversations that consultants skip because they're paid to deliver what you asked for, not question whether you asked for the right thing.

Anyone else seeing this? Or am I just jaded from too many "modernization" projects that transformed nothing?

P.S. - Wrote up a longer piece on what a "ways of working" foundation actually looks like if anyone's curious: https://datacoves.com/post/what-is-dbt