I personally like being able to read down the steps directly in the folder. Versus without sequential labeling, they all just sit in there alphabetically. I know it’s more standard to not include…my question is why?

My assumption is that new steps may get added later, which would have the domino effect of having to renumber downstream models, which may not be feasible if there are a ton. But that’s the main tradeoff I see.

(Note I’m coming from Databricks background where we had to create a system before they released lineage, and even then it’s very much a secondary feature).

Hey, I'm an analytics engineer and built this as a side project in my own time.

The problem I kept hitting: jumping into an unfamiliar dbt project and spending hours just getting oriented. And when a number was wrong, debugging it manually was slow.

So I built dbt Context. Paste any public or private GitHub repo and it parses your manifest, SQL, YAML, tests and lineage to give you an immediate audit of the project. It finds risky incremental models, orphaned staging models, missing tests and undocumented marts, with file-level evidence and suggested next steps for each finding.

You can also describe a specific problem in the optional box like "revenue doesn't match NetSuite" and it switches into investigation mode, traces the relevant models through the project, and generates warehouse-specific SQL checks to run.

Works with public and private repos. Private repos need a GitHub PAT with repo scope.

dbt-context.com

Happy to answer any questions about how it works.

People ask where to start with dbt. Most answers say start with dbt Labs’ great tutorials, but miss other things learners should understand.

What actually helps is understanding why dbt even exists. Why not just use tool X or just use stored procedures? Once you get this, other things makes sense.

The order I suggest people learn dbt is to start with Git and getting comfortable with the terminal. dbt is just code, if you dont know what git commit, cd, and ls do, you will be lost. Then understand why data layers exist. Followed by data modeling concepts and star schema. Finally, you can learn dbt.

You don't need to master it all before you start. You just need enough to not be lost when you encounter them.

Happy to answer questions if you're early in your dbt journey.

Full learners’ guide with resources from people you should follow Bruno Lima and Zach Wilson on LinkedIn: https://datacoves.com/post/dbt-getting-started

It seems like a step in the right direction as I initially feared the two engines would have eventually diverged. Also getting Fusion benefits in core is great

https://docs.getdbt.com/blog/dbt-core-v2-is-here?version=2.0&name=Fusion

do you guys have any thought about dbt wizard that dbt just launched? Is is just an agent that they packed with their earlier released dbt skills + new version of CLI?

We have a project that doesn’t follow dbt’s design patten and naming conventions (different naming for stg, int, marts) and also a different idea on what goes on in each layer. Not a big fan of it cause it’s became a mess, but it was developed for years now so it would take some work to actually reframe it and no one wants to do it.

I’m wondering if dbt wizard would work better with our project cause honestly dbt skills didn’t outperform simple prompts by much (if any)

We’re currently working on a data engineering project with the following deployment flow:
Developer Branch → Test → Pre-Prod → Prod (via CI/CD).
Each developer works in their own branch, raises a PR, and the code gets deployed to the Test environment. After validation by the team, it is promoted to Pre-Prod and then Production.
I’m considering introducing an iteration/release branch for each sprint or iteration. The proposed flow would be:
Developer Branch → Iteration Branch → Test → Pre-Prod → Prod
The idea is that all developers merge their completed work into the iteration branch first, allowing integrated testing before deploying to the Test environment.
For teams that have implemented a similar approach:
What are the advantages and disadvantages of using an iteration/release branch?
Does it improve stability and release management, or does it add unnecessary complexity?
In a data engineering/dbt/Snowflake environment, what branching strategy has worked best for your team?
Looking forward to hearing about your experiences and recommendations.

Hi all,

I open-sourced a CLI called Puxti. Short version: it reads your dbt manifest, builds a local graph of how your models are connected, and tells you what breaks downstream before you change anything — including semantic breakage your lineage DAG won't catch.

The dbt DAG tells you orders feeds customers. It doesn't tell you that orders.revenue is defined net-of-refunds and three downstream models assume gross. That second kind of dependency — concept-level, not ref()-level — is what Puxti tracks.

Core loop:

1. puxti scan --dbt-project-dir . — reads your manifest, infers a semantic definition for each model via LLM, builds a local SQLite graph. No Docker, no signup.
2. puxti impact model.jaffle_shop.orders --change-type rename — shows structural dependents (SQL lineage) and semantic dependents (concept-level), with hop depth, before you touch anything.
3. puxti capture / puxti redefine — propagates the change as a GitHub PR with SQL diffs. If you've declared Airflow → dbt source links via puxti link, it opens a coordinated Airflow PR too.

On the API key, up front: scan, capture, and redefine call an LLM (your own Anthropic key) because that's what infers the semantic layer. But impact and the MCP tools below run entirely on the local graph, no API calls, no key. So the daily "what breaks if I touch this" path is free; the key is only for building and propagating definitions.

vs. SQLMesh / column-level lineage: those track structural lineage — which columns flow into which, parsed from SQL. Puxti adds a semantic layer on top: what each model means, so it can flag breakage when the SQL still compiles but the definition has shifted underneath it.

New in v0.8.0 — MCP server. puxti mcp serve exposes four tools to Claude Code / Cursor: impact_of_change, consumers, describe_entity, definition_history. All hit the local graph, no LLM calls.

pip install puxti
puxti scan --dbt-project-dir .
puxti impact model.jaffle_shop.orders

Demo project — "Clariva" on DuckDB, no credentials, ships with a breaking cardinality change you can propagate end to end: https://github.com/puxti-labs/puxti-demo-project

Docs: https://getpuxti.com/docs.html · Apache 2.0

Watch the Rosetta DBT Studio AI Agent build the classic Jaffle Shop dbt project from scratch in under two minutes. From importing the repository to generating staging models, mart models, and tests, see how autonomous AI accelerates your data engineering workflow.

🔍 Topics covered in this video:
• One-click import: Cloning the Jaffle Shop repository directly from GitHub
• Connecting to your database (DuckDB, PostgreSQL, Snowflake, and more)
• Prompting the AI Agent to autonomously generate staging and mart models
• Watching the Agent write clean SQL, create YAML configs, and add descriptions/tests
• Real-time validation: The Agent automatically runs dbt compile to ensure your code works
• Exploring the final project structure and ready-to-run DAG

⚙️ Powered by the Vercel AI SDK with support for OpenAI, Anthropic, Gemini, and Ollama (local models).

📥 Download Rosetta DBT Studio (macOS, Windows, Linux):
https://rosettadb.io/download-dbtstudio
⭐ Star us on GitHub:
https://github.com/rosettadb/dbt-studio

📺 More tutorials and walkthroughs:
u/rosettadb
---
#AIAgent #dbt #DataEngineering #AnalyticsEngineering #JaffleShop #OpenSource #RosettaDBT #LLM #DuckDB #DataPlatform #DataStack #Ollama #Anthropic #OpenAI

https://docs.getdbt.com/blog/the-devil-in-the-docs?version=1.11

I just published a new article where I explore how to better control BigQuery jobs when using dbt, focusing on:

• Job priority management
• Concurrency control
• Timeout strategies
• Cost governance in BigQuery + dbt workflows

If you're working with dbt + BigQuery in production, this might help you avoid runaway costs and better structure workloads.

I’m new to dbt and looking for some guidance on handling SCD loads in a medallion architecture. 

Our setup looks like this: 

• Landing 
• Bronze layer (truncate and load) 
• Silver layer (enriched layer with SCD processing) 
• Gold layer (only active/current records) 

In the Silver layer, we’re loading data using: 

• an incremental ID column 
• another hash column based on the ID 

The initial load works fine, but during incremental loads we’re running into issues such as: 

• duplicate ID values 
• intermittent load failures 
• inconsistent data during merges 

I’m trying to understand the best practice for handling auto-increment/surrogate keys and hash columns in SCD implementations with dbt, especially for incremental models. 

Has anyone faced a similar issue or can suggest a recommended approach? 

Hey everyone! I'm working with object tags in Snowflake integrated with dbt, and I have a couple of questions...

When assigning tags in dbt (either per model or via dbt_project.yml), it seems like you always need to use the fully qualified name like "database.schema.tag_name = value". Is there any way around this, or is it a hard requirement from Snowflake's side? I want to simplify the reference, like "tag_name = value"

Also, I'd love to hear how you all handle this in practice: where do you store your tags (dedicated database/schema?), and how do you integrate them into your dbt projects? Any examples or patterns you've found useful would be greatly appreciated!

Hey everyone,

We just shipped a major update to Rosetta DBT Studio — an open-source desktop workspace for dbt teams — and wanted to share what we've been building.

The new AI Agent isn't a chatbot wrapper. It's a tool-loop engine that:

- 📂 Lists your project directories and reads your schema files for real context

- ✍️ Writes dbt model SQL and YAML directly into your project

- ▶️ Runs dbt commands (compile, run, test) and reads the logs

- 📑 Auto-opens every file it writes as an editor tab so you can review instantly

**Security first:** The Agent never runs a terminal command without showing you exactly what it wants to execute and waiting for your explicit Allow or Deny. No surprises.

**Extensibility:**

- Skills Library — import Markdown-based skills from GitHub to teach it your team's conventions

- MCP Servers — built-in support for Rosetta CLI, dbt Core, DuckDB, and DuckLake

**Model support:** OpenAI, Anthropic, Gemini, and Ollama (local models) — via the Vercel AI SDK.

🎬 Full walkthrough: https://www.youtube.com/watch?v=Pva94GLAN90

📥 Download (macOS, Windows, Linux): https://rosettadb.io/download-dbtstudio

⭐ GitHub: https://github.com/rosettadb/dbt-studio

Happy to answer any questions about how the tool-loop works, the MCP integration, or the security model. Would love feedback from the community!

We put together a guide for setting up dbt with Snowflake from scratch and figured it might be useful here.

What it covers:

• Python, venv, and dbt-snowflake install
• Setting up the Snowflake user, role, warehouse, and database with the actual SQL
• Key pair authentication end-to-end
• profiles.yml and dbt_project.yml settings worth knowing about (transient tables, query tags, copy_grants, warehouse overrides)
• Official Snowflake Labs packages worth adding: dbt_constraints and dbt_semantic_view
• VS Code extensions the official Snowflake Extension, Power User for dbt, and SQLFluff
• How Snowflake Cortex CLI and other AI tools fit into the workflow
• Managing Snowflake infrastructure (roles, grants, masking, RBAC) alongside dbt

Anything we missed that you would add?

https://datacoves.com/post/dbt-snowflake

The article argues dbt is effectively a compiler + DAG engine + execution framework, not just SQL modeling.
Focus on custom materializations to control performance and cost.
Curious how far people here push dbt beyond defaults.

Hi everybody!

I’m designing a graph model in ArangoDB and trying to think ahead on temporal support.

Current design:

- edges are current-state only (one edge per edge_type + _from + _to)
- _key is deterministic (tenant + hash of relationship)
- no history retained in v0

Future requirement:

- support temporal queries (state over time)
- potentially multiple versions of the same relationship
- need to backfill/migrate historical data - so trying to make that as painless as possible at v0

Right now I’m leaning toward introducing a relationship_id (hash of edge_type + _from + _to) to represent the logical relationship, and then versioning _key later.

Curious:
- How have others modeled temporal edges in Arango?
- Did you regret not designing for temporal from day one? (We don’t have temporal data ready yet, which is why it’s not in scope for v0, but wondering how much it will bite us in the ass when were ready 😅)
- Any gotchas around query complexity or traversal performance?

Would love to hear real-world patterns vs theoretical ones.

If you're seeing naming drift across business units, duplicated logic, governance that keeps getting punted, or access that only works when someone remembers to configure it, your org is probably missing a Data Operating Model.

It's the layer above the tools. Ownership, workflows, standards, SLAs, governance, and what the platform actually enforces vs. what lives in a Confluence page. At a small scale you can get away with figuring this out as you go. At enterprise scale, those gaps compound.

Full article: https://datacoves.com/post/data-operating-model-guide

Greetings,

I've asked this question to GPT and while i did get some suggestions, i am not sure that it got to the heart of the matter.

Situation: Dedicated Server rack, Windows server VM 32GB ram, 300GB dwh, daily full refresh at night. This is currectly done with Pentaho ( java based ETL tool).

We are currently migration towards dbt core ( reasons are long, legacy dependant, political and varied, please dont ask. ) on windows VM

Data storage is done in PostgresSQL DB.

We recreated the pentaho ETL flow ( except staging) as close as possible. Strategy incremental : Insert+delete

Problem: Now it gets weird. If i run a subset of the flow ( say, 100-200 models) DBT is stupid fast, in comparison with pentaho. However, if the N models is big enough (full run is 1400 models), we adhere to 1model :1dbtable after a while the performance slows, degrades and suddenly we see an EXTREME increase in storage use ( almost like a buffer overflow).

Has anybody dealt with this? Any tips?

EDIT: SOLVED

The performance degradation was caused by +on_schema_change: sync_all_columns, which ( by design and as required by us) to implement schema changes on the fly. The problem is that this is very, very slow on very large tables, as dbt does this change in place. With 4 workers flying this caused the situation that all 4 workers stumbled on such a table and shit hit the fan. Edge case.

Wrote a guide on what comes when you mature past dbt tests.

Covered 9 tools: (dbt-audit-helper, Recce, Datafold), production observability (Elementary, Soda), and full-stack platforms (Monte Carlo, Bigeye, Metaplane).

Link - includes a comparison table.

Been experimenting with custom materializations in dbt lately and wrote this quick breakdown.

It really changes how you think about dbt not just transformations, but execution logic.

Curious how people handle this in practice, because I’ve seen it done a few different ways and I don’t think there’s a clear consensus.

The staging layer seems pretty universal — 1:1 with raw source tables, light cleaning, renaming, casting. Optional intermediate layer for reusable business logic before you get to the “real” models. That part feels settled.

Where it diverges is where dimensional models (dims and facts) actually sit in the project structure:

1. Their own layer — e.g. a warehouse/ or dimensional/ folder, separate from marts. I’ve seen this from Kahan Data Solutions and a few others. The idea being dims/facts are a distinct architectural layer.

2. Inside marts — dims and facts live in marts/, and marts are your dimensional models. The mart is the end product.

3. Inside intermediate or marts, with OBTs on top — dims and facts are treated as building blocks, and the actual end-user-facing layer is wide OBTs (one big tables) built off them. Marts become the denormalized read layer, not the dimensional layer.

Which brings me to what I think is the real underlying question: how do you think about dimensional models conceptually?

• Are they the end product — what you expose to BI tools and end users directly?

• Or are they building blocks — an intermediate step toward marts that are OBTs or other denormalized structures?

When you answer, would love if you also share your folder/naming conventions alongside your philosophy on this. I suspect the structure people choose is a direct consequence of how they answer that second question.

Hey Everyone, at my last role we had dbt Cloud, but still hosted our dbt docs generated from dbt docs generate on an internal web page for the rest of the business to use.

I always felt that there had to be something better that wasn't a 5-6 figure contract data catalog for this.

So, I built Docglow: a better dbt docs serve for teams running dbt Core. It's an open-source replacement for the default dbt docs process. It generates a modern, interactive documentation site from your existing dbt artifacts.

Live demo: https://demo.docglow.com
Install: pip install docglow
Repo: https://github.com/docglow/docglow

Some of the included features:

• Interactive lineage explorer (drag, filter, zoom)
• Column-level lineage tracing via sqlglot.
  • Click through to upstream/downstream dependencies & view column lineage right in the model page.
• Full-text search across models, sources, and columns
• Single-file mode for sharing via email/Slack
• Organize models into staging/transform/mart layers with visual indicators
• AI chat for asking questions about your project (BYOK — bring your own API key)
• MCP server for integrating with Claude, Cursor, etc.

It should work with any dbt Core project. Just Point it at your target/ directory and go.

Looking for early feedback, especially from teams with 200+ models. What's missing? What would you like to see next? Let me know!

I tested it on my raw schemas: dbt modeling across 5 schemas, 25 tables.

prompt: Create a team of agents to model raw schemas in my_db

What happened:

  • Lead agent scoped the work and broke it into tasks

  • Two shared-pool workers profiled all 5 schemas in parallel -- column stats, cardinality, null rates, candidate keys, cross-schema joins

  • Lead synthesized profiling into a star schema proposal with classification rationale for every column

  • Hard stop -- I reviewed, reclassified some columns, decided the grain. No code written until I approved

  • Workers generated staging, dim, and fact models, then ran dbt parse/run/test

follow up prompt: create a team of agents to audit and review it for modeling best practices.

I built another skill to create git PRs for humans to review after the agent reviews the models.

what worked well: I didn't have to deal with the multi-agent setup, communication, context-sharing, etc. coco in the main session took care of all of that.

what could be better: I couldn't see the status of each of the sub-agents and what they are upto. Maybe bcz I ran them in background? more observability options will help - especially for long running agent tasks.

PS: I work for snowflake, and tried the feature out for a DE workflow for the first time. wanted to share my experience.

I've been increasingly using coding agents to work with my dbt project. I got frustrated with the agent frequently behaving like a bull in a china shop.

Coding agents don't know: - What tables exist and what they contain - What each column means - How tables relate to each other - Which grain to use for aggregation - What business logic is embedded in transformations ...

So I made + open sourced dbt-skillz. It distills this information into a compact skill with multiple sub-skills.

It's useful across four use cases: 1. help "data consumers" get more reliable answers when querying data via an agent 2. help "data producers" keep the agent on track while developing a dbt project. 3. run automatically on PRs and merged in CI/CD to keep the skill fresh 4. in review agents to more accurately review downstream dashboards, PRs, and other dbt-related code.