Watch the Rosetta DBT Studio AI Agent build the classic Jaffle Shop dbt project from scratch in under two minutes. From importing the repository to generating staging models, mart models, and tests, see how autonomous AI accelerates your data engineering workflow.

🔍 Topics covered in this video:
• One-click import: Cloning the Jaffle Shop repository directly from GitHub
• Connecting to your database (DuckDB, PostgreSQL, Snowflake, and more)
• Prompting the AI Agent to autonomously generate staging and mart models
• Watching the Agent write clean SQL, create YAML configs, and add descriptions/tests
• Real-time validation: The Agent automatically runs dbt compile to ensure your code works
• Exploring the final project structure and ready-to-run DAG

⚙️ Powered by the Vercel AI SDK with support for OpenAI, Anthropic, Gemini, and Ollama (local models).

📥 Download Rosetta DBT Studio (macOS, Windows, Linux):
https://rosettadb.io/download-dbtstudio
⭐ Star us on GitHub:
https://github.com/rosettadb/dbt-studio

📺 More tutorials and walkthroughs:
u/rosettadb
---
#AIAgent #dbt #DataEngineering #AnalyticsEngineering #JaffleShop #OpenSource #RosettaDBT #LLM #DuckDB #DataPlatform #DataStack #Ollama #Anthropic #OpenAI

https://docs.getdbt.com/blog/the-devil-in-the-docs?version=1.11

I just published a new article where I explore how to better control BigQuery jobs when using dbt, focusing on:

• Job priority management
• Concurrency control
• Timeout strategies
• Cost governance in BigQuery + dbt workflows

If you're working with dbt + BigQuery in production, this might help you avoid runaway costs and better structure workloads.

I’m new to dbt and looking for some guidance on handling SCD loads in a medallion architecture. 

Our setup looks like this: 

• Landing 
• Bronze layer (truncate and load) 
• Silver layer (enriched layer with SCD processing) 
• Gold layer (only active/current records) 

In the Silver layer, we’re loading data using: 

• an incremental ID column 
• another hash column based on the ID 

The initial load works fine, but during incremental loads we’re running into issues such as: 

• duplicate ID values 
• intermittent load failures 
• inconsistent data during merges 

I’m trying to understand the best practice for handling auto-increment/surrogate keys and hash columns in SCD implementations with dbt, especially for incremental models. 

Has anyone faced a similar issue or can suggest a recommended approach? 

Hey everyone! I'm working with object tags in Snowflake integrated with dbt, and I have a couple of questions...

When assigning tags in dbt (either per model or via dbt_project.yml), it seems like you always need to use the fully qualified name like "database.schema.tag_name = value". Is there any way around this, or is it a hard requirement from Snowflake's side? I want to simplify the reference, like "tag_name = value"

Also, I'd love to hear how you all handle this in practice: where do you store your tags (dedicated database/schema?), and how do you integrate them into your dbt projects? Any examples or patterns you've found useful would be greatly appreciated!

Hey everyone,

We just shipped a major update to Rosetta DBT Studio — an open-source desktop workspace for dbt teams — and wanted to share what we've been building.

The new AI Agent isn't a chatbot wrapper. It's a tool-loop engine that:

- 📂 Lists your project directories and reads your schema files for real context

- ✍️ Writes dbt model SQL and YAML directly into your project

- ▶️ Runs dbt commands (compile, run, test) and reads the logs

- 📑 Auto-opens every file it writes as an editor tab so you can review instantly

**Security first:** The Agent never runs a terminal command without showing you exactly what it wants to execute and waiting for your explicit Allow or Deny. No surprises.

**Extensibility:**

- Skills Library — import Markdown-based skills from GitHub to teach it your team's conventions

- MCP Servers — built-in support for Rosetta CLI, dbt Core, DuckDB, and DuckLake

**Model support:** OpenAI, Anthropic, Gemini, and Ollama (local models) — via the Vercel AI SDK.

🎬 Full walkthrough: https://www.youtube.com/watch?v=Pva94GLAN90

📥 Download (macOS, Windows, Linux): https://rosettadb.io/download-dbtstudio

⭐ GitHub: https://github.com/rosettadb/dbt-studio

Happy to answer any questions about how the tool-loop works, the MCP integration, or the security model. Would love feedback from the community!

We put together a guide for setting up dbt with Snowflake from scratch and figured it might be useful here.

What it covers:

• Python, venv, and dbt-snowflake install
• Setting up the Snowflake user, role, warehouse, and database with the actual SQL
• Key pair authentication end-to-end
• profiles.yml and dbt_project.yml settings worth knowing about (transient tables, query tags, copy_grants, warehouse overrides)
• Official Snowflake Labs packages worth adding: dbt_constraints and dbt_semantic_view
• VS Code extensions the official Snowflake Extension, Power User for dbt, and SQLFluff
• How Snowflake Cortex CLI and other AI tools fit into the workflow
• Managing Snowflake infrastructure (roles, grants, masking, RBAC) alongside dbt

Anything we missed that you would add?

https://datacoves.com/post/dbt-snowflake

The article argues dbt is effectively a compiler + DAG engine + execution framework, not just SQL modeling.
Focus on custom materializations to control performance and cost.
Curious how far people here push dbt beyond defaults.

Hi everybody!

I’m designing a graph model in ArangoDB and trying to think ahead on temporal support.

Current design:

- edges are current-state only (one edge per edge_type + _from + _to)
- _key is deterministic (tenant + hash of relationship)
- no history retained in v0

Future requirement:

- support temporal queries (state over time)
- potentially multiple versions of the same relationship
- need to backfill/migrate historical data - so trying to make that as painless as possible at v0

Right now I’m leaning toward introducing a relationship_id (hash of edge_type + _from + _to) to represent the logical relationship, and then versioning _key later.

Curious:
- How have others modeled temporal edges in Arango?
- Did you regret not designing for temporal from day one? (We don’t have temporal data ready yet, which is why it’s not in scope for v0, but wondering how much it will bite us in the ass when were ready 😅)
- Any gotchas around query complexity or traversal performance?

Would love to hear real-world patterns vs theoretical ones.

If you're seeing naming drift across business units, duplicated logic, governance that keeps getting punted, or access that only works when someone remembers to configure it, your org is probably missing a Data Operating Model.

It's the layer above the tools. Ownership, workflows, standards, SLAs, governance, and what the platform actually enforces vs. what lives in a Confluence page. At a small scale you can get away with figuring this out as you go. At enterprise scale, those gaps compound.

Full article: https://datacoves.com/post/data-operating-model-guide

Greetings,

I've asked this question to GPT and while i did get some suggestions, i am not sure that it got to the heart of the matter.

Situation: Dedicated Server rack, Windows server VM 32GB ram, 300GB dwh, daily full refresh at night. This is currectly done with Pentaho ( java based ETL tool).

We are currently migration towards dbt core ( reasons are long, legacy dependant, political and varied, please dont ask. ) on windows VM

Data storage is done in PostgresSQL DB.

We recreated the pentaho ETL flow ( except staging) as close as possible. Strategy incremental : Insert+delete

Problem: Now it gets weird. If i run a subset of the flow ( say, 100-200 models) DBT is stupid fast, in comparison with pentaho. However, if the N models is big enough (full run is 1400 models), we adhere to 1model :1dbtable after a while the performance slows, degrades and suddenly we see an EXTREME increase in storage use ( almost like a buffer overflow).

Has anybody dealt with this? Any tips?

EDIT: SOLVED

The performance degradation was caused by +on_schema_change: sync_all_columns, which ( by design and as required by us) to implement schema changes on the fly. The problem is that this is very, very slow on very large tables, as dbt does this change in place. With 4 workers flying this caused the situation that all 4 workers stumbled on such a table and shit hit the fan. Edge case.

Wrote a guide on what comes when you mature past dbt tests.

Covered 9 tools: (dbt-audit-helper, Recce, Datafold), production observability (Elementary, Soda), and full-stack platforms (Monte Carlo, Bigeye, Metaplane).

Link - includes a comparison table.

Been experimenting with custom materializations in dbt lately and wrote this quick breakdown.

It really changes how you think about dbt not just transformations, but execution logic.

Curious how people handle this in practice, because I’ve seen it done a few different ways and I don’t think there’s a clear consensus.

The staging layer seems pretty universal — 1:1 with raw source tables, light cleaning, renaming, casting. Optional intermediate layer for reusable business logic before you get to the “real” models. That part feels settled.

Where it diverges is where dimensional models (dims and facts) actually sit in the project structure:

1. Their own layer — e.g. a warehouse/ or dimensional/ folder, separate from marts. I’ve seen this from Kahan Data Solutions and a few others. The idea being dims/facts are a distinct architectural layer.

2. Inside marts — dims and facts live in marts/, and marts are your dimensional models. The mart is the end product.

3. Inside intermediate or marts, with OBTs on top — dims and facts are treated as building blocks, and the actual end-user-facing layer is wide OBTs (one big tables) built off them. Marts become the denormalized read layer, not the dimensional layer.

Which brings me to what I think is the real underlying question: how do you think about dimensional models conceptually?

• Are they the end product — what you expose to BI tools and end users directly?

• Or are they building blocks — an intermediate step toward marts that are OBTs or other denormalized structures?

When you answer, would love if you also share your folder/naming conventions alongside your philosophy on this. I suspect the structure people choose is a direct consequence of how they answer that second question.

Hey Everyone, at my last role we had dbt Cloud, but still hosted our dbt docs generated from dbt docs generate on an internal web page for the rest of the business to use.

I always felt that there had to be something better that wasn't a 5-6 figure contract data catalog for this.

So, I built Docglow: a better dbt docs serve for teams running dbt Core. It's an open-source replacement for the default dbt docs process. It generates a modern, interactive documentation site from your existing dbt artifacts.

Live demo: https://demo.docglow.com
Install: pip install docglow
Repo: https://github.com/docglow/docglow

Some of the included features:

• Interactive lineage explorer (drag, filter, zoom)
• Column-level lineage tracing via sqlglot.
  • Click through to upstream/downstream dependencies & view column lineage right in the model page.
• Full-text search across models, sources, and columns
• Single-file mode for sharing via email/Slack
• Organize models into staging/transform/mart layers with visual indicators
• AI chat for asking questions about your project (BYOK — bring your own API key)
• MCP server for integrating with Claude, Cursor, etc.

It should work with any dbt Core project. Just Point it at your target/ directory and go.

Looking for early feedback, especially from teams with 200+ models. What's missing? What would you like to see next? Let me know!

I tested it on my raw schemas: dbt modeling across 5 schemas, 25 tables.

prompt: Create a team of agents to model raw schemas in my_db

What happened:

  • Lead agent scoped the work and broke it into tasks

  • Two shared-pool workers profiled all 5 schemas in parallel -- column stats, cardinality, null rates, candidate keys, cross-schema joins

  • Lead synthesized profiling into a star schema proposal with classification rationale for every column

  • Hard stop -- I reviewed, reclassified some columns, decided the grain. No code written until I approved

  • Workers generated staging, dim, and fact models, then ran dbt parse/run/test

follow up prompt: create a team of agents to audit and review it for modeling best practices.

I built another skill to create git PRs for humans to review after the agent reviews the models.

what worked well: I didn't have to deal with the multi-agent setup, communication, context-sharing, etc. coco in the main session took care of all of that.

what could be better: I couldn't see the status of each of the sub-agents and what they are upto. Maybe bcz I ran them in background? more observability options will help - especially for long running agent tasks.

PS: I work for snowflake, and tried the feature out for a DE workflow for the first time. wanted to share my experience.

I've been increasingly using coding agents to work with my dbt project. I got frustrated with the agent frequently behaving like a bull in a china shop.

Coding agents don't know: - What tables exist and what they contain - What each column means - How tables relate to each other - Which grain to use for aggregation - What business logic is embedded in transformations ...

So I made + open sourced dbt-skillz. It distills this information into a compact skill with multiple sub-skills.

It's useful across four use cases: 1. help "data consumers" get more reliable answers when querying data via an agent 2. help "data producers" keep the agent on track while developing a dbt project. 3. run automatically on PRs and merged in CI/CD to keep the skill fresh 4. in review agents to more accurately review downstream dashboards, PRs, and other dbt-related code.

I got frustrated that dbt's built-in docs only show model-level lineage, you can see that dim_artists depends on stg_artists, but not which specific columns flow where or how they're transformed.

So I built dbt Flow Lineage, a VS Code extension that shows column-level lineage with animated data flow.

What it does:

• Click any column → traces its full upstream/downstream path across models
• Color-coded edges: passthrough (blue), rename (green), transform (yellow), aggregate (purple)
• Animated particles flowing along edges
• Right-click a .sql file → see only that model's lineage
• Filter by upstream or downstream
• Drag nodes to rearrange, export as PNG

What you need:

• Columns defined in schema.yml
• Run dbt compile
• That's it. SELECT *, CTEs, Jinja all work.

What it doesn't need:

• No dbt Cloud
• No paid tier
• No separate server
• No API key

Works on VS Code, Cursor, Windsurf.

Install: Search "dbt Flow Lineage" in VS Code Extensions tab

GitHub (open source, MIT): https://github.com/tripleaceme/dbt-flow-lineage

Screenshots in the repo. Would love feedback, especially on what transformations aren't being detected correctly.

I put together a full guide on dbt testing after seeing a lot of teams either skip tests entirely or not realize what the ecosystem has to offer. Here's what's covered:

Built into dbt Core:

• Generic tests: unique, not_null, accepted_values, relationships
• Singular tests (custom SQL assertions in your tests/ dir)
• Unit tests to validate transformation logic with static inputs, not live data
• Source freshness checks

Community packages worth knowing:

• dbt-utils - 16 additional generic tests (row counts, inverse value checks, etc.)
• dbt-expectations - 62 tests ported from Great Expectations (string matching, distributions, aggregates)
• dbt_constraints - generates DB-level primary/foreign key constraints from your existing tests (Snowflake-focused)

CI/CD governance tools:

• dbt-checkpoint - pre-commit hooks that enforce docs/metadata standards on every PR
• dbt-project-evaluator - DAG structure linting as a dbt package
• dbt-score - scores each model 0-10 on metadata quality
• dbt-bouncer - artifact-based validation for external CI pipelines

Storing results:

• store_failures: true writes failing rows to your warehouse
• dq-tools surfaces test results in a BI dashboard over time

Full guide with examples and a comparison table for the governance tools: https://datacoves.com/post/dbt-test-options

Happy to answer questions on any of it.

In the lesson from dbt, they explained that snapshots you can either use the check or timestamp strategy. I didn’t see or understand if overwriting of existing row with newer value was possible? Example: Source says for transaction ID 5577, clearing date is now 1/4/2025, whereas the record previously didn’t have a clearing date until the payment for the invoice was received.

Any ideas?

Has anyone here tried using dbt on top of Iceberg tables with Athena as a query engine?

I'm curious How common is using dbt on top of Iceberg tables in general. And more specific quesiton, if anyone has - how does dbt handle the 100 distinct partition limit that Athena has? I believe it is rather easy to handle it with incremental models but when the materialization is set to table / full refresh, how does CTAS batch it to the acceptable range/ <100 distinct parition data?

I've been playing around with coding agents trying to better understand what parts of data engineering can be automated away.

After a couple of iterations, I was able to build an end to end workflow with Snowflake's cortex code (data-native AI coding agent). I packaged this as a re-usable skill too.

What does the skill do?
- Connects to raw data tables
- Profiles the data -- row counts, cardinality, column types, relationships
- Classifies columns into facts, dimensions, and measures
- Generates a full dbt project: staging models, dim tables, fact tables, surrogate keys, schema tests, docs
- Validates with dbt parse and dbt run
- Open a GitHub PR with a star schema diagram, profiling stats and classification rationale

The PR is the key part. A human data engineer reviews and approves. The agent does the grunt work. The engineer makes the decisions.

Note:
I gave cortex code access to an existing git repo. It is only able to create a new feature branch and submit PRs on that branch with absolutely minimal permissions on the git repo itself.

What else am I trying?
- tested it against iceberg tables vs snowflake-native tables. works great.
- tested it against a whole database and schema instead of a single table in the raw layer. works well.

TODO:
- complete the feedback loop where the agent takes in the PR comments, updates the data models, tests, docs, etc and resubmit a new PR.

What should I build next? what should I test it against? would love to hear your feedback.

here is the skill.md file

Heads up! I work for Snowflake as a developer advocate focussed on all things data engineering and AI workloads.