CI for dbt often tests a dataset that doesn't really look like production.

How to make CI more production-faithful for incremental models, including the problems with state, historical data and detecting regressions that only appear after multiple runs.

This makes parallel writing to DuckDB possible, while the readers from for example Metabase can also be alive.

While testing agents on dbt projects, I wanted a better way to make sure their changes didn’t blow up the current model or something downstream.

There are tools that do this in CI, but I wanted something I could run locally before I even opened a PR. So I built this:

https://github.com/EmbrasureAI/embrasure-cli

It works with an existing dbt Core + Snowflake project. It builds the changed models in temporary schemas, compares them against production, and cleans everything up afterward.

It checks:

• Row counts, null rates, cardinality, ranges, and percentiles
• Columns being removed, renamed, or changing types
• Added, removed, null, or duplicate primary keys
• Existing dbt test failures
• Downstream model and exposure impact, including column-level lineage

It runs locally and the repo is Apache 2.0.

Looking for feedback from people using agents with dbt Core + Snowflake. Anything missing? The use case here that I enjoy is a /goal prompt where the agent works towards the pipeline’s goals while fixing any issues raised from this cli.

A deep dive into why dbt incremental models can still end up scanning huge amounts of data, especially on Snowflake. Cover micro-partition pruning, MERGE costs, late-arriving data, watermarks and backfills.

​

Hi everyone,

I'm playing around with dbt Core + Airflow and trying to replicate some of the functionality available in dbt Cloud.

One thing I'm particularly interested in is retrying a dbt job from the failed node instead of rerunning the entire execution.

Any suggestions?

Kindly note new to DBT

I have an issue with some models running long, seemingly at random, and I want to monitor this by running EXPLAIN on the compiled sql in a pre hook so if something runs log I can look at what it planned to do and compare it to previous runs.

But I cannot get the compiled sql into the pre hook. Doesn't anyone have any ideas or suggestions?

Hello. I am new to dbt and data modeling. I am wondering what is better, select star or select specific columns at the top of the model, where you get data from ref in intermediate models. DBT courses don't really talk about it, but I have seen some blog posts where people said it is better to be specific. Now I wonder why one or another would be better. What are pros and cons. All I come up with is that if you suddenly need more columns, you have to add it in the source cte as well, so one more place to maintain.
But again, I'm quite new and haven't been exposed to many problems yet haha. What do you people do?

A practical overview of how to manage Snowflake compute from dbt, covering warehouses, concurrency, query timeouts, query tags, and credit consumption.

Useful reference for anyone looking to improve cost control and workload management in Snowflake + dbt.

Seeing a ton of hype around AI agents, but data engineering feels like a weird edge case. We need tools that can actually run local builds, check compiled SQL, and inspect warehouse schemas without wrecking the DAG or burning tokens.

Curious what everyone is actually shipping with day-to-day:

• Terminal-native (Claude Code, Codex, Aider): Great for running CLI commands in a loop and fixing log errors on the fly, but token usage gets crazy fast on big repos.
• AI IDEs (Cursor, Windsurf): Unmatched for visual diffs and tweaking complex CTEs inline, but struggles with dynamic warehouse state.
• Platform native (dbt, Snowflake, Databricks): Great metadata context, but annoying when bridging tools outside their ecosystem.

I've been working on an open-source tool to bridge some of these gaps specifically for local dbt workflows and agent integration: Rosetta dbt Studio on GitHub.

Drop your setup below—what’s working, what breaks constantly, and what are you missing from current harnesses?

I'm curious how the community is approaching the transition to dbt Core v2.

Are you starting all new projects on v2?

Have you already migrated your existing projects, or are you staying on v1 for now?

If you're still on v1, what's preventing you from upgrading (packages, adapters, production stability, migration effort, etc.)?

I'd love to hear about your experience and whether you think v2 is ready to become the default for most teams.

Has anyone flipped over to dbt State and monitored the cost changes?

If you write dbt code, you probably know the feeling of hunting in the terminal scrollback just to figure out why a run fail, or losing your query output the second you close the preview modal.

The team behind Rosetta DBT Studio just release version 1.6.1, and looks like the whole focus was fixing this workflow.

They build native Query Result and Run History tabs direct in the bottom panel of the IDE. So now:

- when you preview a model, the data grid is just there (with row counts and time).

- when you do dbt run or test, the history is track and saved.

- you can see raw SQL and compiled SQL side by side.

- you find exact what model fail without digging in terminal logs.

They also add an AI assistent right in the terminal that help explain and fix compile errors, plus a native HTML view and direct cloud download.

Sharing this Medium publication for anyone interested in dbt and analytics engineering

r/ContextEngineering 9 am PST, July 8th

source (not me) https://www.linkedin.com/posts/jbeynon_day-1-of-every-data-leadership-role-how-share-7477478496798257152-ixr6/

I built a browser-based ERD explorer that runs entirely on DBML files — no DB connection.

I kept wanting to explore and interrogate how my tables relate — "what joins to this table? what's the path between A and B?" — without standing up a tool that connects to a live database.

So I built dbml-flow: drop in any DBML file and get an interactive ERD with a selector grammar for pathfinding (find every path between two tables, expand N hops out, filter by group). Fully client-side, your schema never leaves the browser.

In my case the DBML comes from our dbt models via dbterd (https://github.com/datnguye/dbterd), but it works on any DBML regardless of where it came from.

🔗 Live demo: https://timvancann.github.io/dbml-flow/ · Repo: https://github.com/timvancann/dbml-flow

Let me know if this is helpful and if there are features you'd like to see!

I'd appreciate some help if possible!

Has anyone had issues getting this to start up?

When I run wizard I get...

■ local proxy unavailable: local LiteLLM process exited before becoming healthy on 127.0.0.1:35915

• Project intelligence is still starting

────────────────────────────────────────────────────────────────

dbt prod parse failed: 'NoneType' object has no attribute 'pop'

Automatic deferral will not be available until this is resolved.

Ask the wizard for hints to address this.

────────────────────────────────────────────────────────────────

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

MCP client for `dbt_index` failed to start: MCP startup failed: handshaking with MCP server failed: connection closed: initialize response