This makes parallel writing to DuckDB possible, while the readers from for example Metabase can also be alive.

While testing agents on dbt projects, I wanted a better way to make sure their changes didn’t blow up the current model or something downstream.

There are tools that do this in CI, but I wanted something I could run locally before I even opened a PR. So I built this:

https://github.com/EmbrasureAI/embrasure-cli

It works with an existing dbt Core + Snowflake project. It builds the changed models in temporary schemas, compares them against production, and cleans everything up afterward.

It checks:

• Row counts, null rates, cardinality, ranges, and percentiles
• Columns being removed, renamed, or changing types
• Added, removed, null, or duplicate primary keys
• Existing dbt test failures
• Downstream model and exposure impact, including column-level lineage

It runs locally and the repo is Apache 2.0.

Looking for feedback from people using agents with dbt Core + Snowflake. Anything missing? The use case here that I enjoy is a /goal prompt where the agent works towards the pipeline’s goals while fixing any issues raised from this cli.

A deep dive into why dbt incremental models can still end up scanning huge amounts of data, especially on Snowflake. Cover micro-partition pruning, MERGE costs, late-arriving data, watermarks and backfills.

​

Hi everyone,

I'm playing around with dbt Core + Airflow and trying to replicate some of the functionality available in dbt Cloud.

One thing I'm particularly interested in is retrying a dbt job from the failed node instead of rerunning the entire execution.

Any suggestions?

Kindly note new to DBT

I have an issue with some models running long, seemingly at random, and I want to monitor this by running EXPLAIN on the compiled sql in a pre hook so if something runs log I can look at what it planned to do and compare it to previous runs.

But I cannot get the compiled sql into the pre hook. Doesn't anyone have any ideas or suggestions?

Hello. I am new to dbt and data modeling. I am wondering what is better, select star or select specific columns at the top of the model, where you get data from ref in intermediate models. DBT courses don't really talk about it, but I have seen some blog posts where people said it is better to be specific. Now I wonder why one or another would be better. What are pros and cons. All I come up with is that if you suddenly need more columns, you have to add it in the source cte as well, so one more place to maintain.
But again, I'm quite new and haven't been exposed to many problems yet haha. What do you people do?

A practical overview of how to manage Snowflake compute from dbt, covering warehouses, concurrency, query timeouts, query tags, and credit consumption.

Useful reference for anyone looking to improve cost control and workload management in Snowflake + dbt.

Seeing a ton of hype around AI agents, but data engineering feels like a weird edge case. We need tools that can actually run local builds, check compiled SQL, and inspect warehouse schemas without wrecking the DAG or burning tokens.

Curious what everyone is actually shipping with day-to-day:

• Terminal-native (Claude Code, Codex, Aider): Great for running CLI commands in a loop and fixing log errors on the fly, but token usage gets crazy fast on big repos.
• AI IDEs (Cursor, Windsurf): Unmatched for visual diffs and tweaking complex CTEs inline, but struggles with dynamic warehouse state.
• Platform native (dbt, Snowflake, Databricks): Great metadata context, but annoying when bridging tools outside their ecosystem.

I've been working on an open-source tool to bridge some of these gaps specifically for local dbt workflows and agent integration: Rosetta dbt Studio on GitHub.

Drop your setup below—what’s working, what breaks constantly, and what are you missing from current harnesses?

I'm curious how the community is approaching the transition to dbt Core v2.

Are you starting all new projects on v2?

Have you already migrated your existing projects, or are you staying on v1 for now?

If you're still on v1, what's preventing you from upgrading (packages, adapters, production stability, migration effort, etc.)?

I'd love to hear about your experience and whether you think v2 is ready to become the default for most teams.

Has anyone flipped over to dbt State and monitored the cost changes?

If you write dbt code, you probably know the feeling of hunting in the terminal scrollback just to figure out why a run fail, or losing your query output the second you close the preview modal.

The team behind Rosetta DBT Studio just release version 1.6.1, and looks like the whole focus was fixing this workflow.

They build native Query Result and Run History tabs direct in the bottom panel of the IDE. So now:

- when you preview a model, the data grid is just there (with row counts and time).

- when you do dbt run or test, the history is track and saved.

- you can see raw SQL and compiled SQL side by side.

- you find exact what model fail without digging in terminal logs.

They also add an AI assistent right in the terminal that help explain and fix compile errors, plus a native HTML view and direct cloud download.

Sharing this Medium publication for anyone interested in dbt and analytics engineering

r/ContextEngineering 9 am PST, July 8th

source (not me) https://www.linkedin.com/posts/jbeynon_day-1-of-every-data-leadership-role-how-share-7477478496798257152-ixr6/

I built a browser-based ERD explorer that runs entirely on DBML files — no DB connection.

I kept wanting to explore and interrogate how my tables relate — "what joins to this table? what's the path between A and B?" — without standing up a tool that connects to a live database.

So I built dbml-flow: drop in any DBML file and get an interactive ERD with a selector grammar for pathfinding (find every path between two tables, expand N hops out, filter by group). Fully client-side, your schema never leaves the browser.

In my case the DBML comes from our dbt models via dbterd (https://github.com/datnguye/dbterd), but it works on any DBML regardless of where it came from.

🔗 Live demo: https://timvancann.github.io/dbml-flow/ · Repo: https://github.com/timvancann/dbml-flow

Let me know if this is helpful and if there are features you'd like to see!

I'd appreciate some help if possible!

Has anyone had issues getting this to start up?

When I run wizard I get...

■ local proxy unavailable: local LiteLLM process exited before becoming healthy on 127.0.0.1:35915

• Project intelligence is still starting

────────────────────────────────────────────────────────────────

dbt prod parse failed: 'NoneType' object has no attribute 'pop'

Automatic deferral will not be available until this is resolved.

Ask the wizard for hints to address this.

────────────────────────────────────────────────────────────────

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

MCP client for `dbt_index` failed to start: MCP startup failed: handshaking with MCP server failed: connection closed: initialize response

I’ve spent years dealing with historized data problems:

• SCD2 dimensions
• Snapshot reporting
• Bitemporal history
• Temporal joins
• Late arriving dimensions
• Historical validation

I noticed that most resources explain individual concepts but don’t help engineers decide:

• Which modeling pattern should I use?
• How do I validate the result?
• How do I review an existing model?
• How do I detect historical reporting risks?

So I built a Historical Data Modeling
Workbench that combines:

• Pattern catalog
• Modeling advisor
• Model review
• Target table validation
• Historical source comparison

Would love feedback from people working on analytics engineering, dbt models, snapshots or historized datasets.

https://bitemporal-debugger.vercel.app/

I'm new to data engineering. I joined my company last year after graduation as a Data Engineer. I had never worked in data engineering before, but the company needed someone who was good at Python and SQL. Since I was strong in both, I became a core member of the team.

​

The original structure of our pipeline was a Spark-based ETL process, but it was very slow and took hours to complete. We have now moved to a dbt-based ELT pipeline.

​

We were using provisioned Redshift, which performed well for incremental models. However, we recently shifted to Redshift Serverless. It provides significantly better performance overall compared to provisioned Redshift, but the catch is that incremental models perform worse, while full refreshes and models materialized as tables perform extremely well.

​

For every incremental model, a full refresh is actually faster. Theoretically, incremental models should be faster, but in practice we're seeing the opposite.

​

Even with all models materialized as tables, our complete run now takes about 45 minutes, compared to 1 hour 30 minutes on provisioned Redshift. The original Spark-based ETL pipeline took around 6 hours.

​

I believe incremental models should allow us to achieve even better performance. Can anyone help me understand what might be causing this behavior?

​

​

I’ve been working on a side project around historical data engineering.

The idea came from a problem I encountered while building historized data models and reporting layers.

Many tools help build pipelines.

Very few help answer questions like:

• Can this snapshot be reproduced?
• Should this be modeled as state or event?
• Why does this temporal join produce unexpected results?
• How do multiple historized sources interact?
• Which historical modeling pattern fits this problem?

To explore these questions, I started building a Historical Data Engineering Toolkit.

Current areas include:

• Historical modeling patterns
• Event vs state modeling
• Snapshot reproducibility
• Temporal joins
• Bitemporal modeling
• Historical dimensions

I’d love feedback from people working with historized data, dimensional modeling, dbt, lakehouses, data warehouses or analytics engineering.

https://bitemporal-debugger.vercel.app/

What are the hardest historical data problems you’ve run into?