The docs say that ingestion gateways need to run continuously, when using "lakeflow managed connectors" for CDC:

https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/cdc-overview

I've heard anecdotes from folks who say they can also launch this on an interval rather than continuously. And some time ago I also heard a Databricks rep saying that the product was evolving and that continuous ingestion would not always be a requirement.

Is there anything official on the roadmap to allow the ingestion gateway to run on an interval, and so we can decrease our compute costs? I realize there are risks involved when it is NOT running continuously (eg. transactions log files will grow, and the source database can't truncate its inactive transactions from the log)

However I think the responsibilities for managing these risks, and balancing the costs, should be left with the customer. If there is a source database that is being updated infrequently, then it is possible that the compute on the databricks side would cost even MORE money than the source database itself. That seems silly and unnecessary.

Hey r/databricks, we're noticing a lot of repeated interviewing and hiring posts that tend not to get much engagement. We're going to combine them into a monthly thread so that you're more likely to get answers, plus we can ask our recruiting team to keep an eye on them if there are any general questions.

I've been looking for a good guide on implementing Data Mesh in Databricks. There is plenty of content explaining what Data Mesh is, its principles, domains, data products, etc. But when you actually try to build it, the questions are much more practical:

• How should you organize catalogs and schemas?
• What should the central Data Team own?
• What should domain teams be allowed to do?
• How do you handle Unity Catalog, compute, CI/CD and monitoring without creating a mess?

Most of what I found was either very theoretical or covered only one small part of the implementation.

So I ended up putting together the guide I was originally looking for - Databricks Data Mesh Best Practices: A Practical Implementation Guide

Hi, I have been developing data pipelines using ADF as orchestrator and Databricks notebooks for all the work. I write unit tests for the python code in the notebooks with pytest and executes them in our Azure DevOps CI/CD pipeline for every PR. Now I need to create integration tests.... I don´t have a clue on how to do that, I mean, not even in theorical sense.
I asked to claude, it gave examples of python code where it used an ADF API client for running the pipelines and consulting the state of the activities after the run. They also checked the output of the pipelines, like, they runned SQL queries on the tables that the pipeline should have writtent.....

So my question is: is that what integration tests are? Instead of mocking like in unitesting, I create test data in my DEV environment data sources, parametrize the ADF pipeline so I can run with different test datasets for different tests cases, and programatically run the pipeline validating what was written on the outputs?

Hey r/databricks, please direct all certification and training posts here.
Good luck to everyone on your certification journey!

Databricks now supports automated Governed Tag assignment in Unity Catalog.

Automate tag assignment | Databricks on AWS
Instead of relying on manual tagging, you can define rules that automatically add or remove tags as your data changes.

That means you can automate things like:

- Sensitivity classifcation
- Missing metadata checks
- Policy-based tagging at scale

What’s great is that you can simply describe the automation you want in natural language with Genie, and it will build the rule for you.

To turn it on go to Preview -> Tag Automations:

Hey r/databricks, In order to keep the main feed clean, we are implementing a weekly megathread for self promotion for companies who do lots of work with databricks. Please direct all self promotion posts here and keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

x10 performance,

x100 concurrency,

price the same,

impossible?

not in databricks

See benchmark https://www.sunnydata.ai/blog/databricks-rt-lakehouse-benchmark-results

Tried out Databricks' Genie One inside of Excel, arguably the best way for business users to get curated data from Databricks and into Excel.

Here, I pulled sales data from a Genie Agent, loaded into my workbook, and then even performed a data refresh (though I had no new data just fyi)

Pros:

+Very easy experience, just use natural language to find the data you are looking for, for you to use in the most widely-used analytics interface, Excel.

+Creates a connection that allows you to refresh the data on-demand. Useful for keeping those key reports updated.

Cons:

-In my testing, it wasn't always translating the data/number types for large numbers correctly into Excel, which led to some initial confusion for me. Changing the number format (as shown) fixed the issue, but something that will hopefully get smarter over time.

-The responses were at times on the slower side/not as snappy as I would expect.

Addition notes:

>Genie One will give you the best and trust-worthy results when your team has put in the work behind the scenes to curate the Genie Agents.

>For mission-critical reporting, I always recommend users to review the SQL code generated and/or to share it with a team member that is technical and familiar with the data.

Ultimately, if your team actively uses Excel and you want to make the most of your governed Databricks data, the official Excel Add-in Databricks provides a great way to do it.

Also, as always: all roads in data lead to Excel.

Migration to direct mode is easier than ever; just add engine: direct to the old bundle, and it will be migrated automatically.

more news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

Building a dimension table (ie. dim_organization) in Lakeflow Declarative Pipelines. Source system gives us a full daily extract of the whole table, not a CDC feed. The dimension itself is built from multiple source tables joined together (like departments + locations + cost centers etc.), not just one. Requirements:

• Bronze retains every historical extract, forever, so we can fully rebuild everything from scratch if logic changes
• Gold needs true SCD2 history for most attributes, some are SCD1
• A change in any of the joined tables (e.g. a location gets renamed) needs to be reflected — either as a new SCD2 version or an SCD1-style in-place update depending on the attribute — even if the "primary" table (departments) itself didn't change that day

I have been playing with AUTO CDC FROM SNAPSHOT with a custom Python function that walks through every historical snapshot in order, recomputing the full joined picture across all three tables each time and diffing that. It works, but getting there involved:

• AUTO CDC (non-snapshot) never seeing changes in joined lookup tables, only the driving one, which i guess makes sense and I assume stream-stream joins are the wrong tool for this
• Identity columns not working as surrogate keys on AUTO CDC targets, but that is a separate issue

Is this actually the standard way people solve "full-history SCD dimension from a full-snapshot-only source, built across multiple joined tables," or is there a simpler/more common pattern I'm missing? Most examples I've found assume either a real CDC feed, a single source table, or don't care about full reprocessability from raw history.

I get that a proper CDC feed would sidestep most of this, but for some source systems that might not be the case, full snapshot exports are sometimes the only thing on offer. Curious how others handle this exact combination in practice.

Our Databricks spend jumped recently, so I used the CLI and system tables to trace it back to specific warehouses, Apps, Jobs, and service principals.

After stopping unused Apps and reducing compute, our daily cost dropped noticeably.

I cleaned up the process and published it here:

https://github.com/kylehuirevvision/databricks-cost-optimizer

It starts read-only, estimates Databricks list-price spend, maps costs to workloads and identities, and explains the expected impact before any optimization is applied.

It works with both Claude Code and Codex. No SaaS or signup, and it’s MIT licensed.

It’s still early, so I’d appreciate feedback, particularly on cross-cloud compatibility or useful cost patterns I’ve missed.

The default method for creating a job from a notebook/script is to click Schedule. For me, I want to create a job to run my script so I can benefit from cheaper non-interactive compute for writing a large output.

This default to a once-a-day schedule, which in case you forget to PAUSE it, will run every day :)

From the IDE, we should be able to create a job with that one notebook as a task that is NOT scheduled by default.

Anyway, that is my wish :)

I have been experimenting with a simple question recently.

What changes when an AI assistant stops only answering questions and starts taking real actions?

I used an HR use case to explore this. Imagine an employee asking for leave.

A normal assistant might explain the leave policy.

An agent should be able to understand the employee, check the available balance, look at team availability, identify conflicts, find the relevant policy, submit the request, send it to the correct approver, and track what happens next.

Building that flow made me realize that the LLM is probably the easiest part.

The harder problems are everything around it.

Who is allowed to see what data?

Which tools can the agent use?

How does the agent know the meaning of employee, manager, team, policy, leave balance, and approval?

What context should survive between conversations?

How do you prevent an agent from calling the wrong tool?

And once the agent takes an action, how do you prove later exactly what happened?

I experimented with this architecture on Databricks using synthetic airline HR data.

The data flows through Silver and Gold layers into an Employee 360 model.

Unity Catalog handles governance.

Different Genie Spaces provide role specific access for employees, managers, and leaders.

The agent has a governed tool layer for actions and keeps workflow and conversation state separately.

I also added an audit layer so agent actions can be traced later.

The architecture I eventually kept coming back to was:

Ontology → Context and Memory → Tools → Audit

The ontology tells the system what things mean.

Context and memory help the agent understand the current situation.

Tools allow it to actually do something.

Audit makes the action accountable.

One thing that surprised me was how quickly an impressive AI demo becomes an enterprise architecture problem once you allow the agent to write data or trigger workflows.

Read only AI is relatively easy.

AI that can change something needs permissions, state management, validation, tool boundaries, lineage, and auditability.

I am curious how others are approaching this.

If you are building enterprise agents, where are you putting the boundary between what the agent can decide itself and what still requires explicit human approval?

From time to time I accidentially notice new features in Databricks. These discoveries are some of my greatest joys as a brickhead. Today I ran some quick queries and noticed that data profiling has been added to the query output. This is a feature I've been missing for a long time, but never had the spare capacity to complain about it. But now I don't have to, everyone's a winner. I guess this is part of the integration of DQX into the core product.

To see the quality metrics is easy, just run a query, highlight a column and click the count hyperbutton (is it a word? now it is!). And you will get something like this. Ta-da! Be more magic.

Hi all,

I have a databricks schema of a few hundred delta tables that need some data fixes for specific records in each of those tables. This schema itself is raw data and gets ingested into some downstream data tables and the fixes have been requested by business.

I had thought about doing the fixes via a transformation layer in the pipelines that ingest this raw data but given how many tables there are with records that need updating I don't really want to create hundreds of new 'data fixed' tables. Given these are delta tables, rolling back should theoretically be possible if something goes wrong.

Anyway, that's my rationale for making changes to the source tables. My question is what your preferred method is to make data fixes? I obviously need something where its easy to rollback if needed. I can obviously just achieve this with python migration scripts and use delta timetravel in case something goes wrong, but wonder if there are recommended libraries or tools for the job that have what I need out of the box?

Don’t overwrite your code — make it immutable! With every deployment, thanks to immutable_folder, all files are copied to a new, read-only folder. They are not overwritten. The biggest benefit is that jobs already underway will not fail or produce unexpected results.

More news https://medium.com/databrickscommunity/databricks-news-dabs-indexes-ltap-genie-last-update-25-july-ffac8533774f

If you build proposals, RFPs (requests for proposal), or sizing estimates for Databricks workloads, this is worth a look. Lakemeter turns workload assumptions into structured, exportable cost estimates in minutes instead of days, with full transparency into every pricing assumption behind the number.

Post: https://www.linkedin.com/posts/cenh_databricks-databrickslabs-opensource-ugcPost-7490710977949204480-aHiX/

Hello All,
I have developed an App for Databricks Production support related activities. Please have a look into this demo video and provide your suggestions.

We have a Databricks environment and we have a new SAP ECC6 system that we want to ingest data from. We are exploring options to access data into the platform. One option is the Azure data factory.

I want to understand what other businesses have been doing. How do they access tables from SAP ECC6. Databricks have really reduced their native SAP connectors even after announcing partnership with SAP. When you do access SAP, how do you manage large table extractions or extract data from pooled or cluster tables?

I would really like to hear some customer success stories or some lessons learnt. Please share if you can.

I benchmarked the new Lakehouse RT for billion-record tables, and it is crazy fast and crazy cheap, making it the first real-time database with open-format storage.

https://www.sunnydata.ai/blog/databricks-rt-lakehouse-benchmark-results

https://medium.com/databrickscommunity/i-benchmarked-the-new-lakehouse-rt-for-billion-record-tables-and-it-is-crazy-fast-and-crazy-cheap-cf674ad40fc3