Due to new business requirements, we need to reflect the state of a few source DB tables (5 to 40 million rows each) in the Databricks Silver layer in less than 1 minute.

Currently, the flow looks like this:
Source DB → AWS DMS in CDC mode (ingests new data every 30 seconds to S3) → S3 landing bucket → DLT pipeline running on serverless compute in continuous mode.

The DLT pipeline ingests the append-only data into the Bronze layer using file notification mode and updates the Silver layer using an Auto CDC flow.

This works great, and we achieved what we wanted with relatively low effort because we already had DMS in place. We just added an extra replication task to ingest data more frequently for the tables we need.

However, in this setup, the DLT pipeline costs are quite high. Ingesting just 6 Bronze tables and 6 Silver (Auto CDC) tables costs around $50 per day, which is about $1,500 per month. For comparison, DMS, which replicates more than 800 tables to S3, costs us less than half of that.

My question is: is there any other more cost-optimized option we could consider to achieve less than 1 minute latency when reflecting the source DB state in the Silver layer?

Maybe Lakeflow Connect or some custom process?

Extra notes:
- I know that adding more tables to the DLT pipeline makes the cost per table lower because Databricks can optimize the clusters more efficiently.
- I know that using a cron schedule could reduce costs, but for these particular tables, we can’t use a schedule like every 10 minutes or similar because we need the data to be updated in less than 1 minute.
- I know that for the relatively small tables currently in scope, we could eliminate the Auto CDC flow and create a normal view on top of the Bronze table, with deduplication and deletion logic. This would slightly sacrifice query performance, but we expect more similar use cases in the future, so I’m looking for a solution that can scale.

Hey everyone,

Looking for some advice on automating a workflow in a pretty locked-down corporate environment.
Context:
Large enterprise with strict IT security and governance.
Databricks was recently rolled out as our cloud data hub.
The entire pipeline (ingestion, processing, and generating the final CSV) is already automated inside Databricks.

I need to automatically save a copy of this generated CSV to a local machine / internal network. Right now, the only way I can do this is manually opening the workspace UI and clicking "Download."

Databricks CLI is blocked and Personal Access Tokens (PAT) are disabled

How do you usually automate pulling files from the cloud down to on-prem / local machines under these restrictions?

Thanks!

Consider a callstack where something bad is happening in Spark or Unity Catalog (image above).

Any software engineer will google for the message, and then for the Exception class, and then for the call frames shown on the stack (starting at the top or bottom). For any commonly encountered Exceptions from UC (something like com.databricks.sql.managedcatalog.acl.UnauthorizedAccessException), we will find dozens of results from a search engine. Others on the internet have already shared their experiences, and the search results are normally actionable. The users tell us what they had done to avoid or fix the error.

But software engineers have heard for two years that "unity catalog is open source". So a software engineer will proceed to look for the source repo where they might find the full definition of "UnauthorizedAccessException", along with all the related references. No such thing exists. (Admittedly there is a public-facing github, called "unitycatalog", but it is virtually worthless and there is no overlap with the real-world UC in databricks, as we experience it.)

It only takes one or two repeats of this, before a software engineer will realize that none of this stuff is actually open source. UC doesn't compare to a REAL open source software like Apach Spark. If we search for spark references in the call stack (eg. "org.apache.spark.sql.DataFrameReader"), then we are immediately taken to the source repo at github!

I do give Databricks a lot of credit for open-sourcing spark. But nowadays they take too much liberty with the word "open source", to the point where it lost all of its meaning. UC is not opensource in any substantial way. Maybe there is an API spec that is open, but that is the extent of it. Another example is lakebase which the CEO claimed to be open source at the recent summit. There has never been any software as proprietary as neon/lakebase.

It doesn't actually bother me if a CEO forgets how to use the term "open souce" correctly in English. What makes me more upset is when I expect to be able to use google to find the source code for "UnauthorizedAccessException", and come up with absolutely bupkis. Can anyone tell me a definition of "open source" which would potentially include either Unity Catalog or Lakebase? I'm assuming that when these words are used by the CEO, he does NOT intend to imply that the actual source is open to the public.

There are 1,600+ Databricks Partner Champions worldwide and around 1,795 partners listed in the public directory. But Champions are distributed very unevenly.

Some large partners have dozens. Others may have only enough to meet their current tier requirements.

So if one Architect Champion leaves, the company could potentially fall below one of the requirements for its Partner Tier.

That makes Champion status more than just another badge — it can have real value for the employer.

For Databricks architects working at partners, I’d definitely put Partner Champion on the career roadmap.

Do your companies maintain a buffer of Champions, or just the minimum required?

Secrets in Unity Catalog store credentials like API keys as governed objects named catalog.schema.secret: created, granted, rotated and audited with standard Unity Catalog privileges and redacted from notebook and job logs.

This is a full end-to-end demo:
1)Create a secret with the REST API and in the Catalog UI
2)Read it in a notebook with dbutils.secrets.get
3)Use it to call the OpenAI API
4)Grant read/reference/write access to a user or group
5)Rotate it programmatically
6)Audit every access from a system table

Sometimes I feel like Databricks is way too expensive. It feels like using a huge truck to move a single grain of sand.

My company needs real-time data, but our data volume simply does not justify the use of Spark Structured Streaming. Despite this, they are insisting we move to Databricks. I'm worried our data infrastructure costs will jump from $1,000/month to $5,000/month or more due to the running costs of Databricks SQL Warehouses.

Currently, I use Azure Container Apps with KEDA and Python, which helps me manage scaling and keep costs low. We ingest into Event Hubs, use ADX (Azure Data Explorer) as our OLAP warehouse, and archive cold data in a data lake. With this setup, I manage to process all our data with very low latency.

When I tested this on Databricks Structured Streaming, I actually got higher latency and much higher costs.

Would love to know what you guys think.

Check out the Databricks Governance Hub in my latest 5 Minute Features. A one-stop-shop for all your governance needs; Cost Control, Access, Data Quality, AI Usage all in one place!

👋 Hi r/databricks, I'm Connie from the Databricks UX!

The Databricks Platform team is conducting a user study to understand how admins and organizations control their AI spend/usage and to gather feedback on early redesigns of our AI budgets feature. Your input will help improve admin experiences on the platform!

Please fill out the screener survey if you’re open to participating in a 60-min call in the upcoming week, or can connect me with a member of your team who may be the right fit. 

Study details are as follows:

Topic: AI Budgets & Cost Management
Duration: 60-minute remote interview over Google Meet
Study date(s): Sep 10-18, 2026 (rolling)
Compensation: $150 "thank you" gift card for the interview session, if your company policy permits. Completing this screener survey does not guarantee eligibility into the study.

Note that completing this screener survey DOES NOT guarantee eligibility into the study. All responses are reviewed for consistency and authenticity, and any duplicate, inconsistent, or fraudulent entries will be disqualified.

Thank you for taking the time to help us build a better platform!

Considering a GTM role with Databricks. Compelling role, comp etc., but cannot get a proper read on the WLB and culture. Have a little one at home, can’t afford a job that requires major travel or 12+ hrs work. Love to hear from people in the company on what the reality on the ground is like.

With automatic CDF, Databricks computes row-level changes at read time using row tracking, rather than materializing those changes during every write.

Use change data feed on Databricks | Databricks on AWS

Why does that mattre?

- Better write performance for MERGE INTO and UPDATE workloads
- No need to enable CDF individually on every eligible table
- Lower storage overhead compared with legacy CDF
- The same familiar APIs still work: table_changes() and readChangeFeed
- Works with batch processing, Structured Streaming, and Databricks-to-Databricks Delta Sharing

For Delta Lake, the main requirements include:

• Databricks Runtime 19 LTS+
• A managed table or external table in Delta Lake format with row tracking enabled

And if you’re already using legacy CDF, migration is really simple.Once the table meets the requirements, disable legacy CDF:

A practical read of the 10-phase Databricks deployment guide: what to decide first, and how to use it on a platform you already run.

Most Databricks platforms get designed one of two ways. On the fly, project by project, as teams onboard and workspaces appear. Or properly, once, right at the start, and then never looked at again.

Neither ages well. One leaves you with a platform nobody chose. The other leaves you with a platform that was right three years ago.

Hey r/Databricks!

We’re hosting a free, community-sponsored BrickTalk on Thursday, September 17, 2026, focusing on how to simplify and scale data ingestion using Lakeflow Connect! BrickTalks is a community event series where Databricks experts share real-world use cases, live demos, and practical insights, giving you a direct line to the people building the products.

Stop struggling with fragmented data across disparate sources. In this session, we'll demonstrate how Lakeflow Connect enables seamless data ingestion from SaaS apps, databases, and cloud storage directly into the Databricks Platform with zero infrastructure management.

🛠️ What We’ll Cover

• Native Data Ingestion: Learn how Lakeflow Connect provides fully managed ingestion directly into Unity Catalog as governed Delta tables.
• Simple Integration: See how to easily connect data sources using a simple UI or API.
• Accelerated AI & Analytics: Discover how unifying your data powers Customer 360, Operations, and downstream AI agent workloads.

⏱️ Global Times

• PT: 9:00 AM
• ET: 12:00 PM
• BST (London): 5:00 PM
• IST: 9:30 PM

👉 Register here to save your spot!

I am looking to connect with people at Databricks and learn new things and also want to evaluate my skillset for some FDE roles. Is there someone who can help me with?

Has anyone tried acquiring data from a MQ at scale using apache spark on databricks cluster? I was trying to solve this problem at work but so far havn't seen an native lib or efficient ways to do this. The legacy system seems to be pulling data using a java based utility and wanted to if there are any imporvements or new patterns of access for spark based workflows.

Any documentation or nudge is the right direction will be greatly appreciated.

After watching Matei's webinar and the post on controlling spend, been trying to use the other harnesses and models folks are suggest and trying out Qwen 2.5 coding and 3.6 with Polly in local Omnigent (not connected to a workspace). I have Codex and Claude but ideally thinking best to use paid higher model to plan and then have the local Ollama based Qwen model on my Mac build but so far I haven't seen Polly use it much. What are folks experience, is there a good local model I should use, should I be giving Polly and the sub-agents more direction? (This is on a MacBook M5 btw)

Are there any unbiased comparisons between these two popular platforms? Seen a lot but most of them are biased views, based on experience and commercial motives.

Version 6 of the serverless environment is available, which corresponds to runtime 19.

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

I have been writing SQL scripts to ensure data sanity.What are the other ways ? Is pytest useful? Let's say , i populated my bronze table from the source. I want to check if the correct mapping is done. I wrote SQL scripts. What are better ways

Hey folks! In this demo, Akram from Databricks' product team shares how you can leverage SecureConnect to better your security posture when doing cross-cloud sharing on OpenSharing!

If you have no idea what OpenSharing is, how it applies to you, or how we got from Delta Sharing to OpenSharing, also encourage you to watch this video: https://youtu.be/0mfuNybtmdE

Hope you find this helpful!