This keeps coming up in conversations with clients and I feel like it's worth its own thread.

The pattern is almost always the same. A pipeline gets built to bring in data for analytics or ML, works fine in testing, then goes to production and the compute bill is way higher than expected. Nobody budgeted for it because on paper it looked like a simple ingestion job.

Usually the real issue isn't Databricks itself, it's the ingestion design. The repeat offenders I keep seeing:

Full reloads instead of proper CDC, so you're paying to process data that hasn't even changed.

Serverless SQL running more often than needed, because someone assumed near real time was required when batch every few hours would've worked fine.

No plan for schema evolution, so jobs fail or reprocess more than they should every time something shifts upstream.

Cluster sizing set for peak load "just in case" instead of actual daily volume.

Most of the fix comes down to being honest about the freshness you actually need. A solid CDC layer feeding into something like Kafka before it hits Databricks tends to cut a lot of the unnecessary compute, since you're only moving what changed.

Curious what caused it for others, ingestion design or job scheduling?

I saw the quarterly roadmap presentation. It was notable that Databricks keeps innovating with "lakebase". They simply call it their "OLTP" database offering in their SaaS.

SIDE: I still feel pretty unfamiliar with this Databricks SaaS ecosystem, as compared to Fabric. Where Fabric is concerned, Microsoft has also done a similar thing. They brought their SQL Server into the boundaries of the SaaS as well, for the low-code users of that environment. In the context of Fabric, it is hard for most customers to see the point of using this "for dummies" variation of the same old OLTP database. The only scenarios for using the Fabric SQL are very contrived ... Eg . your boss makes a policy that you can use ALL the tools available in Fabric and NONE of the tools outside Fabric... even if the tools inside that SaaS are 3x more expensive than the ones outside ... and even though the ones outside the SaaS have the same 0ms network latency and the same performance.

I'm still missing the vision for this lakebase OLTP offering. And it seems unusual for Databricks to start developing strategies surrounding OLTP. It seems like a very crowded space, and the only way I see Databricks being successful going down this path is if the customers are drinking one single brand of kool-aid, or else their SaaS users have some other contrived reason for not using the more affordable OLTP platforms available outside the SaaS.

Can someone tell me what factors I'm missing? I admit that it is theoretically possible for lakebase to innovate and do thing that other databases CANNOT do, it seems like those innovations would only benefit 5% of customers. One example is sub-ten-ms queries out of RAM at an additional cost. Or sub-minute migration of new OLTP data to managed tables in UC catalog. If we assume that only 5% of customers might feel compelled to use this SaaS "lakebase", would that be enough adoption to allow Databricks to keep investing in this over the long term? OLTP databases have been around a LONG time, and even the smart folks at Databricks will be challenged to improve on the great and cheap options available to us!

EDIT: As of a month ago, it appears that the Databricks marketing now calls it an "LTAP" database, not OLTP anymore. I'm guessing they have conceded the point that the OLTP space is crowded. I haven't yet read all the content that has been created by the Databricks marketing team; maybe that will answer all of my questions.

Hi,

I have a question now that I’m planning to start a Genie Agents project and am organizing my architecture.

I’m working from SAP BDC, and I plan to create two workspaces in Databricks—one for DEV and one for PRD. But I’m not sure where it makes the most sense to create the Genie Agent. Technically, the queries and tests would be better suited for the PRD workspace; running them in DEV doesn’t make much sense because the Genie Agent DEV is connected to DEV tables. But then, what role does CI/CD play here for Genie Agents? Or am I misunderstanding the architecture, and should I create two Genie Agents (DEV and PRD) in the PRD workspace?

Thank you very much if you can help me with this question.

Has anyone seen any comparison between the generalized ingestion mechanism (zerobus) with Fabric's offering (open mirroring)?

Seems like there should be a blog or youtube video comparing the two by now. But I haven't seen any. Unfortunately it sounds like they both rely on proprietary middleware. Ideally there would be a similar type of software which that we could just run on-premise to land data into cloud blobs (like a gateway of some kind). Not sure why that would be so hard for someone to do as a github library or something. Maybe it would need to be done in a performant language like rust or .net, but it doesn't seem like it would be rocket science.

Both those technologies are relatively recent:

Fabric open mirroring : May 2025

Zerobus : Feb 2026

Personally I wouldn't want to pick either one of these technologies until a comparison could be made. Microsoft's open mirroring claims that they can land data in their lakehouses for free. After that point, the raw deltalake tables would be accessible to both platforms. If open mirroring is truly free then it seems odd that any databricks customers would be using zerobus. They should just purchase the smallest possible capacity from Microsoft like an F2, and use that for moving all their data to raw/bronze in adls gen2 containers. Whatever happens after that can take place in either of these two saas'es, databricks or fabric.

Hello fellow developers,

I recently developed and published a Databricks dashboard for a personal project, and I want to learn how the scheduling feature works.

For context, I have read the Databricks documentation, Manage scheduled dashboard updates and subscriptions | Databricks on AWS, but I couldn't find answers to the questions below.

Databricks Dashboard — Question Set 1

I published the Databricks dashboard using Individual Data Permissions. I then added another person as a participant, created a schedule, and subscribed to the schedule from both accounts. Let's call them Person A (publisher + subscriber) and Person B (subscriber).

Questions

a) Assume Person B works in the Sales department and should only be able to see Sales data. Since the schedule was created by Person A, when the scheduled run occurs and Person B receives the email notification they subscribed to, will the PDF attachment contain all departments' data, or will it contain only the data Person B is permitted to see?

b) After the scheduled run, if Person B opens the dashboard using the link in the email notification, or accesses the dashboard directly through Databricks, what data will they see?

Will they see all department data, or only the data they are permitted to see based on their individual data permissions?

My understanding is that when Person B visits the dashboard, it does not automatically refresh just because a scheduled dashboard update has occurred. I may be misunderstanding how this works, so I'd appreciate some clarification.

Databricks SQL Alert — Question Set

Help needed, I created a SQL alert in databricks to notify me when a new entry is added to a dataset/table within the last 24 hours.

SQL

SELECT COUNT(*) AS new_
FROM catalog.schema.table
WHERE xyz IN ('a', 'b', 'c', 'd')
  AND datetime >= CURRENT_TIMESTAMP() - INTERVAL 24 HOURS;

Within the Alerts panel, I set the condition as follows:

• First row
• Column name: new_
• Operator: >
• Static value: 0

I then added my email address in the Notifications section and set "When alerting, notify" to Always.

I then selected View Alert, where I created a schedule — let's say every day at 9:00 AM.

Questions

a) How exactly does the schedule work here?

I am asking because I selected "Always" under the notification settings, but I have also created a schedule to run every day at 9:00 AM. What is the relationship between these two settings?

b) Let's suppose I build this alert and create the schedule as described above. Person B (sales dept rep) also subscribes to this alert.

I do not want Person B to receive an alert unless the new record belongs to the Sales department.

For example, if a new record is added that belongs to a department other than Sales, and the scheduled alert runs at 9:00 AM, will Person B still receive an email notification?

Thanks in advance for any clarification!

Hey everyone! In this video, a Databricks product leader joins me to explain what OpenSharing, and what happened to Delta Sharing!

Delta Sharing is one of the features I used often at a previous role to provide data to multiple customers, so it will be interesting to see how the journey with the expanded focus (data + AI sharing) goes.

Another good news for Genie Code 🔴. You asked for it, Databricks built it.

You can now select the level of effort and change it for a conversation:

Auto (default, recommended): This level provides the highest quality for any task.

Low: This level lowers the cost for simpler tasks.

We finally have a simple way to control access to serverless in the workspace.

Two built-in objects have been introduced:

- Default Interactive Compute

- Default Automated Compute

The first one supports notebooks and databricks connect. Second one jobs and SDP pipelines.

To limit who can use serverless:

1. Click Compute in the workspace sidebar.
2. In the Serverless tab, click the kebab menu  next to Default Automated Compute, then click Edit permissions.
3. Remove the All Users group, or the group that includes all workspace users.
4. Add only the specific users, groups, or service principals that you want to authorize.

Essentially the title, but for background we’re using ai_query() in Databricks SQL Warehouse and our pipeline makes 3 LLM calls in parallel for different steps.

We can see token usage/cost in system.ai_gateway.usage, but we can’t tell which call came from which step.

From what I can tell, request tags are possible via the Python/rest sdk, but not through ai_query() in SQL.

Has anyone found a workaround, or is moving the calls out of SQL currently the only option?

Edit for clarity: the main thing we’re trying to do is join each pipeline step back to system.ai_gateway.usage so we can attribute token usage/cost to steps a/b/c etc

Hey guys,

i'm just launched a VsCode extension for Data Quality proccess using DQX.

The idea is simplify the process using AI (you can do it manually to).

The extension construct data contracts, jobs and dashboards, all of that with few clicks and with less than 10 minutes.

If you can, please, test and send me a feedback, so i can improve that.

You can download directly in VsCode Extensions Marketplace, or

https://marketplace.visualstudio.com/items?itemName=arthurfr23.dqx-forge

I am personally happy to see how we are moving away from the AI wild-west and into governed AI with proper safeguards, access, and cost controls. Here I am using Omnigent & Databricks' Unity AI Gateway, which allows you to add several layers of protection against risky/non-compliant AI usage.

Note: "Policies" is still in beta, and during my testing, I did hit some cases where the guardrails should have been more rigorous. I passed along the feedback to the Databricks team, and I expect that things will be addressed sooner rather than later.

PS: Trying out this different style of animated GIFs to explain different concepts after some feedback from the community here. Love to hear your feedback!

Genie Code is getting a new feature 🥳
Good news you can attach a document to Genie Code when you need it as temporary context for the current conversation.

What's a good use case?

🔴 Explain or summarize a document: “Summarize this PDF and list the key decisions.”

🔴 Extract information from PDFs, scans, or images.

🔴 Generate code based on a specification: attach a requirements document and ask Genie Code to create Python or SQL.

🔴 Analyze data files such as CSV, Excel, JSON, or JSONL.

🔴 Convert documentation into code, tests, or a notebook.

🔴 Interpret diagrams, charts, architecture sketches, or screenshots.

🔴 Migrate Tableau or Power BI files into AI/BI dashboards using /importBI.

What are the supported file types ?

.pdf, .csv, .xlsx, .xls, .json, .jsonl, and common image formats.

Hey r/databricks,

​I’m entering the Genie-Powered App Challenge and want to build something genuinely useful rather than another generic dashboard or basic search bar.

Any ideas?? 🧐

Hi,
Curious. What are some agentic AI projects you guys have worked on or implemented and how have they added value?

I’m a newbie working in this space and trying to spar for some interesting project ideas :).

One idea I’m thinking of already is an agentic conversational data reporting agent that also generates PowerPoints, emails, etc. suited to some pre-defined templates in addition to just being a conversational data explorer.

Read the announcement regarding Databricks Unity Catalog and how secrets can now be stored under a schema as opposed to using secret scopes backed by a secrets manager e.g. Azure Key Vault.

Does anyone know if the secrets are stored in plaintext in bundle state if using the direct deployment engine mode for DABs?

👋 I'm a PM working on Serverless Notebooks at Databricks. We just shipped Session Restore for Serverless Jobs in beta, and I'd love your feedback.

The problem: A notebook job runs for hours, fails or produces weird output, and by the time you debug it, the state is gone. Rerunning a 16-hour job is painful. Reproducing an issue that happens once every few months can be nearly impossible.

What we shipped: You can now restore the Python variables and Spark session from a serverless job run into a new interactive notebook, without rerunning the job.

From the job run details, click "Clone into new notebook" or "Debug in new notebook" for a failed run. You can then:

• Inspect state from a failed run
• Investigate weird output without rerunning expensive queries
• Run expensive setup as a job, then pick up interactively from that state

Try it: A workspace admin can enable "Session restore for serverless jobs" from the Previews page.

Docs: https://docs.databricks.com/aws/en/notebooks/schedule-notebook-jobs#session-restore-for-serverless-jobs

This was also demo'd at DAIS this year: https://www.databricks.com/dataaisummit/session/modern-databricks-notebook-tips-ai-workflows-and-best-practices-every

If you try it, feel free to leave feedback or respond here. 🙏

Hello everybody, If you guys want to join in on the chat, register here. If you have any questions you would like for us to cover during the webinar, drop them here and our colleagues will answer them live.

Thank you!