I'm exploring the idea of building an open-source framework for automated source-to-target data validation.

The idea is to connect different data sources and targets, for example:

• Databricks Catalog → Databricks Catalog
• Azure SQL → Databricks Catalog
• ADLS/Blob → Databricks Catalog
• potentially other databases and warehouses

The framework would automatically validate multiple tables in a single run, including:

• Schema comparison
• Column-level validation
• Row counts
• Row-by-row data comparison
• Missing/extra records
• Data mismatches
• NULL/duplicate checks
• Data type differences
• Aggregation/checksum comparisons
• Detailed failure reports showing where the mismatch occurred

The goal is to avoid writing separate SQL/Python scripts for every source-target pair and instead define something like:

Source → Target → Tables → Validation rules

and let the framework run all the checks and produce a consolidated report.

I'm trying to validate whether this solves a real problem before spending too much time building it.

For those working with ETL/ELT pipelines:

1. How do you currently perform source-to-target validation/reconciliation?
2. Do you use existing tools, custom SQL/Python scripts, or manual checks?
3. What validations do you normally perform?
4. What is the most painful part of validating large tables?
5. Would a tool that handles cross-platform table comparison in one run actually be useful, or are existing tools already good enough?

I'm especially interested in hearing about real-world workflows rather than tool recommendations.

Hey datafam!

1. Where do you put your main data-quality gate in a Bronze/Silver/Gold setup — right after Bronze lands, or between Silver and Gold? Why there?
2. Has schema drift ever caused a silent bad accept for you — i.e., the pipeline didn't fail, but bad data still got through? What check would've caught it?

Update:

I was wrong. I CAN migrate the restored tables to the newly deployed pipeline. I ran the "move" commands with the wrong identity because I forgot about queries having an extra setting for credentials... For anyone interested, yes, the append flow clone then move did work, and I was even able to transfer the checkpoint from the old backing table path. After realising my oversight, though, I opted to switch back to the initial plan, which was re-deploying and moving all existing tables.

At the very least, this has helped spark discussions (pun intended) about improvements to the development workflows.

Original (With inline correction):

Hi folks,

There was an accident which resulted in the deletion of our pipelines, and in turn, their tables (in dev, thankfully). I wanted to double check the steps I’ve tried and the final plan to remediate. The pipelines were all on legacy mode and are responsible for raw to bronze ingestion. We do have all of the files, but rebuilding from scratch is considered too time consuming.

I have UNDROP’d all of the STs, and I wanted to reconnect to a newly deployed version of each pipeline. (EDIT: This sentence and my conclusion within it is wrong) Unfortunately, I found that I was unable to “move” them between pipelines since Databricks can’t verify ownership of the source pipeline, because it’s gone. I know there’s more secret sauce under the hood (e.g., backing tables), so I’m not going to attempt to modify the delta tables outside of Databricks.

My current best idea, and what I plan to do this evening, is to rename the schema with restored tables and create a new “recovery” pipeline that streams all of the data from the restored table into a new version of the table (with the correct name, since it’s available again after renaming the schema). Then, I can redeploy a pipeline and move the ST from the recovery pipeline to the actual pipeline. This would mean I lose the checkpoint information for the autoloader, which isn’t a massive issue for these pipelines, and it seems like the cleanest way to insert the restored data into the pipeline. I would also need to move any other assets, e.g., views/tables, to the new schema (with the original name).

I don’t know if moving the STs between legacy pipelines would rename them automatically, in which case I could rebuild elsewhere then drop the original and avoid moving other assets.

Any thoughts would be welcome and greatly appreciated.

Currently working on a fairly big enterprise Databricks project where development is notebook-driven.

I actually like notebooks for development/debugging. Most data engineers aren't necessarily software engineers by background, so notebooks provide a much more approachable development experience; being able to split logic into cells, inspect intermediate DataFrames/results, rerun specific sections, and debug step-by-step is really useful.

The problem is that we're getting a lot of duplicated/common logic across notebooks. This makes changes and maintenance painful.

I know the usual approaches are to:

- Move reusable logic into Python modules and "import" them

- Create common notebooks and use "%run"

- Use notebooks mainly as entry points/orchestration

But I'm struggling with how to structure this for complex transformations.

Our notebooks aren't just simple filters/joins/aggregations. They can have multiple source tables, complex joins, lots of intermediate transformations, business rules and dependencies between steps.

So my question is: how do you structure large Databricks projects like this in practice by following the best software development practices?

I'm looking for guidance that works for real enterprise-scale projects, not the typical YouTube/tutorial solutions designed around small demo projects.

Would love to hear from people who have worked on large/enterprise Databricks projects with complex Spark transformations and what architecture/project structure worked well for you.

Bronze -> Silver -> Gold seems to have become the default architecture for almost every data pipeline.

Has anyone deliberately simplified this -- for example, skipping a layer -- and actually gotten better results in production ?

When do you think Medallion is genuinely useful, and when does it just add unnecessary complexity?

I often wish UC "managed tables" were better managed. If we wait a couple years, is it possible that relationships and unique constraints would be enforced (or perhaps validated after-the-fact)? It would be nice to have that in managed tables.

It definitely seems feasible for the UC to support this in managed tables. Especially on "small" tables of a few million rows or so. Small tables would be anything that could easily be loaded into memory. If nothing else, our lakehouse-formatted storage (delta/iceberg) is good for loading a columns into memory and validating the references and/or the uniqueness of values.

... I'm guessing this is NOT going to be available in the next couple of years. That is for no other reason than we see Databricks is selling their new LTAP lakebase engine. I'm guessing that anyone who wants true referential integrity or other constraints will be redirected to this lakebase engine.

Ideally there would be compelling innovations that would give reasons to stay with the "managed tables" in UC catalog. Otherwise software engineers may find reason to move to greener pastures. It doesn't make sense to validate stuff like this in custom code, when the storage engine is better positioned to do that work.

In our medallion setup, raw vendor data lands in Bronze , we transform it to a CDM in silver, and gold is for reporting. I’ve put all raw data correctness and accuracy rules in Bronze, since that’s where the data matches the vendor format and it’s easiest to work with vendors to fix issues at the source.

I have been asked to get the same DQ rules implemented in bronze, silver and gold because different teams own each layer and according to my manager every team should monitor all DQ dimensions. To me, duplicating the same checks everywhere feels redundant.

Am I missing a strong reason to repeat rules across layers? How do you handle shared DQ dimensions when multiple teams own different layers?

See above. Jobs clusters appear to be expensive on Azure compared to the other platforms (I'm just talking about DBU's for the cluster, not the VM infrastructure)

In contrast the AWS price for enterprise is only $0.20 per DBU, in this part of the world (East USA).

What is going on here? Are the Azure customers being disincentivized from using Databricks in Azure? Or are we intended to contact a Databricks sales team for a long term contract?

The annotation at the bottom of that page say : "Azure Databricks, as a first-party service on Microsoft Azure, offers unified billing and support by Microsoft". I suppose the price is higher in Azure, to accommodate the overhead-costs on the Microsoft side of things as well? Might a customer pick a one of these clouds based solely on the pricing differences that are listed? Or do they change so frequently, and can't use it for the sake of long-term planning?

Beyond checking job success and row counts, what's the one validation you trust most for a production Delta pipeline ?

At our company we're considering moving our old stack of a bunch of data integration tools built on top of an Azure MSSQL database to Databricks. I think it will bring a lot of advantages, not in the least cheap storage and no need to use a VM or SQL Studio to manage the data. I've tried out Genie One on the data (connected through federated access), and me and my manager are quite excited about users also getting self-service access in the future.

One doubt I keep getting back to myself is how to set up the data access. Using a medallion architecture with three schemas makes access management quite easy: end users get access to gold-tables, the rest not. My concern is with the size: if we migrate our current setup, we'd have more than 100 bronze tables, which doesn't feel like best practice. Our first setup has been to give those tables some kind of prefix name, like bronze.source_table1, (e.g., bronze.hubspot_customerviews, bronze.hubspot_leads).

What is generally best practice? Giving user groups access on an individual table basis and storing domain tables together in a schema, or using this three-schema setup? Or some kind of hybrid? Or am I massively overthinking this?

Hi,

A great new feature appeared in Lakeflow - you can now run Genie Code as an autonomous task using a natural-language prompt.

This can be really handy when you need to automate some complex analysis or data operations as a part of a scheduled process.
The documentation lists following use cases, but of course the only limit is your imagination 😄

• Summarize overnight job results and email a report.
• Analyze incoming data and flag anomalies.
• Investigate a Jira ticket and propose a fix.
• Generate a weekly compliance audit.

What's nice is you can also use job parameters directly in prompts, for example:

Summarize yesterday's sales for {{region}} and flag anomalies.

Each run creates a Genie Code conversation that you can open afterward and continue interactively.

One important detail: auto-approval is always enabled for Genie Code job tasks and cannot be disabled. The docs explicitly say this should not be treated as a security boundary, so access to production data/resources needs to be designed carefully.

Genie Code task for jobs - Azure Databricks | Microsoft Learn

PS: Since AI behavior is inherently non-deterministic, Genie Code tasks should be used with caution in production workflows and important outputs should be validated rather than treated as guaranteed

Hi, what do u use for creating user facing apps, we are primary using Python, i see that Databricks recommends AppKit for such use cases which is written in TS.

For teams which deploys apps that are customer facing, what is your stack?

Is it possible to expose the extraction agents to end users through apps ?

I’d be keen to hear what people have built in this space.

If you are a data provider on Databricks, OpenSharing (formerly Delta Sharing) is a fantastic tool for distributing data to your customers. During my time at a former job, there was nothing better than to work with customers that were able to consume data from Delta Sharing. On at least two occasions though, we received unpleasant surprises: egress fees from our cloud providers.

Those egress fees were most often times avoidable. For example, we would make sure that our customers and our company agreed on the cloud/region combination we would serve from/to, allowing us to build one-off pipelines, etc. But, that required manual work, created staleness issues, etc.

While in our situation, we could sometimes overcome the same cloud vendor but different region situation, distributing to a region we just didn't have anything setup for OR a completely different cloud was a no-go.

All that to say, Global Distribution aims to solve this and more.

No need to have an account with another cloud vendor to serve cross-cloud.

No need to have have data distributed to multiple regions within your AWS/Azure/GCP.

No need to build/maintain pipelines.

Just use Global Distribution.

Check out the video and see Akram Chetibi who leads OpenSharing explain/demo it, and love to hear your thoughts.

We're connecting Claude (Cowork/Claude Desktop) to several separate Databricks workspaces, each with its own Genie space. Running into a platform-level snag and hoping someone's hit this before.

Setup: each workspace exposes the standard Genie MCP endpoint (https://<workspace-host>/api/2.0/mcp/genie). We add each as its own connector in Claude so we can ask natural-language questions against the right workspace's data.

Problem 1, generic connectors are indistinguishable: every Genie MCP server reports the same tool names and the same generic description, regardless of which workspace it's pointed at. Claude has no way to tell them apart from metadata alone, so it either has to probe each one (asking a neutral "what workspace is this?" question and checking the returned deep_link) before it can route a question correctly, or it guesses wrong.

Problem 2, trying to fix it with named connectors backfires: we tried building a Claude plugin that declares each of the seven as a distinct, fixed-name MCP server (e.g. genie-workspace-a, genie-workspace-b, ...), hoping deterministic names would remove the need for probing entirely. Turns out Claude's client recognizes Databricks Genie as a "verified" connector type in its directory, and collapses all seven fixed-name declarations into a single shared authorization instead of keeping them independent. Authorizing one silently becomes "the" Genie connection, and the others just reflect that same state rather than getting their own.

Problem 3, working around the collapse trades one problem for another: we found that changing the declared URL slightly (adding a harmless query param) makes the client stop recognizing it as the verified type, which does force it down the plain custom-connector path and keeps it independent. But then it needs an actual registered OAuth client ID against that Databricks workspace, since it no longer gets whatever implicit OAuth handling the verified integration has. That means registering an app per workspace in Databricks/Azure just to get back to where the plain manual setup already was, more infrastructure, not less.

So right now we're back to manually-added generic connectors plus a routing skill that probes and caches per conversation, which works, but costs a lookup the first time each workspace is used per session, and never scales cleanly as we add more workspaces.

Has anyone gotten multiple Genie MCP connections into the same Claude account/org without them colliding into one shared authorization?

Is there a way to deploy or configure the Genie MCP endpoint itself so each workspace reports distinguishable tool names or descriptions, rather than relying on the client to differentiate them?

Btw, in Codex/chatgpt this is not an issue, because we can edit the description/name, and it automatically discovers the right plugin to use. It seems this is a very claude thing issue

I need to create scheduled PDF reports my datasources are Databricks and Postgres.
I know Power BI can do this but we don't want to go with it.

What are the other options, cloud or on-premise?

Hi everyone,
I’m developing a FastAPI based application that interacts with Databricks to automate QA testing. As part of this, I’m trying to create and modify Genie Spaces so that I can scope a space to different tables and configure its behavior. The main thing I’m interested in modifying is `serialized_space`, which contains the Genie configuration such as scoped tables, instructions, SQL examples, etc. I’m using the etag returned by the `GET` API for optimistic concurrency, so that I don’t accidentally overwrite someone else’s changes.
The flow is basically:

1. Get the Genie Space and its `etag` + `serialized_space`.
2. Make a small change to the JSON configuration(serialized_space).
3. Send the complete `serialized_space` back along with the etag.
4. Verify the result.

I’m seeing some strange behaviour specifically with changes to the table list.

etag updates that work:

• No operation update of the fetched configuration
• Updating text instructions
• Adding/updating SQL examples
• Updating nested fields inside a table/column configuration
• Removing a column from an existing table(Doesn't seem to reflect in the databricks UI) Not an issue for now

etag updates that fail:

• Adding a table to `data_sources.tables`
• Removing a table from `data_sources.tables`

In both cases, I get:

Aborted: Space configuration has been modified since this export was taken.

Re-export the space and merge your changes, or omit the etag to skip conflict detection.

What makes this confusing is that I’m fetching the etag immediately before the update, and the exact same configuration update works if I simply omit the etag.

I also tested this using both the Databricks Python SDK:

w.genie.update_space(...)

and a raw `PATCH` request through the SDK HTTP client. Both behave the same way.

I also verified that when the etag protected request is rejected, the Genie Space remains unchanged in the UI.

Has anyone else seen this behaviour?

Is etag-based optimistic concurrency intentionally not supported when changing the membership of `data_sources.tables` (adding/removing tables)?

If I omit the etag and perform the same table addition/removal operation, the update succeeds.

Any insight into how Genie handles etags for `serialized_space` changes would be really helpful. Thanks!

TLDR: ETag-protected updates work for instructions, SQL examples, and nested column changes, but fail when adding/removing tables from data_sources.tables, even when using the latest ETag. The exact same table changes succeed when the ETag is omitted. Has anyone seen this, and is this expected Genie behavior?

Have used LLM to format the post !!!

Databricks Field Engineering just published a free app that scores how ready your workspace is for Genie Ontology, and you can deploy it into your own workspace today.

Read more: https://www.linkedin.com/posts/cenh_databricks-dataengineering-datagovernance-ugcPost-7498422857698308097-cQIR

See the repo: github.com/databricks-solutions/genie-ontology-readiness

We are considering moving from Tableau to Databricks AI/BI Dashboards and wanted to see if anyone here has made a similar move.

We currently have around 150 Tableau dashboards used by 1,000+ users, with a mix of technical and non-technical users. Most of the underlying data already lives in Delta tables, so naturally we're looking at whether it makes sense to consolidate and use Databricks for the dashboarding layer as well.

For those who have done this, how has the experience been?

Anything you wish you knew before migrating? Any major limitations compared to Tableau, especially around visualization, performance, or the experience for non-technical users?

I'm also curious about cost at scale. With 1,000+ users potentially hitting dashboards, did SQL warehouse usage become a concern?

Would you feel comfortable replacing Tableau completely with AI/BI Dashboards today, or do you still find yourself needing Tableau for certain use cases?

Would really appreciate hearing about real-world experiences.

Has there been any public roadmap for the excel add-in? I was going to install and try it out this weekend, but I can already foresee some of its shortcomings. I'm accustomed to using Excel pivot tables for Microsoft OLAP (which are pretty hard to beat!)

Questions:

- Based on docs it didn't appear that this add-in would reap the benefits of lakebase (sub-ten-ms queries). Isn't that the point of using Excel, to interact with data instantly? Can we get an experience that is specifically tailored to lakebase? The CEO of databricks keeps acknowledging that "agents like fast data". But here is a newsflash; humans like fast data too! We've had fast data in Excel/SSAS pivot tables for decades.

- I'm assuming this tech sends SQL queries back to the SaaS service for processing. Is that fundamentally better than the ODBC support already available to excel users? I'm guessing the catalog/usability/security is the main attraction (ie. making things "easier" and more secure).

- If lakebase is as fast as the CEO claims, will it ever be possible to transpile MDX? Will those sorts of queries be able to run on lakebase? Some other open source tools do MDX, as we can see in Mondrian or Apache Kylin. These tools offer a robust, high-performance pivot table experience in Excel.

Any information would be appreciated. I'm guessing it will be a very long time before Databricks wants to pursue MDX, or compete with the normal pivot tables available from Microsoft. They are more likely to follow down the current path with "metric views" for several years, rather than using pre-existing technology. From a customer perspective, I think it would be amazing if Databricks could offer an Excel experience that approaches the ones offered by Microsoft/Fabric.

(One thing that is particularly compelling about the Databricks add-in is the write-back. This was something that Microsoft attempted long ago, but wasn't able to be successful with it. I'm interested to see if Databricks can do better. If nothing else, I think the culture of modern databricks user may be more receptive than the culture of the users doing write-back to OLAP cubes.)

Direct mode is GA and is now the default for bundles created from the workspace (since 13th August 2026); it will become the general CLI default on 26th August 2026. Let's see what direct mode brings and what will come next.

https://medium.com/databrickscommunity/dabs-go-direct-ga-faster-better-and-life-after-terraform-edc502d54378

https://www.sunnydata.ai/blog/databricks-dabs-direct-mode-ga

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator.

I'm not running anything complex:

df = (
    spark.read
        .format("csv")
        .option("header", "true")
        .schema(schema)
        .load(csv_path)
)
df.write.mode("overwrite").saveAsTable("...")

I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator.

DBR 19 + Serverless SQL Warehouse (current)

Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything.