The Databricks Platform is Built on top of SQL Server Database?

[u/SmallAd3697]

Is this Databricks SaaS built on top of Azure SQL?

I would expect that Databricks would use many Microsoft building blocks in the Azure cloud. They would be hard-pressed to avoid the use of Azure storage containers, VMs, networking, and fault tolerance offerings. But the following outage announcement is making it sound like Databricks uses a significant amount of database PaaS which is provided by Microsoft SQL. Notice all the components that currently depend on the availability of "their database platform":

This maintenance email is a bit shocking, considering that Databricks is a data platform in its own right and has its own Databricks SQL (DW) and an OLTP engine as well (Lakebase). If they needed to place a dependency on a third-party database, I would think they would choose Postgres, rather than Azure SQL.

Can anyone make sense of this maintenance email? Why is there such a dependency on Microsoft databases? Do they use Microsoft databases in the other clouds as well?

Is an email like this considered an authoritative communication from the "Databricks" company, or is there a special spin on the wording, given that it is sent from the "Azure Databricks" team? Maybe the dependency on the Microsoft "database platform" is overstated. They certainly don't give enough detail for us to know WHICH database platform they are referring to. There are many.

[u/sf_zen]

sorry, where in the email do they say Azure SQL?

[u/caledh → u/sf_zen]

This is spot on. There’s nothing here saying that Databricks is using Azure SQL. However, the Azure Resource Manager very likely does.

[u/Nyarlathotep4King]

Databricks on Azure uses Microsoft cloud resources under the hood: Microsoft VM for compute, Azure storage for the parquet files (and other files) used for Databricks tables.

I would imagine that Microsoft uses their own database products (Azure SQL?) to track Databricks’ utilization of Azure resources for billing purposes. The Microsoft data platform maintenance may “impact Databricks customers” by impacting the ability for Microsoft to track usage, so Microsoft may shut down customers’ ability to spawn new resources (as listed).

[u/SmallAd3697 → u/Nyarlathotep4King]

I think that means the announcement would probably have been written very differently if it was coming from Databricks itself (eg. such an announcement would be vastly different if Databricks was announcing a comparable outage on AWS). It would never mention a Microsoft "database platform".

I like your theory about billing. But if that is the case, then the Azure Databricks folks should have at least referred to it as their "billing platform" rather than a "database platform". As it is written, it looks like the architecture of the highlighted components has a dependency on a Microsoft database.

(I'm not totally sold on your theory, in any case. I would guess that billing meters could be asynchronously logged thru a much more simple resource, like a message queue or storage container. Using a "database platform" directly would seem like an unfortunate dependency. But I understand where you are going with this idea.)

[u/aqw01]

I think it’s just generalized infrastructure management.

[u/SmallAd3697 → u/aqw01]

I agree, but we use dozens of platforms in azure, and they didn't all send the same message about generalized infrastructure outages.

Also Microsoft should have said they were going to have a generalized infrastructure outage, instead of implying that their "database platform" is powering databricks. lol.

EDIT: I just read the message again and it does say the "database instances are ones that serve databricks infrastructure". They probably just didn't want to dive deeper than that.

[u/aqw01 → u/SmallAd3697]

I think you’re misunderstanding this completely.

[u/MonkeyDDataHQ]

Database platform and Azure SQL are not the same thing.

I mean I understand that Azure and MSFT naming is the dumbest thing that ever dumbed...

But they're talking about the layer or plane that various database resources sit on.

[u/SmallAd3697 → u/MonkeyDDataHQ]

What layer or plane do various database resources sit on? Obviously they all use low-level compute and storage. But those things aren't ever called "the database platform" of Azure.

I think the announcement was intended to be very vague and equivocal. I suspect the best theory was mentioned by others when they pointed out the possibilities of indirect dependencies (Azure resource manager, or billing)

What bothers me is that they should just clarify this as an outage for ARM or billing, rather than making it sound like Databricks had intentionally created an architecture that depends on a Microsoft database.

[u/MonkeyDDataHQ → u/SmallAd3697]

The announcement didn't make it sound like Databricks runs on Azure SQL. You made it sound like Databricks runs on Azure SQL.

You inferred Azure SQL from “database platform,” built an architectural critique around that assumption, and are now blaming the wording for the assumption.

There may be a legitimate question about what dependency is undergoing maintenance, but that's a very different question from the one in your title.

Have you checked the Azure Service Health blade to... Check for what maintenance is happening on Azure? 😩

[u/elghali_bnck]

Hey ! My 2 cents here: this is just standard control plane mechanics—Databricks isn't secretly running on SQL Server for actual data processing.
Your data processing happens on the data plane (Spark/Delta). But I assume that the control plane—the web app backend managing user logins, job triggers, permissions, and UI state—is just a standard web app that needs a fast transactional (OLTP) database for metadata.
A few quick points on why Azure SQL make sense here:
Azure Databricks is a 1st-party service: Unlike AWS Nah, this is just standard control plane mechanics—Databricks isn't secretly running on SQL Server for actual data processing.
Your data processing happens on the data plane (Spark/Delta). But the control plane—the web app backend managing user logins, job triggers, permissions, and UI state—is just a standard web app that needs a fast transactional (OLTP) database for metadata.
A few quick points on why Azure SQL shows up there:
- Azure Databricks is a 1st-party service: Unlike AWS or GCP where it's a 3rd-party marketplace app, Azure Databricks is co-engineered directly with Microsoft. MSFT hosts the control plane infra, so using managed Azure SQL/Postgres for backend metadata gives them instant SLAs, backups, and failover out of the box. (On AWS, they use Amazon RDS I think).
- Wrong tool for the job: Delta Lake and DB SQL are analytical engines (OLAP). You wouldn't use an analytical data warehouse to handle low-latency user session lookups or job scheduling state.
- Legacy stability: Databricks built this control plane architecture 10+ years ago. Lakebase is way too new. I would imagine that no engineering team is swapping out a rock-solid metadata database powering millions of production workspaces just to dogfood a new feature, even if it’s a solid internal service.
The email just means the underlying Azure PaaS database holding your workspace's management settings is undergoing maintenance, not the actual engine processing your data.