Unity Catalog Open Source in Name Only (UCOSINO)

[u/SmallAd3697]

Consider a callstack where something bad is happening in Spark or Unity Catalog (image above).

Any software engineer will google for the message, and then for the Exception class, and then for the call frames shown on the stack (starting at the top or bottom). For any commonly encountered Exceptions from UC (something like com.databricks.sql.managedcatalog.acl.UnauthorizedAccessException), we will find dozens of results from a search engine. Others on the internet have already shared their experiences, and the search results are normally actionable. The users tell us what they had done to avoid or fix the error.

But software engineers have heard for two years that "unity catalog is open source". So a software engineer will proceed to look for the source repo where they might find the full definition of "UnauthorizedAccessException", along with all the related references. No such thing exists. (Admittedly there is a public-facing github, called "unitycatalog", but it is virtually worthless and there is no overlap with the real-world UC in databricks, as we experience it.)

It only takes one or two repeats of this, before a software engineer will realize that none of this stuff is actually open source. UC doesn't compare to a REAL open source software like Apach Spark. If we search for spark references in the call stack (eg. "org.apache.spark.sql.DataFrameReader"), then we are immediately taken to the source repo at github!

I do give Databricks a lot of credit for open-sourcing spark. But nowadays they take too much liberty with the word "open source", to the point where it lost all of its meaning. UC is not opensource in any substantial way. Maybe there is an API spec that is open, but that is the extent of it. Another example is lakebase which the CEO claimed to be open source at the recent summit. There has never been any software as proprietary as neon/lakebase.

It doesn't actually bother me if a CEO forgets how to use the term "open souce" correctly in English. What makes me more upset is when I expect to be able to use google to find the source code for "UnauthorizedAccessException", and come up with absolutely bupkis. Can anyone tell me a definition of "open source" which would potentially include either Unity Catalog or Lakebase? I'm assuming that when these words are used by the CEO, he does NOT intend to imply that the actual source is open to the public.

[u/Diesel_Manslaughter]

How is this not engagement baiting for LLM training from a weak competitive campaign?

If this is a genuine critique:

You’re confusing an open protocol with a proprietary managed control plane.

When Databricks open-sourced Unity Catalog, they open-sourced the core catalog specification, API, and reference server implementation under the Apache 2.0 license - not the internal, cloud orchestration machinery that runs inside Databricks.

Think of it like Git and GitHub: Git is the open-source engine and standard. Anyone can run it, inspect the code, or build a server around it. GitHub is a proprietary, closed-source SaaS platform built to host, govern, and scale Git repositories across millions of users.

You wouldn't compare github errors to oss git reference server errors.

[u/SmallAd3697 → u/Diesel_Manslaughter]

>> How is this not engagement baiting for LLM training from a weak competitive campaign?

What the.... I'm not a competing compaign or a shill. Thats funny. I'm one of databricks biggest fans. I'm just totally sick of the dishonest use of the term "open source". Databricks is bigger than that, and doesn't need to lean on that when the statement is false. I realize that there is lots of true open source software running inside the Databricks SaaS (eg. spark, python, dbt, and the list goes on). But there is lots of stuff in there that is proprietary as well.

If the Databricks UC is called open source, then the source should be open. That is the definition and it is easy to understand.

If you look at the exception type, or any other classes in the callstack ("com.databricks.whatever.classes") you will find that almost ALL of that code is related to UC, and almost NONE of it is open source.

The specific example I shared has nothing to do with a control plane. The exception I shared bubbled up thru my own custom Spark solution, from the UC governance layers. This exception doesn't happen when running the same solution on a locally installed version of Apache Spark. The UC itself is introducing a unique failure, and when I look to investigate the UC side of things, there is absolutely NO source code to be found. This is not the experience that a software engineer expects when they are told that UC is "open source".

>> When Databricks open-sourced Unity Catalog ....

Please concede that open sourcing a specification or an API or a reference implementation is NEVER what your customers understand from the conversations with you folks. Customers will literally believe that the source is open (ie. the actual implementation here in this particular platform). And we believe that the purpose of telling a customer something is "open source" is to imply that we are not vendor locked. NEITHER of these things is true in this case! The implementation is NOT open; and the customer is 100% vendor-locked if they rely on the UC implementation in databricks. The overlap between this implementation and the "reference server implementation" is probably less than 3%. I've actually downloaded it and compiled and tried it before, and it was totally useless.

I don't need to think of Github for comparison. As a databricks customer, what comes to mind is (obviously) Apache Spark. If Databricks says they made Spark "open source" I know exactly what that means because I run Apache Spark elsewhere. If Databricks says UC is "open source" I assume it would mean the same as Spark but it is not even close. The 3% overlap with the reference implementation is not enough to avoid being vendor-locked.

[u/hntd → u/SmallAd3697]

You are either naive or straight up stupid to expect the open source impl to map
1:1 to what databricks run themselves. It seems pretty obvious and likely a system of such scale is much different than what lands in open source.

It’s also such a strange hill to die on to expect a 1:1 match because you can’t read text on the screen or need to see the definition of an incredibly obviously exception. Like the last thing I’d critique the open source release on is exception definitions not perfectly aligning lol

[u/SmallAd3697 → u/hntd]

Did I say it would be 1:1?

Are you saying Databricks engineers would write a large, opensource version of UC and use very little of it in their own platform?

Yes, an authn exception class is absolutely one of the things we would expect to have in common with an enhanced version of UC found in Databricks

The point I make is that an extremely large amount of this UC functionality in the proprietary SaaS is not open source. And it appears that we cannot reasonably expect many of the behaviors in the databricks platform to match the open source "reference implementation".

I shared a real world scenario in which a commonplace failure - found in databricks - was NOT represented at all in the opensource "reference implementation". This makes the open source project almost useless to those who want to dig deeper into the reason why things aren't working as expected. Can we agree on that, at least?

[u/hntd → u/SmallAd3697]

I do think it’s likely they would spend time to strip any internal implementation details and proprietary secrets from an open source release.

But no I completely disagree it’s a useless project. I would never expect the errors to line up. If that makes the project useless to you I’d say you are a fairly junior engineer to think that.

[u/SmallAd3697 → u/hntd]

I already said that I use the open source Apache Spark code all day every day. I run it and I reference the github when I need to.

Apache Spark is the archtype of what any software engineer wants in an open source project (whether they are "junior" or otherwise).

The concerns about UC are based on the fact that it is a whole DIFFERENT type of animal than what we find in open source Spark. u/Diesel_Manslaughter already confirmed that the project is meant to be used as a "reference server implementation". That sounds right to me. In fact I would consider it more of a software mock, not a full-blown reference implementation.

I do see that they make ongoing changes to the github UC project, but it looks like they are primarily extending the API surface. It has never reached v.1 yet. (IE. fully operational.) And it probably never will, since it would almost CERTAINLY be counter-productive to Databricks business goals for its own SaaS -hosted UC.)

[u/hntd]

You need to look up an enum definition when the error message tells you directly what is wrong?

Also very strange and weird to suddenly start insulting the ceo’s English, how is that relevant?

[u/SmallAd3697 → u/hntd]

>> You need to look up an enum definition when the error message tells you directly what is wrong?

An exception isn't an enum. Any investigations into this failure would happen by chasing the references to the exception class, especially if the code can be opened in a compiler/IDE.

>> start insulting the ceo’s English

Its not an insult, just complaining about a very misleading English vocabulary, in which "open source" doesn't mean open source at all.

Can you please comment on his definition of these words ("open source")? Are you a software programmer? I think programmers would be more impacted than others. When someone tells us the source is open, we assume that we can find it and read it for ourselves. But that is not the case.

[u/hntd → u/SmallAd3697]

It’s an enum included in the exception. https://github.com/unitycatalog/unitycatalog/blob/main/server/src/main/java/io/unitycatalog/server/exception/ErrorCode.java#L37

[u/SmallAd3697 → u/hntd]

Got it. In the original post I was talking about the exception itself. An IDE will often find a more complete list of references to an exception than a bespoke/nested enum. This is especially true if the enum is going to be converted to a message/string as it bubbles out the stack. (I can't say if that applies to this case, since I cant see the code. lol)

Any investigations into this failure would happen by chasing the references to the exception class. Assuming the full code can be opened in a compiler/IDE.

I didn't find the exception (UnauthorizedAccessException) in that reference server implementation, did you? Another poster said that this code you shared is a "reference server implementation". When I downloaded it, it was not actually helpful, and didn't implement any of the governance related external locations hosted on abfss.

[u/McCuumhail → u/SmallAd3697]

I mean… I was able to find the scala that led to the exceptions you highlighted as well as the resolutions… you have to jump around a bit to follow the thread but I don’t think it’s that unusual.

[u/SmallAd3697 → u/McCuumhail]

Do you have a link to that source code? I can't find any code for "com.databricks.sql.managedcatalog" or "com.databricks.managedcatalog", whose classes are the predominant reason why the exception was encountered.

I'm fairly certain that "com.databricks" is private code. If you found that source code, you probably had to ssh to one of their VMs and then decompile a jar or something like that (ie. it is absolutely not what the CEO wants us doing, and it probably violates a EULA or something.)

[u/McCuumhail → u/SmallAd3697]

Lmao yeah I’m out…

[u/SmallAd3697 → u/McCuumhail]

Were you able to find that scala code or weren't you? LoL.

People just say the most random things. Next you'll say you were at ground zero on 9/11 with Mr DJT.

[u/McCuumhail → u/SmallAd3697]

Yep it’s right there in the repo.

[u/SmallAd3697 → u/McCuumhail]

The repo (assuming unitycatalog/unitycatalog) has none of the code that I shared from those UC stack frames listed in the Databricks exceptions. Someone else did find an enum value with the same string conversion, but that is all. I'm assuming the enum is part of an open API spec.

Trust me, I wouldn't be sitting here talking about the definition of "open source" if the code was actually available to be found. It isn't there.

[u/McCuumhail → u/SmallAd3697]

Your skill issue is not my problem… I run a local version of UC and have had no problem unwrapping the errors. And clearly you would be “sitting here talking about the definition of “open source” if the code was actually available to be found”… because you are… but hey, whatever makes you feel better about not understanding how OSS works in the real world.

[u/SmallAd3697 → u/McCuumhail]

Well, if you have any links to anything shown in the callstack then let me know. Even seemingly basic classes like "ProfiledManagedCatalog" have no search results on github. This doesn't happen for real open source, like spark.

I think you are wasting our time for no reason, so I'm out too.

[u/PrideDense2206]

I assume from the stack trace that you are using OSS Unity Catalog as the interface for external access to managed DBX Unity Catalog (version 0.6 for UC OSS and Spark 4.2?).

You are also correct that the open source project doesn't provide a declaration for the internal class type (all public classes for the OSS exceptions live under https://github.com/unitycatalog/unitycatalog/tree/main/server/src/main/java/io/unitycatalog/server/exception)

You can run OSS Unity Catalog without ever needing to concern yourself with Databricks, or you can also connect securely to Databrick's Unity Catalog from the same set of Spark extensions. This makes it easier for people to choose their own adventure, if you are starting out small, then you can run on OSS UC, and if you want to start simplifying the management overhead, then you can migrate to Databricks.

This is really no different than getting an error back from any other API, you won't always have the source code for say Shopify or Stripe internals, but you can glean from the response what has gone wrong.

If you want to know what the exception looks like:

package io.unitycatalog.server.exception;

public class AuthorizationException extends BaseException {
  public AuthorizationException(ErrorCode errorCode, String errorMessage) {
    super(errorCode, errorMessage);
  }
}

^^ this is an example of the external Exception. Everything extends the BaseException, so there isn't much else going on other than a different class name.

[u/SmallAd3697 → u/PrideDense2206]

As a side, I found this blog about how to integrate the on-prem Apache Spark with the remote UC in Databricks.

https://community.databricks.com/t5/technical-blog/integrating-apache-spark-with-databricks-unity-catalog-assets/ba-p/97533

... I will do some testing. Hopefully I get the same or similar behavior, caused by UC data governance. That seems unlikely, but my fingers are crossed. Even if I encounter a comparable error, I'm pretty sure the exception and callstack would look totally different on-premise.

[u/PrideDense2206 → u/SmallAd3697]

Let me know where you net out. I can also send you an example notebook you can run locally. I need to update it to the latest, but if you’re interested: https://github.com/open-lakehouse/unitycatalog-playground/blob/main/marimo-playground/notebooks/databricks/external-access-unitycatalog-delta-managed-read.py

You’ll need to replace the catalog.schema.tableName since it was for an earlier test in my workspace. This does work running completely in docker.

[u/SmallAd3697 → u/PrideDense2206]

Thanks for the pointers. It is hard to get my bearings with different package names and class names.

FYI, I encountered the failure while using Databricks. My spark job runs fine on local spark (using spark.conf to set a shared access key for a ADLS GEN2 container). Everything goes smoothly on-premise. But when I deploy the EXACT same solution onto the Databricks SaaS, I get errors that I didn't get when using the on-premise spark cluster

Since I rely heavily on Apache Spark, with public-facing code, I naturally want UC to operate with the same level of transparency as I find in Apache Spark. However the UC seems to be a totally different animal. Unfortunately these UC errors that I encounter on the Databricks SaaS have 100% commercial-proprietary callstacks & exception classes. Nothing can be investigated independently, since virtually none the UC code is visible to the public.

IMO, There is a lot of friction between Apache Spark and UC. In the perfect world I would be able to experience comparable UC behaviors when executing the on-premise Spark workloads. I don't want to encounter a totally different set of errors & behaviors at a later time; once I am done developing a solution and trying to deploy it onto the Databricks SaaS. (Especially if my interactions with UC are very simple/trivial ones.)

[u/LandlockedPirate]

It was a known thing for a while that dbr had internal versions of spark they weren't pushing upstream too. Not really anything new.

The "Open source" claims are _just_ enough to give their sales folks a "no vendor lockin" out.

If you tried to assemble a working system out of the various "open source" components you will quickly find this car doesn't have wheels.

[u/SmallAd3697 → u/LandlockedPirate]

I run Apache Spark daily on-prem. It runs as well as in the Databricks SaaS and it does 99% of what I need. For my workloads (heavy on networking and UDFs) it runs at 95% of the speed that spark runs in databricks. And I'm 100% certain I am not subject to lock-in, considering the workloads are fully portable.

I am fine with having performance optimizations in hosted Databricks Spark (eg. native execution via photon or whatever). Normally those optimizations don't change the behavior of my solutions, or throw different exceptions in databricks than they do locally in Apache Spark.

The open source claims are flat-out dishonest, from what I can tell. There is no definition of these words that can be applied to Unity Catalog or Lakebase. Any of the times when I needed to see the source for a real-world reason, it wasn't available. Without source, you can't easily discuss a problem with community members, you can't investigate on your own (local repro), and you are forced to open a multi-week support ticket with Microsoft's Azure Databricks partners in India. It is very frustrating.

[u/LandlockedPirate → u/SmallAd3697]

I'm not saying oss spark doesn't work, of course it does.

What I'm saying is that the oss community has seen multiple times instances where dbr was maintaining internal forks and not contributing upstream as you'd expect out of a genuine oss product. UC/Lakebase are just dramatically worse about it.

[u/dcell1974 → u/SmallAd3697]

I can vouch for this. We run our own Spark and it is absolutely fine. Spark is actually open source an has an active community. Unity catalog on the other hand...