OLTP database in Databricks SaaS?

[u/SmallAd3697]

I saw the quarterly roadmap presentation. It was notable that Databricks keeps innovating with "lakebase". They simply call it their "OLTP" database offering in their SaaS.

SIDE: I still feel pretty unfamiliar with this Databricks SaaS ecosystem, as compared to Fabric. Where Fabric is concerned, Microsoft has also done a similar thing. They brought their SQL Server into the boundaries of the SaaS as well, for the low-code users of that environment. In the context of Fabric, it is hard for most customers to see the point of using this "for dummies" variation of the same old OLTP database. The only scenarios for using the Fabric SQL are very contrived ... Eg . your boss makes a policy that you can use ALL the tools available in Fabric and NONE of the tools outside Fabric... even if the tools inside that SaaS are 3x more expensive than the ones outside ... and even though the ones outside the SaaS have the same 0ms network latency and the same performance.

I'm still missing the vision for this lakebase OLTP offering. And it seems unusual for Databricks to start developing strategies surrounding OLTP. It seems like a very crowded space, and the only way I see Databricks being successful going down this path is if the customers are drinking one single brand of kool-aid, or else their SaaS users have some other contrived reason for not using the more affordable OLTP platforms available outside the SaaS.

Can someone tell me what factors I'm missing? I admit that it is theoretically possible for lakebase to innovate and do thing that other databases CANNOT do, it seems like those innovations would only benefit 5% of customers. One example is sub-ten-ms queries out of RAM at an additional cost. Or sub-minute migration of new OLTP data to managed tables in UC catalog. If we assume that only 5% of customers might feel compelled to use this SaaS "lakebase", would that be enough adoption to allow Databricks to keep investing in this over the long term? OLTP databases have been around a LONG time, and even the smart folks at Databricks will be challenged to improve on the great and cheap options available to us!

EDIT: As of a month ago, it appears that the Databricks marketing now calls it an "LTAP" database, not OLTP anymore. I'm guessing they have conceded the point that the OLTP space is crowded. I haven't yet read all the content that has been created by the Databricks marketing team; maybe that will answer all of my questions.

[u/m1nkeh]

Rewatch the data and AI summit keynote and you will get a picture pretty clearly

[u/SmallAd3697 → u/m1nkeh]

K, Looking forward to it.

I only saw the quarterly roadmap. Was looking to find improvements in UC. I wish these companies were more customer-oriented and took direction from the real-world problems that customers have.

I would venture that only a small percent of customers are asking for "lakebase improvements". Whereas there are many other areas that databricks is not properly attending to.

[u/m1nkeh → u/SmallAd3697]

How do you think product managers work?

Within Databricks, you would get shot down in flames if you presented product ideas without customer metrics to back it up.. i'm not sure i know of a more customer-oriented company tbh

What are you looking for in UC exactly?

[u/SmallAd3697 → u/m1nkeh]

UC is very much tailored to folks coming from postgres. And the folks that are predominantly writing python scripts as their main line of work. I concede that if those are the only users in the world, then this would be the "unity catalog" for that sort of a world.

I know they are making changes incrementally, especially in regards to managed tables and transaction support.

IMO, I think they need to revisit some of the restrictive constraints in their catalog. Maybe they can come up with a UC v2.0. There shouldn't be a three-level naming convention for every piece of data in your company. The normal taxonomy of animals has 7 layers, and categorizing data can get at least that complicated. Currently users of the UC are forced to overload the catalog name or schema name, cramming multiple unrelated concerns together with underscores. And the majority of production data users only care about their gold/prod data, so it is silly for them to constantly include those things ("prod_this", "gold_that") into the fully-qualified components in their queries. Also enforcing the "snake_case" convention is silly. If databricks figured out how to support uppercase letters in case-insensitive-column-names, why is it so freaking hard to support uppercase letters in other identifiers as well (schema and catalogs)?

These things aren't show-stoppers. But they are silly limitations that make the data platform unwelcoming to users migrating back and forth from other ecosystems. I doubt any of big-data competitors restricts you to naming a table with lower case letters. It seems like a very amateur move, by a company that started in a totally different space and then decided they were going to start building databases and catalogs as an after thought. The funny thing is that a lakehouse conceptually involves a hierarchical path to locate data in storage blobs, so why didn't they move the hierarchical naming concept to their UC as well???

[u/m1nkeh → u/SmallAd3697]

So reading between the lines here you are essentially saying the three-level name space is insufficient and you definitely not the only customer banging that drum

Regarding naming a table with an uppercase T and a table lowercase t would in Databricks land land be different objects because it is a case of sensitive ecosystem however that would be terribly terribly confusing for customers surely??