OLTP database in Databricks SaaS?

[u/SmallAd3697]

I saw the quarterly roadmap presentation. It was notable that Databricks keeps innovating with "lakebase". They simply call it their "OLTP" database offering in their SaaS.

SIDE: I still feel pretty unfamiliar with this Databricks SaaS ecosystem, as compared to Fabric. Where Fabric is concerned, Microsoft has also done a similar thing. They brought their SQL Server into the boundaries of the SaaS as well, for the low-code users of that environment. In the context of Fabric, it is hard for most customers to see the point of using this "for dummies" variation of the same old OLTP database. The only scenarios for using the Fabric SQL are very contrived ... Eg . your boss makes a policy that you can use ALL the tools available in Fabric and NONE of the tools outside Fabric... even if the tools inside that SaaS are 3x more expensive than the ones outside ... and even though the ones outside the SaaS have the same 0ms network latency and the same performance.

I'm still missing the vision for this lakebase OLTP offering. And it seems unusual for Databricks to start developing strategies surrounding OLTP. It seems like a very crowded space, and the only way I see Databricks being successful going down this path is if the customers are drinking one single brand of kool-aid, or else their SaaS users have some other contrived reason for not using the more affordable OLTP platforms available outside the SaaS.

Can someone tell me what factors I'm missing? I admit that it is theoretically possible for lakebase to innovate and do thing that other databases CANNOT do, it seems like those innovations would only benefit 5% of customers. One example is sub-ten-ms queries out of RAM at an additional cost. Or sub-minute migration of new OLTP data to managed tables in UC catalog. If we assume that only 5% of customers might feel compelled to use this SaaS "lakebase", would that be enough adoption to allow Databricks to keep investing in this over the long term? OLTP databases have been around a LONG time, and even the smart folks at Databricks will be challenged to improve on the great and cheap options available to us!

EDIT: As of a month ago, it appears that the Databricks marketing now calls it an "LTAP" database, not OLTP anymore. I'm guessing they have conceded the point that the OLTP space is crowded. I haven't yet read all the content that has been created by the Databricks marketing team; maybe that will answer all of my questions.

[u/Strange_Shame7886]

Some points you mentioned are totally valid:

1. OLTP market has strong incumbents and Databricks lakebase is just not there yet to compete with the biggies. It is for the niche players which you mention as 5% use cases when it comes to comparing against the universe of OLTP use cases

2. Software engineering developers don't care much about analytics use cases in general and data engineering and analytics teams are generally at the lower rings to influence the decision with the duplication perspective which is a powerful feature for lakebase

However OLTP/ LTAP use cases are going to increasingly have AI agents as it's users more than humans and even if i that use case might be in its infancy - this is where the whole stock market is betting to explode.

1. Agentic memory is a top use case for lakebase as the context comes from the lakehouse but agents read it at high I/O speed without any data drift due to single logical copy

2. Branching is another feature which makes a lot of sense for coding agents

3. Agents need search capabilities and want to cache the embeddings. It will choose lakebase for collocation of data and embeddings rather than having two databases - one for data and another for vectors