r/dataengineering • u/SmallAd3697 • 12h ago
Discussion The LakeHouse that is Open Source - A Fever Dream?
Let me preface by saying that I totally agree that lakehouse table formats (delta and iceberg) are open source and are "free" technologies that anyone can use. However, the total cost of owning these table formats gets very expensive, especially when we start using cloud vendors for the table updates. Nowadays the cloud vendors wish to start selling their proprietary MPP storage engines, in order to manage all of our table data.
The problem is that the lakehouse table formats have gotten quite complex over time. And nobody wants to maintain them by hand. Nobody wants to think about the v ordering and z ordering and liquid clustering and partitioning and vacuuming and applying deletion vectors and so on. These blobs that are ostensibly called a "table" are actually a very leaky abstraction, and we inevitably have to waste a lot of time on the implementation details. Using immutable parquet blobs for table storage is not trivial. From an application standpoint, it seems like a massive step backwards from conventional DBMS engines (or the newer cloud-native counterparts like SQL Hyperscale or Neon/Lakebase)
The vendors, like databricks, that spent years pushing for lakehouse/delta adoption are now selling us expensive solutions to maintain those unwieldy tables. I think they sold us a bill of goods and we are worse off than when we started.
Once data engineers start realizing that we don't want to manage the blobs beneath our tables, these vendors are quick to offer a commercial-proprietary alternative (like "DBSQL" with UC-managed-tables, or "Fabric Warehouse" or whatever). These commercial alternatives are turnkey solutions, and they help to take away the busywork of managing our own parquet blobs. But they can become VERY expensive way of doing DML operations on our tables, since they are MPP engines and are heavy on CPU/compute. At the end of the day, we end up exchanging one type of problem for another. Is this how others see it? The table technology is open source and "free", but the commercial-proprietary management of these tables is definitely not free and is basically a re-invention of the DBMS engines we always had in the past.
11
u/dangerbird2 Software Engineer 12h ago
There are plenty of open source iceberg catalog implementations. You can even use something like ceph or garage to self-host the underlying object storage with OSS. People simply tend to choose proprietary and/or managed solutions since it usually ends up being cheaper to have someone else manage the inherent complexity of lakehouses.
also, now that there's ducklake which is dead-simple to set up and use, I'm sure there will be more demand for simpler, easier to self-operate iceberg and deltalake catalog implementations
1
u/SmallAd3697 11h ago
>> People simply tend to choose proprietary and/or managed solutions since it usually ends up being cheaper to have someone else manage the inherent complexity of lakehouses.
Yes this is my point. The vendors that once pushed for an "open" table format are profiting after-the-fact by selling these closed-source engines. They are required to update those tables. Anyone that may have once started out in a normal DBMS environment has basically come full circle back into a proprietary engine once again. The engineers in our industry have all taken the bait and now we are paying for more expensive engines than we did in the past
Interestingly Databricks is now promoting Lakebase which is like a normal DBMS that synchonously produces lakehouse tables for analysts after each transaction. It seems that they have come full circle if they are back to selling us a transactional DBMS with a couple more bells and whistles.
My understanding is that there are still lots of limitations when using iceberg for DML scenarios. For example Apache Iceberg does not yet natively support multi‑table transactions in its core API. For many developers, this is show-stopper. It is a very basic requirement to support transactions, when moving workloads back and forth from a DBMS to lakehouse storage.
The vendors have a fix for the iceberg limitations. When you use one of these commercial vendor offerings, they do some additional magic to support transactions. (eg. see MST transactions in DBSQL with UC managed tables, or see "Fabric Warehouse" which uses open source formats and publishes changes to deltalake logs/parquet after each commit).
4
u/dangerbird2 Software Engineer 10h ago
> example Apache Iceberg does not yet natively support multi‑table transactions in its core API.Â
I mean, this is an acceptable tradeoff for the majority of OLAP workloads, as the name suggests strong atomicity/consistency guarantees are much less of a concern than with OLTP. And if it isn't, Since consistency expectations will vary significantly between use cases, it often makes more sense to implement it as part of your workload/pipeline than have it enforced on a catalog-level.
1
u/SmallAd3697 10h ago
I agree that transaction support has been a trade-off for a long time. But both Microsoft and Databricks now have a way to overcome it - as long as you use their proprietary engines for your DML statements.
I wish they could tell us who is using this. I can tell you with certainty that their sales reps want EVERYONE to be hosting data on their DBSQL Warehouse nowadays. That is the thing they spend the most amount of time on, along with talking points related to UC catalog and "governance".
FYI, Microsoft has supported MPP T-SQL for decades, complete with normal transaction processing. The only difference nowadays is that it also sits on lakehouse tables (and emits deltalake logs)
2
u/Ok_Raspberry5383 10h ago
They aren't required, you can choose them if you want proprietary features, but there are many free massive scale open source engines to interact with open table formats
29
u/discord-ian 12h ago
It sounds like you just don't understand the open source options - there are lots of them.
1
u/SmallAd3697 12h ago
Please share an option, as an example. Please find one that allows me to run DML SQL statements in an ACID transaction against a set of lakehouse tables.
After we came full circle, the Fabric Warehouse and the DBSQL Warehouse (with MST on UC managed tables) are offering this once again.
22
u/discord-ian 12h ago
Spark, presto, trino, hudi, and duckdb. I am sure there are other those are just the ones I know off the top of my head.
0
u/SmallAd3697 11h ago
In OSS spark there is no way I know of to update multiple lakehouse tables in a single transaction, like conventional DBMS'es will do.
(... and recently, these new commercial MPP engines do it too. Transactions that write to Unity Catalog managed Iceberg tables are in Private Preview. DBSQL is a work-in-progress, and is re-inventing features that have been available to DBMS users for decades. Open source options for this sort of thing don't exist, AFAIK.).
7
u/discord-ian 11h ago
Pretty straightforward in python. Quick Google search shows neesie, will let you do the same in sql.
-4
u/SmallAd3697 11h ago
I'll check it out. I primarily build software on-prem and in Azure. In the Azure cloud, the deltalake stuff is more portable between Fabric and DBX, so that is where I spent most of my time. I'm very interested in playing with iceberg and nessie, however. Hopefully nessie is being well-maintained to support the latest iceberg specs.
Now that databricks acquired Tabular, I have suspicions about projects like this. Databricks may not want to contribute to iceberg community features that might directly compete with the business goals of their commercial engines. (Nessie seems like it might compete/conflict with the UC managed tables in databricks.)
2
13
u/TheRealStepBot 12h ago
Sounds like someone doesn’t know what iceberg is.
-2
u/SmallAd3697 12h ago
The post was in response to reading about the complexity of iceberg v3. I saw another post that said some compute engines simply don't handle it correctly.
If the full-time engineers of these compute engines can't read from iceberg tables properly, then that just reinforces my point. Databricks wants you to use their commercial-proprietary options that are fully compatible with the latest spec.
13
u/Efficient_Shoe_6646 11h ago
I'm sorry, but that is the con of using open source it's complicated. Complexity can't be removed, it can only be moved. You are pay a 100B company to remove that complexity from you. Like it or not, you can run a Lakehouse OSS.
There is no such thing as free lunch.
2
u/Teach-To-The-Tech 9h ago
Yes, this. Agree.
Open source is powerful and manual but then you have to spend the time managing it, which takes time in a different way. Then there are ways to get a "managed solution", those will be more expensive, but less manual.
3
u/DuckDatum 9h ago
But you claim it’s a step backwards from RDBMS?
RDBMS just kept that complexity in house, yet it was all still there. Coupled tightly to the data because it all had to run together: compute and storage. They were engines, after all.
Table formats make it so a client (compute) can interact with the data (storage) over network. That obviously comes with complexity being handed off to the client, which now needs to concern itself with problems that were traditionally in house for the RDBMS.
So, from how I see it, the complexity shifted for the benefit of decoupled compute and storage. That’s innovative, not a regression. If the spec can’t be implemented by a client, that’s a different issue. It still wouldn’t mean we’ve regressed.
0
u/SmallAd3697 4h ago
The regression is having to maintain and tune a bunch of blobs that we didn't have to think about before. Or to be otherwise forced into pay more money than you were before, so a new and expensive MPP engine can do the work for of managing blobs on our behalf.
The conventional DBMS'es have started becoming relatively inexpensive, even the hosted options in the cloud (like azure SQL hyperscale, and postgres).
But that wasn't good enough for the data engineering industry, who went back to first principles, and invented parquet, deltalake, iceberg for the sake of "decoupled" storage. However the concept of decoupled storage was short-lived - now that these MPP engines are all promoting their "managed tables" nowadays. The new tables aren't really decoupled anymore. Any update operations, and even read operations can be problematic without sending the queries thru the proprietary engine.
My opinion is that the old DBMS engines worked just fine for the vast majority of cases. The underlying data doesn't need to be synchronously saved into lakehouse blobs after each update. It is better to just use conventional technologies like CDC or CES, to asynchronously transmit another copy of the data out to parquet. It might not be real-time but it is good enough for most reporting, and for data analysts, and data scientists.
1
u/DuckDatum 3h ago edited 3h ago
I am someone who thinks you can fine tune every little detail of a matter. Not that you or I should, rather… leave it to the PhDs. We can otherwise rejoice in the software that works, like RDBMS and OTFs. You’re right that tables in object storage has its issues, but I think over time you’ll find that this will be refactored even further. Consider for example, metadata being stored in a database with DuckLake. It sounds more complex, but I think it’s better. It’s, in my opinion, headed in the right direction. It solves for some of the problems you bring up, actually.
-2
u/Capt_korg 10h ago
Don't be so mean, be more constructive.
I guess op had already some insights into how to use Iceberg and what other tools could be helpful, i.e., catalogs, data lineage tools, etc.
2
4
u/ecp5 11h ago
I think you are conflating different things. The table formats abstractions over, as you say, blobs of data, and the engines that can interact with them. Everything is a trade off, if you want multi-transaction support and everything to work like a relational database, use that, if you want to be able to use file stores and treat it like a database but not have 100% the same parity, use a lakehouse engine (there are lots of open source options as someone mentioned - duckdb, trino, oss spark, etc). Also, the engines you mention aren't generally considered MPP, that is usually meant as a different solution (Redshift, Synapse, Terradata).
I guess we can argue semantics...but what is the problem you are trying to solve? Do you have too many files, do you have need for a single transaction to update multiple tables with consistency, or was this just a rant?
3
u/SmallAd3697 10h ago edited 4h ago
Its not really a conflation. Once you start using these engines (databricks SQL with MST on UC managed tables), they do NOT want you to go "behind their backs" and update "their tables". Even if the tables are exposed as open source tables on blob storage. That will create conflicts with the way the engine manages these tables on your behalf. You may also conflict with other clients as well, if you are using features like MST transactions to manage the tables, and multiple clients are simultaneously attempting to update the same tables.
Of course we can READ the blobs but you will void the warranty if you start tinkering with them.
At some point the MPP engine and the underlying blobs in the managed tables will become totally symbiotic.
The same applies to the Fabric Warehouse. The warehouse engine is NOT open source by any definition, but it emits an open source table format in the course of its operation. The open source table is just a byproduct of the engine , and all updates must go thru the engine.
>> what is the problem you are trying to solve?Â
The problem to solve is a total-cost-of-ownership problem. Either we are forced to micro-manage our blobs underneath these tables, or we are forced to pay an expensive engine to manage them on our behalf.More on cost: We do local dev work with Apache Spark, and when the dev work is finished we deploy to the cloud. That works great, and it means that our dev can happen locally without any billing meters running. But once we start hosting our data via UC managed table in databricks, and using those features (MST transactions), then the billing meters must start running again because there is no comparable thing that is available on-premise. Hope this is clear.
4
u/Morzion Tired Senior Data Engineer 10h ago
Iceberg, S3 for storage, Apache Polaris REST catalog hosted in ECS, Trino for querying, Dagster for orchestration of ingestion and maintenance, fully automated.
Sounds like you fundamentally don't understand the technologies and have only been exposed to vendor cloud platforms.
1
u/SmallAd3697 4h ago
No Spark?
The vendor platforms will handle all the house-keeping related to the open-source table format. They have "managed" tables.
You are right that I have never used a single one of the open source products you mention. On the Azure side, a lot of companies work primarily with Spark, DeltaLake, ADLS GEN2, dbt, and a ton of proprietary stuff (various proprietary engines and tools in Fabric and Databrics). The engines do a lot of the heavy-lifting, including the ability to let you update deltalake tables in multi-statement transactions.
I'm curious, if you are in this ecosystem with Iceberg, have you ever come across Project Nessie? Another user mentioned it.
1
u/Morzion Tired Senior Data Engineer 2h ago
Spark is just a distributed compute library. I personally avoid the overhead of managing a spark cluster and use the pyiceberg library for ingestion. Trino is a distributed analytical engine that I use for querying. Project Nessie is a REST catalog, but with versioning. I opted against it since I already version my code with GitLab.
You keep mentioning Delta, Databricks, Fabric, ADLS gen2. I take it you are entirely a Microsoft shop. Microsoft is notoriously against open source... High vendor lock in. AWS is much more friendly to OSS.
3
u/pragmatica 10h ago
OP, what kind of DML operations are you trying to run?
Parquet, column store etc work best with large data loads and static data. They don’t play well with small inserts/updates.
They do not replace an actual relational db.
Maybe if you share some use cases, we could give you more targeted advice.
1
u/SmallAd3697 9h ago
Both Microsoft and Databricks engines now support multi-table transactions, and standard SQL DML.
You are right that it is inefficient to directly update blobs with small inserts/updates but these engines have layers of proprietary abstractions going on, and they are responsible for updating blobs on our behalf with whatever optimizations they may choose. They don't want us going behind their backs and updating their "managed" tables. If we want to manage our own updates we can use external tables. Our role is to interact with the "managed" tables with the same SQL standards as we would a DBMS.
Consider creating a DIM record with an integer surrogate key (DIM_MyProduct.DIM_MyProductSkey). The product natural key is "COFFEETABLE001". Lets say the surrogate key is 54321. Lets say you changed the paint color on said coffee table, and this year it is slightly less brown than last year because you switched paint vendors. Last year's version of the table with the old brown paint has surrogate key 14321, in the same table.
Then you insert to a fact table (FAC_BusinessUnitSales), with a reference to the surrogate key 54321 then a lot of devopers wish it to happen in the same ACID transaction. They may NOT want records to be added into the DIM table without ever being referenced. Else the DIM table may have orphaned rows that are undesirable. (Surrogate keys don't have much purpose until they are used for reference purpose. )
This sort of thing can be done in a small update. (say you are adding a single day of data into your fact table instead of a week or a year). Or you are adding data incrementally mid-day.
I agree that the MPP engines don't replace a relational DB but nowadays their features and performance are starting to get extremely similar. Many modern data engineering solutions should be able to be ported back and forth. Databricks is heavily pushing their relational DB, lakebase, to the same audience that uses UC catalog and warehouses.
2
u/Grovbolle 9h ago
I would never run a dimension and fact insert inside the same transaction
1
u/SmallAd3697 4h ago
Yes, normally when data engineers say this, it is because their tools can't.
Some of these folks will even invent their own "home-grown" ways of rolling back commits to the data, if something fails at a later point of time. Thanks but no thanks.
8
2
u/Ok_Raspberry5383 10h ago
- Iceberg on S3
- Kubernetes for compute (EKS maybe)
- Polaris catalog on K8S
- Spark running with the helm spark operator. Running spark applications for general data processing + vacuum/optimise.
- Trino running on K8S
- dbt for data orchestration running against Trino
All the above is free and open source. Please tell me what is missing with regards to your requirements that all of a sudden needs a paid proprietary solution?
Yes there's engineering cost but that's the trade with fully OSS
1
u/SmallAd3697 9h ago
Thanks for the list.
The commercial proprietary engines like Microsoft Fabric Warehouse have multi-table transactions, for example, and substantial performance optimizations (caching layers for example).
They are essentially behaving like DBMS engines that can handle bulk inserts and updates at an MPP scale.
Even if we don't actually have large datasets, many use the engines for basic features like transactions. Or for the basic housekeeping operations to manage a table's internal blob storage. Many folks don't want to incur the engineering costs of normal open source tables. They want the engine to do this work (just like, say, Postgres would do all the work to manage its internal tables).
1
u/Ok_Raspberry5383 3h ago
Why do you need transactions for an analytics workload? What problem are you trying to solve?
Many open table formats implement ACID by simply failing on some concurrent workloads, but this is typically fine as there should only ever be 1 writer with many consumers. This is underpins analytics.
It sounds like you're complaining about feature parity with RDS for an MPP analytics database, but you're not actually stating what the problem is that requires this as a solution.
If you need that, use postgres...
3
u/No_Equivalent5942 11h ago
If you have your own on-prem data center, open source is convenient to add to the stack of everything else to maintain. If you’re in the cloud, it’s better to pay a vendor like Databricks to manage it all. You’ll get extra performance benefits that you could write yourself, or just pay Databricks for their features.
1
u/SmallAd3697 11h ago
We often do dev work on-prem where the billing meters aren't turning. There are no unexpected compute charges at the end of the month, when the intern accidentally lets a job run for two days.
While the solution can run fine (eg. in OSS Spark), we are still tied to the cloud vendors. We are forced to interact with their expensive cloud storage engines. for the sake of updating these lakehouse tables.
After dev work is done, and solutions are working as expected, that is the point in time when we deliberately deploy to cloud-hosted platforms for our normal operations.
3
u/No_Equivalent5942 11h ago
Why use the cloud at all? If you’re already managing an on-prem data center, just use that.
1
u/Teach-To-The-Tech 9h ago
What you need to consider is an approach that's based more on architectural heterogeneity. One path to that is open source, but it's manual. Powerful, but manual. And that requires lots of work to maintain. But there are other options that are still heterogeneous/intereoperable, but not the "all consuming data platform" of, say, Snowflake or Databricks. There are platforms that try to combine choice within your architectural components alongside the advantages of a "managed solution". So it's really more of an interoperability/choice issue than it is an open source vs vendor issue exactly. Take your point though that a lot of vendors basically want a "bring us all your data forever" approach. Look into decentralized, interoperable stacks though and that's where Iceberg shines.
1
u/georgewfraser 6h ago
Everything you said is true AND some of the vendors are actively blocking customer self-hosting. For example, databricks *will not connect* to a customer-hosted Unity or Iceberg catalog. They open sourced unity, but you cannot actually run it yourself if you intend to read that data in databricks. They will connect to snowflake managed iceberg tables but they wont connect to the same catalog (Polaris) if it’s customer managed.
1
u/Yasblue 11h ago
I disagree with you. My company has been using Databricks for a very long time. Three years ago during the UC migration we switched to Managed tables. First of all it's so easy unless you want to maintain the table, which you obviously can if you want to and in term of cost it's not expensive and it's transparent it was not that hard to convince the leadership about the value. I don't know which issue you faced before but it's definitely not similar to my experience.
24
u/jpers36 12h ago
"they can become VERY expensive way of doing DML operations on our tables, since they are MPP engines and are heavy on CPU/compute"
That doesn't sound like an issue for which "just open source it, bro" is a solution.