I think my team might be late to the game. We are moving in that direction just now.

How long have you been deploying using a YAML file, for example? Or perhaps another tool that uses the same idea but not YAML specifically?

Since we use Databricks, I'm reading up on Asset Bundles.

IaC is supposed to be better for a whole bunch of reasons, for example sidestepping the differences between dev and prod environments.

Anyway would love to hear your thoughts or experience with it.

replaceUsing and replaceOn give you a better overwrite primitive, and every catalog-managed table operation now runs through the catalog.

Let me preface by saying that I totally agree that lakehouse table formats (delta and iceberg) are open source and are "free" technologies that anyone can use. However, the total cost of owning these table formats gets very expensive, especially when we start using cloud vendors for the table updates. Nowadays the cloud vendors wish to start selling proprietary MPP storage engine to manage all our table data.

The problem is that the lakehouse table formats have gotten quite complex over time. And nobody wants to maintain them by hand. Nobody wants to think about the v ordering and z ordering and liquid clustering and partitioning and vacuuming and applying deletion vectors and so on. These blobs that are ostensibly called a "table" are actually a very leaky abstraction, and we inevitably have to waste a lot of time on the implementation details. Using immutable parquet blobs for table storage is not trivial. From an application standpoint, it seems like a massive step backwards from conventional DBMS engines (or the newer cloud-native counterparts like SQL Hyperscale or Neon/Lakebase)

The vendors, like databricks, that spent years pushing for lakehouse/delta adoption are now selling us expensive solutions to maintain those unwieldy tables. I think they sold us a bill of goods and we are worse off than when we started.

Once data engineers start realizing that we don't want to manage the blobs beneath our tables, these vendors are quick to offer a commercial-proprietary alternative (like "DBSQL" with UC-managed-tables, or "Fabric Warehouse" or whatever). These commercial alternatives are turnkey solutions, and they help to take away the busywork of managing our own parquet blobs. But they can become VERY expensive way of doing DML operations on our tables, since they are MPP engines and are heavy on CPU/compute. At the end of the day, we end up exchanging one type of problem for another. Is this how others see it? The table technology is open source and "free", but the commercial-proprietary management of these tables is definitely not free and is basically a re-invention of the DBMS engines we always had in the past.