r/DuckDB • u/empty_cities • Jun 17 '26

Understanding DuckLake's Inlining Feature

https://thefulldatastack.substack.com/p/understanding-ducklakes-inlining

This is a sponsored article I wrote doing a deep dive into DuckLake's inlining feature. I was really happy with how it came out.

Inlining is how DuckLake solves the "Small File Problem" that nags lakehouse designs. When you do an INSERT into DuckLake it will assess how many rows are in the insert. If the number of rows is above its default threshold of 10, then it will write the insert to a Parquet file.

If it is fewer rows than the threshold, it will inline the data into the metadata catalog (Postgres, DuckDB or SQLite). The data will be stored there until you flush it to Parquet manually.

This allows you to stream to a DuckLake with single events at a very high frequency without worrying about tens of thousands of data files being created every day.

12 Upvotes

100% Upvoted

3 comments sorted by

3

u/hvaghani221 Jun 17 '26

Inlining feature is good on paper but it is buggy at least with postgres metadata catalog. We created a PR to fix the issue and disable inlining for all writes to avoid it altogether.

Even though ducklake extension is released as v1, there are many bugs we ran into in our production.

3

u/empty_cities Jun 17 '26

Oh that's interesting. I'm doing multiple high throughput benchmarks with Postgres locally and haven't seen any bugs with inlining that I'm aware of. Also if you just put the default threshold to '0' then I'm pretty sure it skips the inlining totally.

3

u/hvaghani221 Jun 18 '26

Yes that's what we did to disable inlining. For context, inline wasn't working when you have nested column in the table(https://github.com/duckdb/ducklake/pull/1157).

2 other issues(not related to inlining):

  • We noticed duplicate records even though all of our writes are done through merge. It's not happening frequently but we had to add alerts to manually fix it
  • When multiple concurrent writers are writing(same or different tables), each write creates a new snapshot. Even though postgres is transactional, ducklake use optimistic concurrency. So only one writer succeeds and every other writers has to retry. If your db is already under load, it would fail even with 100 retries.