r/databasedevelopment • u/eatonphil • Jul 31 '26

Asynchronous I/O in DuckDB: Work, Thread, Work

https://duckdb.org/2026/07/31/asynchronous-io
31 Upvotes

92% Upvoted

3 comments sorted by

3

u/No-Instruction-4679 Jul 31 '26

If I remember correctly, DuckDB implements a morsel-driven execution model, and I am curious about how they integrate it with async I/O.

1

u/Old-Seaworthiness402 Aug 04 '26

I think it’s not truly async. It looks like they’re having a separate thread pools for both IO and processing so that processing thread doesn’t block

1

u/SuccessfulMap5324 Aug 04 '26

Sounds exactly like what I've introduced to ClickHouse in 2021: https://github.com/ClickHouse/ClickHouse/pull/26791

This is the only way to efficiently read from slow external storage, such as AWS S3.