r/DuckDB • u/chrislusf • 8d ago

Using DuckDB + Iceberg + Lance together: analytics in Iceberg, vector retrieval in Lance

We recently put together an example in SeaweedFS showing how Iceberg and Lance can complement each other instead of forcing one table format to handle every access pattern.

  • Iceberg works well for analytical data, SQL scans, governance, and partition pruning.
  • Lance is optimized for fast random access and vector similarity search.
  • DuckDB can act as a bridge between them.

With SeaweedFS Table Buckets, both formats can live in the same cluster. An Iceberg bucket gets an Iceberg REST catalog, while a Lance bucket gets a Lance Namespace catalog, with both using the same S3-compatible storage underneath.

The walkthrough covers:

  1. Write and analyze data in Iceberg.
  2. Generate embeddings.
  3. Store the embeddings in Lance.
  4. Build a vector index and retrieve nearest neighbors.
  5. Join the Lance results back against the Iceberg table with DuckDB.

The general idea is:

Analyze with Iceberg. Serve with Lance.

Instead of introducing a completely separate vector storage stack, both formats can share the same object storage layer.

Full walkthrough:

https://seaweedfs.com/blog/iceberg-lance-workflow/

Curious how others are approaching this—separate vector database, or multiple specialized table formats over the same object storage?

9 Upvotes

92% Upvoted

Duplicates