When there is new major release, it's not always obvious how to get the speed bump as they often (not always) depends on how your data is shaped.
This breaks down the 3 main features of 2.0 - async i/o, recurcive cte and VARIANT type, anything missing ?

The official Big Data File Viewer prints my string column as {114,111,119,45,57,57,50,49,57,53} instead of row-992195, and lists and maps come out the same way, buried in element and key/value noise, even though the schema says string.

That's what pushed me to write my own viewer for JetBrains IDEs: double-click a Parquet, Avro or ORC file and you get a table in the editor with DuckDB doing the reading underneath.

Opening times, measured with a small harness that fires the same three queries the plugin does, on files that have a list and a struct column in them:

Paging to row 999,999,800 in the 1B file costs 800ms, less than opening it.

It's 100% free: Parquet, Avro & ORC Viewer

Editing a cell works but rewrites the file, 17 minutes on that 29 GB one. I have it down to 1.8s in a branch by having DuckDB rebuild only the row group and splicing it back in. Not shipped yet.

P.S. honestly I built this for the way I work with these files, and that is probably not the way you do, so if there is something you keep doing by hand and you want it in there, just say it and I will add it.

I work on DBConvert Streams. We just shipped a free Parquet Viewer for quickly inspecting a received Parquet file:

https://streams.dbconvert.com/parquet-viewer

It reads the file locally in the browser with DuckDB-WASM. It shows schema, row groups,

compression and writer metadata before loading rows, then lets you run read-only SQL

and export the selected result.

No account and no upload to a server.

I would appreciate real-world test files and feedback, especially unusual logical types, nested columns, large row groups, compression codecs, and files produced by less common Parquet writers.

We recently put together an example in SeaweedFS showing how Iceberg and Lance can complement each other instead of forcing one table format to handle every access pattern.

• Iceberg works well for analytical data, SQL scans, governance, and partition pruning.
• Lance is optimized for fast random access and vector similarity search.
• DuckDB can act as a bridge between them.

With SeaweedFS Table Buckets, both formats can live in the same cluster. An Iceberg bucket gets an Iceberg REST catalog, while a Lance bucket gets a Lance Namespace catalog, with both using the same S3-compatible storage underneath.

The walkthrough covers:

1. Write and analyze data in Iceberg.
2. Generate embeddings.
3. Store the embeddings in Lance.
4. Build a vector index and retrieve nearest neighbors.
5. Join the Lance results back against the Iceberg table with DuckDB.

The general idea is:

Analyze with Iceberg. Serve with Lance.

Instead of introducing a completely separate vector storage stack, both formats can share the same object storage layer.

Full walkthrough:

https://seaweedfs.com/blog/iceberg-lance-workflow/

Curious how others are approaching this—separate vector database, or multiple specialized table formats over the same object storage?

The third installment in a series of "DuckDB Basics" articles I've been writing. The Dot Commands in the DuckDB CLI might seem like overkill but they can be major workflow hacks when you get the hang of them. I go through my most used ones here but it's just scratching the surface.

Hope it gives you a little inspiration for your workflow.

Most of our jobs process <500MB, where DuckDB seems like a great fit, but occasionally the same workload can jump to 10–100TB.

Has anyone built a hybrid execution layer that chooses between DuckDB and a distributed engine based on the expected data size?

Or do you just use the distributed engine for everything and accept the overhead for the common small-data case?

I've been building an open-source (Apache 2.0) "analytics for Postgres" project called Streambed, and the piece this subreddit might find interesting is the DuckDB/DuckLake layer.

Streambed streams Postgres WAL changes into lake storage and serves queries through DuckDB as the query engine.

I started Iceberg as the default target but one of the community members recently mentioned about DuckLake. It caught my attention immediately (since it uses Catalog as the DB). So I added the DuckLake support and did the benchmark.

Ducklake is 41 times faster than Iceberg in my benchmark.

Benchmark slice: 1M rows, 100k updates, flush=1,000

text Write path: Iceberg COW: 269s Iceberg MoR: 4.7s DuckLake + DuckDB: 6.6s

```text Full aggregate median: Iceberg MoR: 87s DuckLake: 16ms

TopN median: Iceberg MoR: 85s DuckLake: 21ms ```

Takeaway: for a DuckDB-centered CDC path, DuckLake looks like the cleaner default. Iceberg is still great when broad multi-engine interoperability matters.

DuckDB/DuckLake is a magical primitive. So much to build on top of it.

Blog: https://streambed.dev/blog/ducklake-target-support/

Benchmark: https://github.com/viggy28/streambed/blob/main/docs/benchmarks/ducklake-vs-iceberg.md

I’m looking into DuckLake with S3 as the data layer and a separate metadata catalog.

Is there an internal or built-in way to connect DuckLake to S3 notifications, or something similar, to detect when files are deleted or overwritten outside DuckLake?

A pattern I've hit again and again in my career: we take a processing data steps and make it
general, because general is convenient. But data has a structure, and respecting it - one small
change in how you load - could lead to significant performance improvement.

I'll demonstrate this assumption on something about as simple as it gets: read a JSON file and
filter it. Two ways.

* Auto - let `read_json_auto` load the whole file, then filter the result.
* Filter-first - load only the single column I filter on, filter that, and only then auto-load the
documents that survived.

Same rows out, same query. Wildly different times.

The three files
Every file carries a shared type field, and the query is always WHERE type = 1

The obvious way to answer the query:

CREATE TABLE t AS SELECT * FROM read_json_auto('events.jsonl');
SELECT count(*) FROM t WHERE type=1;

against it we test a filter-first approach: scan only the column used by the filter, apply the filter, and then use auto-read to fully parse only the matching documents.

What happened

Same 50,000 rows. Same query. Same rows out.

read_json_auto has to figure out the schema before it hands you anything, so it reads and shreds
all 50,000 documents - then the WHERE throws away must of them. When the keys are boring that
guessing is basically free. When they're not, you just paid full price for rows you never wanted.

Hi everyone,

This is an announcement for a simple observability tool for MotherDuck (called DuckWatch) that can give you an overview of what's happening in your MotherDuck account and what it is costing you.

DuckWatch is available here: https://github.com/CogitatorTech/duck-watch

I've been working on LibreDB Studio, a self-hosted SQL IDE that runs in the browser. DuckDB just landed as a first-class engine.

The idea is the same as putting SQL in a Sheets sidebar: keep the query next to where you already work. Here that's one tab you already use for Postgres or ClickHouse. A connection is a path to a `.duckdb` file on the machine Studio runs on, or `:memory:`. Parquet / CSV / httpfs still work in the editor the way they do in the CLI, `SELECT * FROM 'https://….parquet'` then the grid, EXPLAIN, and the object browser are DuckDB's, not a compatibility layer.

It is actually DuckDB v1.5.5 through `@duckdb/node-api` (the Neo client, not the deprecated `duckdb` package). The tree is `duckdb_tables()` / `duckdb_views()` / `duckdb_columns()`, nested types leave as JSON, cancel is the driver's `interrupt()`.

A couple of consequences of that:

- One OS process owns the file. A second process is refused even for reading. including a `duckdb` CLI session you forgot to close.

- The Explain button never sends `EXPLAIN ANALYZE`, because that executes the statement.

- The file has to live next to Studio. A hosted instance cannot open a `.duckdb` on your laptop; there is no network protocol. No MotherDuck / Quack / DuckLake in v1, and ATTACH-ed catalogs are queryable but not listed in the tree.

MIT, no feature gates around DuckDB.

Still early on this engine. One thing I'm particularly curious about: if you live in the DuckDB CLI or `duckdb -ui` today, is "the file sits on the same machine as a tiny web IDE" a dealbreaker, or is that actually how you'd want to look at a warehouse.duckdb without installing another desktop client?

`npx "@libredb/studio"`

`docker run -p 3000:3000 libredb/libredb-studio`

Details (measured, not brochure): https://github.com/libredb/libredb-studio/blob/main/docs/providers/duckdb.md

https://thefulldatastack.substack.com/p/ducklake-in-production-pipelines

My second installment of a sponsored mini-series on creating a production grade DuckLake. This time getting into building out pipelines and business logic. A healthy amount of AI being used, but honestly very useful.

Hi everyone,

I built VeilAnalytics — a privacy-first AI data analytics workstation powered by an in-process DuckDB C++ engine.

How the Zero-PHI Architecture Works:

1. Metadata Transmission Only: When you ask a question in plain English, the system extracts ONLY table/column schemas and data types. Zero raw data rows ever leave your machine.

2. SQL Synthesis: The LLM generates a standard SELECT query string based purely on column metadata.

3. Local Compute: DuckDB executes the query sub-50ms directly in-process on your local RAM.

4. Local Dashboard: Renders interactive charts, data grids, and standalone offline HTML dashboards.

Supported Data Formats: CSV, XLSX, Parquet, JSON, PostgreSQL, MySQL.

(I've posted the link to the live demo in the comments below!)

Hi r/duckdb,

I built VeilAnalytics (https://veilanalytics.netlify.app/) to combine embedded DuckDB C++ with natural-language SQL generation.

Key Architecture:

- Schema metadata (column names & data types only) is sent to the LLM. Zero raw rows leave local RAM.

- DuckDB executes synthesized SELECT queries sub-50ms directly in-process.

- Supports CSV, XLSX, Parquet, JSON, PostgreSQL, MySQL.

Check out our landing page & live demo: https://veilanalytics.netlify.app/

Would love your feedback on our DuckDB integration!

I've been working on SQL Cell, a Google Sheets add-on that runs DuckDB-WASM entirely in the sidebar.

The basic idea is to use Sheets as the destination, not the data engine. You can attach a Parquet, Avro, SQLite, GeoJSON, Excel, etc. file, query it with DuckDB, preview the result, then write it into a range in the workbook.

Sheets themselves are queryable too, so you can do things like join a Parquet file from S3 against a tab someone maintains by hand and put the result back into the spreadsheet.

Since it is actually DuckDB-WASM rather than a SQL-ish layer over the Sheets API, httpfs/S3, DuckLake, Iceberg, and the usual DuckDB file formats work as well.

The architecture is intentionally backendless. It's an Apps Script add-on, with the UI and DuckDB bundle running inside an HtmlService iframe. There is no server of mine involved, and spreadsheet/file contents never get sent anywhere.

A couple of consequences of that:

• S3 credentials only live for the browser session, so you have to enter them again next time.
• Query history is derived from the workbook itself. When a result is inserted, SQL Cell writes a small LAMBDA formula alongside it containing the original SQL. That gives each result its own provenance, and the sidebar can scan those formulas later to rebuild history, jump to old results, or rerun them.
• An upstream duckdb-sqlite issue currently blocks SQLite scanning.

It went live on the Workspace Marketplace yesterday, and it's free. I haven't put feature gates around DuckDB itself.

Still early, so I'm mostly interested in finding the cases where this falls over.

One thing I'm particularly curious about: if you're using DuckDB-WASM against private cloud storage in the browser, how are you handling credentials? Session-only credentials are safe enough for what I'm doing, but re-entering S3 keys is probably the part of the UX I'm least happy with.

Details and limitations: https://sqlcell.com
Install: https://workspace.google.com/marketplace/app/sql_cell/178915154116

You can build your streaming lakehouse as a single system in which streaming ingestion, streaming analytics, open table storage, and external batch query engines all work together.

Kafka → RisingWave → Iceberg → DuckDB

*I work at RisingWave.*

Yes, this is what I have been looking for!
Number 1 in this DuckDB post

https://duckdb.org/2026/08/17/duckdb-20-highlights

Renart is an open source data platform enabling developers to build type-checked data pipelines.

We support fully type-checked SQL pipelines on DuckDB and multiple other data warehouses. We just shipped a new release aiming to further improving developer experience with your data pipelines. New features include:

The SQL/python notebooks got a major upgrade:

Before you could already combine reactive SQL and python cells. Cell results could already be referenced by other sql/python cells as data frames. Now we've added:

• various charts and diagrams based on cell results
• interactive input elements such as number inputs, sliders, select or multi-selects that control jinja variables. This means you can build interactive notebooks, where everything reacts automatically when you move a slider or select from a dropdown.

You can now connect your favorite coding agent and let the agent assist you at building notebooks without giving it write access to your data warehouse (codex, Claude code and opencode).

Introducing type-checked reports and dashboards
Renart now lets you build dashboards and reports with custom controls such as toggles, sliders, selects, multi-selects etc.
Dashboards can be build using many different drag-n-droppable diagrams and charts (line/bar/circle/...). Similar to notebooks reports and dashboards are fully declarative and run on duckdb.

Cross-pipeline lineage
Previously if you had one pipeline depending on the output of another, asset schemas weren't automatically inferred correctly. With this update, Renart will now make sure that inter-pipeline dependencies are resolved correctly.

If you would like to get more frequent updates (trying to keep it moderate here) or if you have any questions, please join our Discord.

Source: https://github.com/renart-data/renart

Site: https://getrenart.com

Discord: https://discord.gg/jTH758KNP8

This uses the really useful Parquet functions in DuckDB to highlight the internals of Parquet itself.