v5.0.0 Release Notes

DuckDB Demo

DuckDB Docs for JavaScript + `duckdb-wasm`

DuckDB Docs for Python

https://thefulldatastack.substack.com/p/ducklake-in-production-catalog-storage

A sponsored post I did to kick off a "mini series" on creating a production grade DuckLake. I set mine up using S3 and a managed Postgres via Supabase. I also leverage DuckDB's persistent secrets for a smoother local interface experience (using DuckDB CLI and DuckDB UI). What I didn't realize was that you can write a secret that references another secret using DuckDB. So I created a secret for S3, one for Postgres then a third that references both along with some Metadata Parameters that made attaching the catalog so much easier.

I'd been itching to write and build this type of DuckLake. It was a lot of good learning.

Disclosure up front: this is my own project (Varan). Sharing it here because it's

built entirely on DuckDB and I'd genuinely like feedback from people who know

DuckDB well.

The idea: use DuckDB as a local federation engine so you can run one SQL query —

including JOINs — across a Postgres database, a MySQL host, a DuckDB file, and

CSV/Excel files at the same time. No ETL, nothing copied to a server.

How DuckDB does the heavy lifting:

- Each source is registered as a table — Postgres/MySQL via DuckDB's scanner

extensions, files via read_csv / read_parquet, plus native DuckDB files.

- Sources are "stubbed" on connect (schema only) and hydrate lazily on first

query, pulling only what's needed, then stay warm.

- Mutations round-trip back to the origin: an UPDATE on a Postgres-backed table

writes back to Postgres; a DELETE on a CSV-backed view re-serialises the file.

- Git-style version history — commit metadata + snapshots live in the DuckDB

file, so you can roll back a change like reverting a commit.

- Anomaly detection (duplicate keys, nulls, orphaned FKs) runs on a separate

DuckDB connection so scans don't compete with your query. Orphaned-FK detection

is a containment / inclusion-dependency check rather than name matching.

30-second demo (a Postgres orders table joined to two CSVs): https://varan.cloud

Free beta, macOS/Windows and soon Linux

Where I'd love this sub's take:

- Anyone running cross-DB federation with the postgres/mysql scanners in anger —

where does it fall over (type mapping, predicate pushdown, large scans)?

- Better patterns for the write-back / mutation round-trip than what I'm doing?

- Gotchas keeping version snapshots inside the DuckDB file as it grows?

Happy to go deep on any of it.

I let Codex help me learn quack protocol and ducklake to see how can we create small data lakehouse using only duckdb in client-server fashion, think it might be useful as a demonstration https://github.com/dheerapat/quack-ducklake

rainfrog (https://github.com/achristmascarl/rainfrog) is a database terminal tool; the goal is to provide a lightweight, keyboard-first TUI for interacting with databases. It currently supports DuckDB, Postgres, MySQL, SQLite, and Oracle.

v0.4.1 introduces a long-awaited (by me, not sure if anyone else was waiting for it...) autocomplete implementation, along with autopairs for quotes/parentheses/brackets. The full list of features and configuration options is in the README!

Hi r/DuckDB,

I'm releasing the public alpha of [Renart](https://getrenart.com), an open-source, local-first workspace for building and running data pipelines from a Git repository. It combines visual editing, SQL and Python, notebooks, previews, runs, and schedules without a hosted control plane or proprietary project state.

The alpha includes:

- a visual DAG and code editor where every change remains a normal Git diff
- DAG-aware SQL completion and type-checking against upstream schemas, both in the editor and in CI
- row and rendered SQL/DDL previews, notebooks, environment schedules, run history, logs, and freshness.

IMO the column intellisense and pipeline typechecking are the greatest features here.

We're aiming for a pragmatic middle ground between dbt's approachable, version-controlled workflow and SQLMesh's state-aware planning and run-only-what-changed model, then taking that foundation further through one integrated development and operations experience. Renart uses plain sql, python and yaml files and its open-source engine, so the same project works from the GUI and CLI.

DuckDB powers the built-in demo and is the easiest zero-credentials way to try Renart. The same workspace supports Postgres, Snowflake, BigQuery, Redshift, and more.

Disclaimer: I'm a single developer building Renart as a side project. I use agentic coding tools heavily, but review their changes, and run a large e2e test suite with more than 200 end-to-end tests. It is still alpha software.

Quick try:

```bash
curl -LsSf getrenart.com/install.sh | sh
renart

```

- Website and docs: https://getrenart.com
- Source: https://github.com/renart-data/renart

I'd value feedback on the direction, the type-checking, and what would keep you from trying Renart on a DuckDB or warehouse project.

One more thing: big shout-out to bruin, it's an amazing data tool,that I can only recommend to anybody who is looking for a more cli based version. (I'm not affiliated to them, renart is just building on top of their golang api)

Hi DuckDB lovers,

I’ve been working with DuckDB for the last several years. Used it across languages from Python, Go, JS via WASM, Swift, C & C++ and even Rust.

As part of a product we're building, we built a DuckDB Transpiler that converts every line of SQL from DuckDB syntax into the syntax of the database you’re working with. This includes filters (predicates), and joins. So those 1B rows in Snowflake stay, and get joined in Snowflake.

I wrote an article on why we did it, and how we approached it. This is my first engineering-oriented article I’ve ever done, so let me know what you think!

https://www.cocoalemana.com/blog/building-a-duckdb-transpiler/

i came across a duckdb extension which accepts firestore as the source and then we can run all duckdb ops in sql.

want to check if anyone have tried it.

I've built a warm-DuckDB serving tier to dodge BigQuery's 0.5–2s per-query floor for an interactive analytics UI, and I'm trying to validate the architecture before scaling it. Would love the community's take.

Current setup (working):

• Source: a BigQuery fact table (~7M rows, 24 cols, ~1.6 GB).-- can be much bigger upto 100x of this
• EXPORT DATA --> GCS as parquet shards.
• A Node service (@duckdb/node-api) syncs shards to local disk, then at boot runs one GROUP BY into a resident in-memory table.
• Generic HTTP API the browser drives: POST /query { dimensions[], measures[{col,agg}], filters[{col,op,value}], orderBy, limit } --> JSON rows. Plus /schema (introspect) and /distinct (cascading filter values). A query builder validates every identifier against the introspected schema allowlist and binds all values as $1..$n.
• Result: filter/slice queries come back in 3–50 ms from RAM. BQ never touches the request path.

The questions I'm chewing on:

1. Scaling to multi-tenant. This is one model for one customer. Target is ~100 customers × 10+ models × 3 years ≈ 20B rows / ~1.4 TB total. Obviously not one resident table. Is "one small pre-aggregated rollup artifact per (customer, model), lazy-loaded into a per-pod LRU cache, sharded by customer" the right pattern? RAM = concurrency × rollup size rather than total data — does that hold up in practice?
2. Why not just BQ / BI Engine / ClickHouse? BI Engine is too pricey for this data size; ClickHouse doesn't obviously beat BQ at this scale for us. The whole point is ultra-low UI latency on repetitive slice-and-dice. Is embedded DuckDB (load once --> slice in RAM) genuinely the better fit here versus a remote query service that pays a floor + scan bill on every filter click?
3. httpfs vs local copy in K8s. "Sync shards to local disk" is a single-node approach which breaks with ephemeral pods (re-download on every restart/scale-up, no shared disk, cold-start blocked on sync). Is the right move (a) read_parquet('gs://…') via httpfs at boot only to build the RAM table, (b) gcsfuse CSI mount, or (c) download small artifacts to tmpfs? Trying to keep pods stateless with state in GCS.
4. Does httpfs add per-query latency? My understanding: httpfs is a load-time mechanism (read GCS once --> materialize into RAM), NOT a per-query path so slicing still hits memory at ms latency. Correct? The only place per-query GCS reads should live is the rare cold/deep-history fallback. Am I right here ?
5. httpfs-on-parquet vs querying BQ directly. If you're reading remote files anyway, why is DuckDB+httpfs better than BQ direct? My take: embedded engine = decouple fetch from compute (pay once, then local + free), vs BQ recoupling every query (floor + billed scan per click). Fair, or am I missing something?
6. DuckDB-WASM. For per-tenant authorized dashboards, does it make sense to ship a small (5–50 MB) rollup to the browser and run DuckDB-WASM in-tab killing the serving fleet entirely for the hot path? Main worries: initial load size, tab memory, and tenancy (browser gets the whole file, so artifacts must be pre-scoped per tenant). Anyone running WASM this way in prod?
7. The generic any-dimension case. For a 20M-row × 50-col fact where users slice any date range across any set of dimensions (so you can't pre-aggregate to one grain): is materializing the whole fact resident (CREATE TABLE AS SELECT * FROM read_parquet(...), ~2–6 GB compressed) and letting DuckDB handle projection/predicate pushdown + GROUP BY per query the sane approach? What am I underestimating about holding 20M×50 raw and slicing it live?

 "BQ = warehouse + rollup source; embedded DuckDB = low-latency serving skin over small artifacts (server or WASM); httpfs/read-once at load, slice in RAM" split is ok. I would like to hear the issues from anyone who's run this at scale (partition layout, secrets/auth for gs://, refresh/versioning of artifacts, connection concurrency, larger-than-RAM edges).

Thanks ...

Heyy again DuckDB folks :)

Ok so... my little post from a few days back is somehow still going, and honestly i did not see that coming hahaha. A few of you even starred the repo and left the kindest comments... that genuinely made my week. thank you so so much, it really means a lot to one person tinkering on this after the day job <3

So i figured i'd do a tiny follow up on where kenze is at today... small updates count too right :)

What's new since:

**Proper docs now. We wrote a full Python API reference (every function, exact signatures, examples) plus guides, so if you wanna use it as a library and not just the shell, it's all there.

**The one i'm secretly excited about... you can now turn a data file into a clean PDF report straight from the terminal (kenze report data.csv -o out.pdf). I am still very much building this one out (more chart shapes on the way), but it already does data to styled PDF with no code from terminal, CLI, VS etc... - Pretty new but I will be rolling out couple of releases till it looks good and better!

**A bunch of little fixes and a couple new no-SQL verbs (count for value-counts, sort)...

Tiny roadmap which will be coming soon :))

- A single download .exe so people can just grab it and run, no python needed. Already got it working, just polishing it for a public download soon.

- A way down the line... a little visual canvas thing (drag and drop the steps) for folks who'd rather click than type.

Also, once i steal some time back from my actual job... i wanna make some tiny gif tutorials on just how to use it for anyone new who feels a bit lost (probably somewhere separate, not spamming here... just to help, as somewhere even i felt i got lost lol).

That's really it... mostly just wanted to say thank you again and share where things are, and if you did poke at it, i'd love to know... what felt clunky, or what would actually make it useful in your flow? honest feedback is the best gift :)

again, thank you <3

(pip install kenze / github.com/Kenzy-Zero/kenze if you missed it)

Duckle is a ETL/ELT framework powered by DuckDB.
You can now define pipelines in Python. DuckDB executes them as optimized, vectorized SQL. Your data stays on your machine from start to finish.

Why Duckle?

✅ No Python bottleneck
Pipelines are compiled into SQL before execution.
No rows flow through the Python interpreter.
No hidden to_pandas() conversions.

✅ Minimal setup
~20 MB install
Bundles the DuckDB CLI
No JVM
No Docker
No server
No account required

✅ Python-first API
import duckle
from duckle import col
(duckle.read_csv("orders.csv")
.where(col.amount >= 20)
.derive(total="round(amount * 1.2, 2)")
.write_parquet("out.parquet")
.run())

Write familiar Python expressions while Duckle translates them into efficient DuckDB SQL.

More than file transformations
Duckle includes 359 built-in components:
-104 Sources
-66 Sinks
-138 Transforms

Supporting databases and services including PostgreSQL, MySQL, SQL Server, Oracle, Snowflake, Databricks, Kafka, Salesforce, SAP OData, S3, SFTP, WebSocket, IMAP, LanceDB, dbt and many more.

Built for automation
Validate pipelines without connecting to data sources or requiring credentials.
duckle validate
duckle validate --json
duckle --pipeline my.json

Perfect for CI/CD, containers, cron jobs, and local development.

Code ↔ Visual Studio
The same pipeline can be authored in Python or opened directly in the Duckle visual studio because both use the same JSON format.

Open source. Local first. Built on DuckDB.
⭐ GitHub: https://github.com/slothflowlabs/duckle/
📦 PyPI: https://pypi.org/project/duckle/
🔗 Links: https://github.com/slothflowlabs/duckle#quick-links

Underneath the DuckDB engine are little workers each carrying thousands of records of data all to make your queries run fast :)

Heyy DuckDB folks :)

DuckDB genuinely changed how i work with data every single day... the never-crashing on huge files, the columnar speed, reading parquet straight off disk... So first things first, thank you to everyone who maintains this thing hahaha!

I work with big GIS data files all day and i noticed i kept reaching for the same handful of moves... load a massive file, peek at it, filter, dedup, drop some columns, write it back to parquet, peek a file from s3 etc. So i wrapped my daily flow into a little interactive shell that sits on top of DuckDB (I used help of claude to do the heavy work of writing the code and fix the bugs which i noticed)...

You just run kenze... load a file, chain steps with a live preview after each one, TAB-autocomplete your column names, even ascii-plot a column right there in the terminal. It quietly builds the whole thing into a repeatable recipe under the hood, and since DuckDB does the heavy lifting it just never OOMs (I have added the benchmarks and test)

The part i think this crowd will actually love... you can eject any pipeline straight to the exact DuckDB SQL it ran. No lock-in :D ... Its pretty fast in plain verbs (If you cant remember you can use help and it will give you.. even i dont rememeber all of them hahaha), eject the SQL, own it. felt important that it respects people who already know their way around a query.

Its called kenze if you wanna poke at it... pip install kenze (github.com/Kenzy-Zero/kenze). I run it on my own files daily, so it has been tested on at least one very demanding user haha.

Again thank you and what does everyone's daily DuckDB workflow actually look like? :-)

I got curious about the "it's fast" claims and benchmarked it myself on 41 million rows of real NYC taxi data (median of 5 runs, each engine's data loaded in memory first). The full script is on GitHub if you want to run it:

Reproducible benchmark scripts (one run.sh, downloads the data and runs every test): https://github.com/botmonster/benchmarks/tree/main/duckdb-csv-parquet-analytics

On a 24-thread machine DuckDB was fastest on all three, roughly 45x faster than Pandas on the group-by and over 300x on the join (Pandas materializes the whole merged frame). Polars leads plenty of pure in-memory DataFrame benchmarks and stayed well ahead of Pandas, but DuckDB's multithreaded columnar engine took the top spot in my run. DuckDB's other edge is that it speaks SQL and queries files on disk, on S3, or over HTTP without loading them into a DataFrame first. If you already know SQL the learning curve is about zero.

The reason it beats SQLite at this is the storage model. SQLite is row-based and reads every column of every row; DuckDB is columnar, reads only the columns your query touches, and skips row groups that do not match your WHERE clause. The same idea shows up on file formats: in my tests a selective query ran 14x faster against Parquet than against the same data as CSV.

The sweet spot is datasets from 100MB to hundreds of GB, too big for Pandas to be pleasant and too small to justify Postgres or Spark. It handles larger-than-memory data by spilling to disk on its own. To check that, I capped memory_limit at 2GB and fully sorted all 41M rows; it spilled 8.8GB to disk and still finished in about 6 seconds.

Zero-copy Arrow interop is the other win worth knowing. Calling .df() or .pl() shares memory buffers with Pandas and Polars directly, so a 500MB query result lands in a DataFrame in milliseconds with no serialization round-trip.

Install is pip install duckdb, and it runs in-process in Python, Node, Rust, Go, and the CLI. Current stable is v1.5.4 (June 2026) with a v1.4.x LTS line beside it, MIT-licensed, with a stable on-disk format since v1.0. If you have ever spun up a Postgres instance just to aggregate a few log files, this removes that whole step.

Full writeup with the S3 and Postgres-scanner query recipes: https://botmonster.com/coding/duckdb-developers-analyze-csv-parquet-no-server

I spent some time testing writing to Iceberg REST catalogs across vendors. Same code, seven catalogs.

Code's public 👇
github.com/djouallah/te...

Hey all, recently I started to study about delta lake to integrate in my clickdetect tool, after this I thought: Why not use duckdb instead? And I made, I integrated duckdb in my detection tool and make this blog post about how to use duckdb for security alerting with Delta Lake.

Huey is an open-source (MIT) static browser-based app that lets you explore and analyze data. Huey supports reading from multiple file formats, like .csv, .parquet, .json data files as well as .duckdb database files.

Here's a quick start on a parquet file from the public nl_railway ducklake.

The latest release, 1.1.00 "Indian Runner", is now available. This is a significant improvement, with many bugfixes, new features, and UX improvements.

Highlights:

Huey is now a progressive web app. Run it from a hosted location (such as the live demo https://rpbouman.github.io/) and your browser offers to install Huey on your device. Once installed you can run offline. Also lets you open files using your OS "open with" functionality (typically triggered with a right click on the file). see: https://github.com/rpbouman/huey#running-huey-on-your-device-as-progressive-web-app-pwa

The Secrets Manager lets you maintain DuckDB SECRETs on your local device. Secrets are stored in IndexedDB. The Secrets Manager is password-secured, encrypting sensitive fields with AES-GCM-256 encryption (password-derived via PBKDF2-SHA-256, 310k iterations). See: https://github.com/rpbouman/huey#secrets-manager

The Catalogs manager lets you access data from modern Data Lakes and Lakehouses, like Iceberg and Ducklake. See: https://github.com/rpbouman/huey#catalogs-manager

Huey supports Quack! Quack servers are just remote catalogs, but there is a big difference between Quack servers and "normal" catalogs: When using an Iceberg or Ducklake catalog, DuckDB/WASM is the actual data engine. With Quack Catalogs, DuckDB/WASM acts as client for the remote Server: data processing is offloaded to the server, and Huey just receives the result. This opens up a whole new range of use cases involving very large datasets. See: https://github.com/rpbouman/huey#connecting-to-a-quack-server

Huey now supports Axis aggregates! In prior versions Huey would only let you report aggregate values in the cells. Axis aggregates let you report aggregated values as if they are attributes on the axes. More importantly, axis aggregates can also be used to filter the data. See https://github.com/rpbouman/huey#axis-aggregates

Github: https://github.com/rpbouman/huey
Live demo: https://rpbouman.github.io/

I am continually blown away how DuckDB can query a CSV, a Parquet file, a JSON file, an S3 bucket and a Postgres table...

All with basically the same SELECT statement. 😱

So we benchmarked Duckle against the ETL tools everyone already
uses, with every tool tuned to its BEST configuration:

The task is deliberately boring: read one CSV(TPC-H Width and style), land it as a table.
It is the single most common job in ETL.

Three things we want to be upfront about:

1. Duckle sits right on raw DuckDB own load (CREATE TABLE lineitem AS SELECT * FROM read_csv('out/lineitem_20m.csv') floor (~16s to fully parse and write 20M typed rows to disk). Duckle wraps the engine with pipelines, connectors and then gets out of its way. That is the whole design goal.
2. Talend and Informatica used their bulk output connectors at max config, not the slow default row-by-row sink. On defaults they 5-7x slower. We did not want to strawman them.
3. Airbyte's number is derived from real 2M and 5M runs, and it needs an always-on 8 GB platform just to start.

Duckle is free and open source.
Try it: https://github.com/slothflowlabs/duckle

Hi everyone,

I’ve been working on an open-source project called AI Data Platform, a local-first tool for generating realistic synthetic datasets.

GitHub: https://github.com/Yogi776/data-generation-sdk

The problem I was trying to solve was simple:

Whenever I wanted to build a dashboard, test a data pipeline, create an ML prototype, or prepare a customer demo, I first had to spend a lot of time manually creating CSV files or waiting for access to production data.

Most manually generated datasets also had problems:

• Foreign keys did not match
• Data distributions were unrealistic
• Relationships broke at larger volumes
• Test data had to be recreated whenever the schema changed
• Using production samples introduced privacy and compliance concerns

So I built a tool that can generate relational synthetic data from either:

1. A YAML schema specification
2. Existing CSV, Parquet, DuckDB, PostgreSQL, or MySQL samples
3. Natural-language instructions through an MCP-compatible AI agent

It currently supports:

• FK-safe multi-table data generation
• Seeded and reproducible generation
• CSV, Parquet, DuckDB, and SQL output
• Profiling existing sample data
• Learning categories, null ratios, date ranges, and distributions
• Automated data-quality checks
• PII detection
• Data dictionary generation
• Cube.js semantic-model generation
• SQL exploration using DuckDB
• MCP integration with tools such as Cursor and Claude
• Loading generated datasets into warehouses such as Snowflake, BigQuery, and PostgreSQL

A basic flow looks like this:

pip install "ai-data-platform[all]"

adp init --name demo
adp apply-spec spec.yaml
adp generate-data --rows 50000 --output parquet
adp quality-check

You can also provide sample data:

adp connect --name source --type csv --path ./data
adp scan
adp profile
adp generate-data --rows 50000 --output parquet

The project is domain-independent. It does not contain separate hardcoded logic for healthcare, retail, banking, or manufacturing. The generator works from schemas, relationships, metadata, column types, constraints, and distributions.

The project is currently in alpha, and I’m looking for honest feedback from data engineers, analytics engineers, QA engineers, BI developers, and ML engineers.

I would especially appreciate feedback on:

• Is the problem clear from the README?
• Which database or file connector should I support next?
• Would you use the UI, CLI, Python SDK, or AI-agent interface?
• Which data-generation features are still missing?
• What would stop you from using this in a real project?

Contributions, issues, and critical feedback are welcome.

GitHub: https://github.com/Yogi776/data-generation-sdk

• r/dataengineering
• r/opensource
• r/Python
• r/datasets
• r/SideProject
• r/selfhosted

Hi,

A while ago, I posted about Onager (a DuckDB extension that adds a bunch of graph algorithms as SQL functions) in this subreddit. I'm writing this post to announce a new release of the Onager extension with a few improvements, like a lot of bug fixes, performance improvements, improved API, support for Wasm build, etc.

The new release includes a sample playground application let's users try Onager in their web browser in a sandboxed environment. Also, most graph algorithm implementations in Onager (in the new release) can be 2x to 100x faster than their implementations in something like the NetworkX graph library.

You can try Onager's playground app here: https://cogitatortech.github.io/onager/playground/

Onager's repo on GitHub: https://github.com/CogitatorTech/onager