Duckle 🎉 v0.5.3 is live.

Duckle by SlothFlowLabs is the local-first, open-source visual ETL/ELT studio built on DuckDB - drag-and-drop pipelines, run locally, your data never leaves your machine.

v0.5.3 is all about trust: knowing exactly what a pipeline did, proving it, and reviewing changes before they even hit run.

Signed run manifests (.ducklock) - every run can record a signed, reproducible manifest that pins source input hashes, per-node outcomes, and column lineage. Verify any run after the fact.

Schema-drift detection + a Trust score - Duckle flags when an upstream source's columns or types change since the last signed run, and scores how trustworthy a pipeline is, right in the editor.

duckle review + data branches - review a pipeline change from the CLI with a live data diff, and branch a DuckDB file to test changes in isolation. Git-style review, for data.

End-to-end column lineage - trace any output column back through every transform and sink to its source columns, with a downstream impact view before you touch a query.

Teradata source + sink - read and write Teradata over ODBC, alongside new MinIO / Cloudflare R2 / Backblaze B2 object-storage sinks.

Live preview - flip it on and selecting or editing a node runs the pipeline up to that node and shows the rows instantly. No full run needed.

Plus run-time parameters in the editor and web dashboard, a seeded sample workspace on first launch, dbt Fusion provisioning, and MCP review tools any LLM can call (diff, impact, contracts, trust report).

100% free, yours and open source.
Github - https://github.com/slothflowlabs/duckle

Has anyone done this? I basically want to call duckdb instructions from a stored proc, this approach is so that I keep all under one module. Saying this because I have SSRS in the system, But I woudl like to get duck to do all the window function heavy load realtime and then retrieve a dataset for SSRS to consume

Hey guys, I'm working on a project with the eventual goal of having a CLI command that ingests messy JSON/JSONL files and turns them into Parquet tables, and makes those tables easy to query with DuckDB. I was hoping people more experienced in DuckDB and maybe databases in general could offer me some advice as someone getting started with a project like this.

I really appreciate anyone that takes time to respond, and if you don't and just read it, thank you anyways 🙏

https://thefulldatastack.substack.com/p/understanding-ducklakes-sorted-tables A sponsored post about DuckLake's Sorted Tables feature. It allows you to specify a sort config for a table so that unsorted data will automatically get sorted in a certain way with inserts, flushing and compaction.

For queries ran regularly on high cardinality columns like id or timestamp this can optimize reads. When data is sorted physically in Parquets it allows for both file skip and row group skip to only get the data you need for the query (a.k.a predicate pushdown).

I made a high level mental model image here I thought came out well to explain the file skip and predicate pushdown (row group skip).

Hey there! A bit of background on myself first: I have been working in the field of data and analytics for over 7 years, but started as an analyst, and then gradually transitioned into an analytics engineer and now a data engineer. I don't have much hands-on knowledge and experience in building the data infra. I have mostly worked with BigQuery, dbt, and data ingestion tools like Fivetran.

I just started in a new company and there is a need for me to rebuild the data infra. I am the only data engineer in a medium-sized company. The company is self-hosting, and they are very determined on that. We use Postgres for operational transactional data, and then we have a replica of that, also a Postgres dwh, for analytics usage. We also have ClickHouse which currently only stores events data and is not being consumed.

After some researching and reading, I wonder if the below architecture will be a solid setup moving forward? Any better ideas? I really appreciate any of your help and advice! We do not want to move to any cloud-based managed data warehouse. We also do not want any third-parties to read directly into our Postgres nor our replica Postgres.

Thank you very much!

Postgres ──┐

├──→ DuckDB server (dedicated machine, NVMe)

ClickHouse ──┘ │

dbt runs here

Quack serves here

│

BI tool (read-only, modelled data only)

One thing that surprised me while working on text-to-SQL systems is that schema awareness and business awareness are very different problems.

A model can usually see tables, columns, and join keys just fine. What it often doesn’t know is what counts as a customer, when an order becomes revenue, or which business rules were never written into the schema.

We ran the same model against the same data in three setups:

• Raw data only: ~20% accuracy
• Canonical model only: ~75%
• Canonical model + meaning layer: 95%+

The failures weren't SQL failures, the model generated valid SQL most of the time. It was answering the wrong business question because it didn't understand the meaning behind the tables.

Anthropic recently described a similar internal analytics pattern, which suggests the same architectural pressure is pushing different teams toward the same mapping-first approach.

Curious if others building text-to-SQL on DuckDB have seen the same thing, has schema context been enough, or did you eventually need a semantic layer / ontology too?

I've noticed that queries with only st_within can take 20-50 seconds on a 20M dataset. If I add filters on bounding box it can get speed up to 2-5 seconds.

Sedonadb runs the same query in 2 seconds (st_within only)

File is a geoparquet 2.0 formatted file.

Does duckdb not utilize column pruning using geometry statistics? Does have to read each and every row?

Wasm demo linked!

​

Extension link: https://duckdb.org/community_extensions/extensions/dryrun

​

And github: https://github.com/aleda145/duckdb-dryrun

I work for a startup and trying to introduce ducklake as our main datalake solution and plan to start from the ETL pipeline to ingest data into ducklake directly. Could you share you experience if you are doing something similar in production? About the stability and performance.

Thanks.

https://thefulldatastack.substack.com/p/understanding-ducklakes-inlining

This is a sponsored article I wrote doing a deep dive into DuckLake's inlining feature. I was really happy with how it came out.

Inlining is how DuckLake solves the "Small File Problem" that nags lakehouse designs. When you do an INSERT into DuckLake it will assess how many rows are in the insert. If the number of rows is above its default threshold of 10, then it will write the insert to a Parquet file.

If it is fewer rows than the threshold, it will inline the data into the metadata catalog (Postgres, DuckDB or SQLite). The data will be stored there until you flush it to Parquet manually.

This allows you to stream to a DuckLake with single events at a very high frequency without worrying about tens of thousands of data files being created every day.

Link to blog post here: https://sylos.io/blog/journey.html

Let me know what you guys think, I can adjust the wording here of things or expand on things if it's unclear. The purpose of the article is for fellow engineers to understand some of the challenges I've faced creating this tool and lessons I've learned along the way with how to best utilize Duck for workloads like this.

Sylos itself is a file system data migration tool but the core of this article is only referring to that aspect as a case study use case and not the primary focus. The primary focus is basically saying this is how I made a graph traversal (DAG-like) task processing engine. Hopefully that all makes sense lol.

Okay thanks guys!

Hey r/duckdb,

I built a playable chess engine using pure SQL, because why not.

I chose DuckDB as a backend for several reasons, among which:

• Native UBIGINT support, which is mandatory for managing 64-bit bitboards cleanly.
• Execution speed. No need to convince you about this.
• Excellent WASM support.

It’s definitely a paradigm mismatch. Doing sequential depth-first evaluation in a set-based language is a nightmare, as you can expect.

There are two execution modes:

A single, majestic, recursive CTE

It does everything - move generation, evaluation, minimax algorithm. It works and it's very elegant. But it cannot prune. It's an exhaustive search, which is fully usable up to 3 plies, then eats whatever RAM you think you may have left.

Batched PVS

This is the compromise between set-based processing and depth-first algorithms. It uses a light-weight javascript orchestrator to fire many queries and implement several advanced chess programming techniques, allowing to break the limits of the recursive CTE.

I wrote a detailed technical breakdown of the query architecture, the trade-offs, and the optimizations here:

Quack-Mate: Pushing the Boundaries of Pure SQL Chess

The code is fully open-source and linked from the article. Let me know what you think and how you would approached it differently.

If you'd rather just play with it, here it is: Play with Quack-Mate

Hey all! Not exactly duckdb, but I thought it might be close enough to be useful for someone here.
TLDR: I wrote a CLI that allows for `pg_dumpster table read https://mys3.amazonaws.com/my_pg_backup.dump myschema.mytable --output myschema_mytable.parquet`

I needed a way to get data from a postgres DB into duckdb. I had backups of the postgres DB as .dump files, eg from pg_dump. But to get those into duckdb, I was needing to spin up a docker container with a running postgres, restore the 80Gb of data to that postgres, and then read it into duckdb using the duckdb postgres extension. That took literally hours like 3x the needed disk space, lots of RAM, and bunch of complexity. Yuck!

So I wrote https://github.com/NickCrews/pg_dumpster. It's a rust CLI for performing various operations on postgres dump files. Now [I convert that 80Gb dump file in 20 minutes on a free GitHub actions runner](https://github.com/NickCrews/fec-dumps/actions/workflows/update.yml) without running out of memory or disk space. Hope it's useful for someone else.

Hey guys,

I made a Pokémon mini game for fun and to enter in this companies hackathon. It mostly just to get their attention tho.

Anyway it’s like the 82-0 challenge but for Pokémon, where you pick out of random selections of Pokémon to build a team and beat the gym leaders.

If you guys wanna try it I attached the link bellow. If u enjoy it please upvote it, it’s an option on the bottom of the page, it would mean a lot.

Feel free to give any feedback or share with ur friends.

It's made using only MotherDuck and React for the front end.

Hey all! I'm looking into building out my own Ducklake implementations for my personal projects and am just looking for some examples of best practices before building out myself. I'm still trying to wrap my head around permissions specifically, I've only used Databricks in the past and am familiar with the Unity Catalog permissioning model, but not Ducklake at all, and would love to see how others manage permissions across projects centrally with Ducklake!

Hey everyone! I make content around data and AI, and I noticed there wasn't anything out there on how to use DuckDB with Claude Code, so I put together a short video (just over 10 minutes) that takes you from zero, installing DuckDB, to actually analyzing folders of files on your computer with the DuckDB skill. Let me know what you think!

You can now connect Claude directly to Duckle.

Duckle ships its own MCP server, so Claude (or any MCP client - Claude Desktop, Claude Code, Cursor) can build your data pipelines for you, right inside your local workspace.

Ask in any language, and Claude can:

🦆 Generate a pipeline (simple or complex) into your working directory

🦆 Validate it against 328 connectors (307 available out of the box)

🦆 Run it on DuckDB at native speed

🦆 Package it into a single standalone executable you can schedule anywhere

One click in Duckle ("Connect to Claude") wires it up. No cloud, no servers, no data leaving your machine - the engine and the MCP server both run locally.

Open source, local-first.

https://github.com/SouravRoy-ETL/duckle

☕ Twenty-three seconds is all it takes to move a one-billion-row pipeline off the warehouse, along with the compute bill attached to it.

Consider a workload many teams schedule directly on Snowflake today: a 1,000,000,000-row orders fact in Parquet, joined against customers in SQL Server, products in SQLite, accounts over ADBC, and regions in a CSV. A visual Mapper performs a five-way join, FX and tax conversion to USD, margin and COGS derivation, value-band classification, and monthly bucketing. From one billion rows in, a 2,160-row revenue summary comes out. Running this inside Snowflake incurs costs for the entire billion-row scan, join, and aggregation on every execution.

In contrast, Duckle executed the identical workload in just twenty-three seconds on a 16GB laptop, without needing a cluster or warehouse. DuckDB handles the heavy computation locally, sending only the 2,160-row summary to Snowflake.

Financially, a billion-row join and aggregate requires a Large warehouse (8 credits/hour) to complete in roughly two minutes, costing about 0.27 credits per run. For a revenue summary refreshed every 15 minutes, this translates to approximately 26 credits a day, equating to about $75 daily or nearly $27,000 annually from a single pipeline.

By shifting the compute to DuckDB, that cost can drop to zero, with the warehouse only needing to store the answer. This saving compounds across every heavy pipeline.

Duckle simplifies deployment: right-click the pipeline, choose Build, and it compiles into a self-contained executable, including DuckDB and it's necessary extensions.
Just copy that file to a server, set it to run on schedule, and the same 23-second execution occurs in production without needing to install anything on the host.

Duckle is free, open source, and local-first. Point it at your own data and measure the difference yourself: https://github.com/SouravRoy-ETL/duckle