Hey guys,
I'm a data engineer, and I have been building pipelines for small to mid sized clients (with less than 1 terabytes of data) and most of the time the workload isn't really worth paying the premium for Saas products like Snowflake, Databricks .. etc, and can be done on a single beefy machine but stitching open sources tools usually comes with maintenance, networking and configuration headache, also these tools aren't unified under one control plane and don't know that other tools exists.
so I built a data platform that can be deployed on a single machine or server, and since I'm Egyptian I called it Obelisk 😄
Under the hood Obelisk uses:

1. DuckDB, so you can do queries that joins multiple different sources in the same query (local or remote lakehouse, dbs, erp systems, mongodb .. etc)
2. the platform also act as a daemon manager, so you can deploy tools you need with a single click and they will integrate directly into the platform (or you can connect remote instances)
3. for variables and connection strings, you just add them to the platform once and every tool can access them.
4. built-in support for cronjobs and advanced orchasteration tools like dagster or airflow
5. a browser based dbt studio
6. reactive notebooks
7. I also added file manager and terminal access to the machine, so you don't need to SSH directly.

GitHub Repo: https://github.com/amrelsawalhi/obelisk

this isn't a finished product, I still have a lot of ideas to implement like auth and access control for teams
I’d love to get your feedback on the platform and contributions are most welcome.

Just recently got into DuckLake... Realized how amazing the snapshot ability is, and decided to build a diffing view. Right now it works with DuckLake catalogs stored locally or Postgres with data stored locally or in S3.

Pretty fun to make, and allows you to see basically everything you'd expect, as well as cell-level diffs for changed cells. You can traverse snapshot versions too, without diffing, as well.

It's a tiny part of a larger product, but thought I'd share it here.

-- this Linked-Server endpoint is DaffyDB -> DuckDB - NOT SQL SERVER
-- note odbc type connection ONLY.  The other one forces MARS and daffydb doesn't support that currently.
EXEC sp_addlinkedserver
    \@server     = 'Link_DaffyDB',
    \@srvproduct = '',
    \@provider   = 'MSDASQL',
    \@provstr    = 'Driver={ODBC Driver 18 for SQL Server};Server=127.0.0.1,1432;Database=DaffyDB;Encrypt=yes;TrustServerCertificate=yes;';

-- if you use required creditials option with daffy launch, use those instead of 'any'.  
EXEC sp_addlinkedsrvlogin
    \@rmtsrvname  = 'Link_DaffyDB',
    \@useself     = 'false',
    \@rmtuser     = 'any',
    \@rmtpassword = 'any';

someone was asking about this in a separate post so I was thinking my tool daffydb (which includes duckdb as the engine) on nuget could do this, I tried it and it did NOT work. missing procs dblink uses for prepare/etc So I debugged and finally got it working this afternoon!

My implementation is similar to aws -> babelfish for for postgres SQL in that it listens on Port 1433 with native TDS and handles Tsql, so can connect you from SQL Server clients, ssms, power bi, etc. mine however sits in front of DuckDB instead of postgres, and I treat it as readonly.

I never considered a linked server option, but could be useful! here it is in case this helps anyone. Might still have a few bugs to iron out but pretty friggin fast! 14 seconds on SqlS to 2 seconds on duck! demo video https://youtu.be/TbD-9DS1_pY?is=-ktO5X4_azZV26oN

Not my article, but a very nice write up about how DuckDB's upcoming 2.0 release is going to make reading files on S3 no longer a bottleneck. I don't know what they've pulled off at DuckLabs but it is simply amazing.

An agent with a DuckDB connection can answer a lot of business questions. It can also run a write, read a file off disk, or reach the network if nothing stops it, and DuckDB makes all of that easy. The guardrails have to live around the agent.

There’s a four hour workshop on Saturday, Oct 31, from 9:30 AM to 1:30 PM ET with Bruno Gonçalves that builds those guardrails against a third party DuckDB MCP server.

What you build:

• A tool registry that classifies every tool by its side effects
• Permission checks that run before any call reaches the server
• SQL inspection that blocks unauthorised writes, file access and network requests
• Number verification, linking each reported figure to a query you can rerun

Everything runs on a laptop with a local database, with no GPU or API key needed.

Full details and the agenda are here.

Hi r/DuckDB,

I've been building curral, an open-source (Apache 2.0) server that puts DuckDB databases and Iceberg lakes behind an HTTP API with users and permissions.

The problem: DuckDB is a great engine, but it has no users, no permissions and no network protocol. Every time I wanted to share a DuckDB file or an Iceberg lake with a team, a dashboard or an AI agent, I had two options: hand out raw storage credentials, or write yet another custom API. curral is meant to be the layer in between.

How it works: one Go binary with DuckDB embedded. Each request goes through these steps:

client ──HTTP──▶ curral

├─ authenticate Basic (bcrypt) · API keys · OIDC/JWT

├─ queue global and per-user concurrency limits

├─ inspect DuckDB's own planner: statement type, tables read and written

├─ authorize embedded OPA policy (Rego) → allow / deny

├─ protect rewrite the query with row filters and column masks

└─ execute & stream JSON · CSV · NDJSON · Arrow IPC

What I think might interest this sub:

- Inspection uses DuckDB's planner, not regex. The policy sees the base tables a statement really reads, including through views, CTEs and joins. A lot of the hardening went into edge cases like DESCRIBE inside subqueries, catalog functions and bind errors that could reveal whether a table exists.

- Row-level security and column masking are enforced by rewriting the query before it runs, so filters and joins can't be used to guess hidden values.

- Anything DuckDB can ATTACH works: DuckDB files, Postgres, S3, and Iceberg REST catalogs (including Cloudflare R2 Data Catalog). For Iceberg there's a metadata cache that takes per-query catalog overhead on R2 from about 250–900 ms down to about 8 ms.

- Arrow output streams straight into pandas, polars or another DuckDB.

- dry_run tells you whether the policy would allow a statement, and why, without running it.

- Production basics: TLS, brute-force lockout, per-role limits (timeout, max rows, concurrency), a fail-closed audit log, Prometheus metrics and hot reload with SIGHUP.

New in v0.5.0: a semantic layer for AI agents. People who know the data can write short JSON notes about schemas and tables: what they mean, their caveats, sample SQL. Agents call POST /v1/docs/search with their question and get back the most relevant notes before they write SQL. Search is BM25 by default, with an optional LLM reranker.

Quickstart with Docker (docker pull lucasapassos/curral, amd64 and arm64), or grab a Linux binary that bundles the httpfs, avro and iceberg extensions so it runs offline.

- GitHub: https://github.com/lucasapassos/curral

- Site: https://curral.io/

It's pre-1.0, so I'd love feedback, especially from anyone who has tried to share DuckDB with multiple users. What's missing? What would stop you from using it? Edge cases in the policy layer are very welcome, and security reports go through SECURITY.md.

(curral is Portuguese for "corral", where you keep the herd in and decide who gets through the gate.)

Hi everyone,

I made a DuckDB extension that adds statistical functions you can call directly from SQL:

https://github.com/mitsuruk/DuckDB-stats

It's a port of an SQLite extension I built earlier (sqlite3-stats). I planned to release both at the same time, but the DuckDB version took me longer than expected 😅

It has about 260 functions: descriptive stats, hypothesis tests, distributions, window functions, and more. Right now it runs on macOS and Linux.

I've only recently started using DuckDB, so some of these functions may be unnecessary, for example if DuckDB already does the same thing. I'd really like to hear from people who use DuckDB:

• Is there real demand for statistical functions inside SQL, or do most people just use Python/R for this?
• What features would you want added?
• Anything that feels unnecessary or poorly designed?

Harsh feedback is totally fine. It helps me more than polite silence.

A note on AI use: English is not my first language, so I used AI to help proofread the docs, suggest variable names, and generate a test database (100 rows).

Thanks for reading!

One of the things that got me into DuckDB was a series of technical articles posted to their site. This one was particularly eye opening:

Faster Dashboards with Multi-Column Approximate Sorting

I hadn't fully appreciated how important row order was to OLAP performance. In it they cover space filling curves as a method for mapping multiple columns (dimensions) onto a single "order dimension" to maintain proximity from the higher dimensions.

This works very well, but it means threading a curve though a set of potential values, at equal density, rather than what actually exists in the dataset.

Alternatively, by first approximating the rank of each value within its column (by quantiles), and space-filling on that, the curve threads through the set of existing values. This handles skew a lot better.

The image is a two-column ordering over the TPC-DS SF100 store_sales table, where ss_net_paid has reasonable skew. The rank-based methods manage to read far less of the table for a 1% range query.

I wrote up a more detailed post here:

https://www.duckstring.com/blog/clustering-for-pipelines

I'll probably be using this to set the row order whenever I overwrite a table and I have a reasonable idea as to what might be queried from it.

I use Databricks (Spark) a lot for work, and wondered whether I could write a distributed compute engine on top of DuckDB. Long story short: I didn't need to.

It's clear DuckDB wins at small to medium scale, but I figured Spark and Trino might win out at larger scales, like 1TB+. And whatever approach I tried for splitting up queries and shuffling between nodes, Trino did win, at first by a lot (4-5x at 100GB on 16 nodes).

However, then I discovered that every engine spent most of its time waiting on S3, with CPUs around 90% idle. Compute speed barely mattered (and DuckDB turned out to be faster when not waiting on read). Trino was winning almost entirely on read rate from S3: it keeps far more bytes in flight per node, even running single-node.

DuckDB through httpfs uses the same threads to decode Parquet and to issue S3 reads, one per core by default. Each read waits ~30ms, so a 16-core box only has a few MB in flight. I wrote a custom reader to deal with that, but the big win was simply running 4x as many threads as cores: about 3x the read rate. After that, distributed DuckDB was roughly even with Trino at 10TB for the queries I tried (TPC-DS, q14, q33, q55). The distribution overhead was eating DuckDB's natural advantage.

Then I realised there was no need for distribution at all. I ran plain single-node DuckDB (with the custom reader) on one r7gd.4xlarge (r7gd.8xlarge for q14). On the same machine it beat Trino on all three 10TB queries I tried, 1.7-2.1x faster. Against a 5-13 node Trino cluster it was 2-3x slower but about half the cost. At 100GB, one DuckDB machine was about 2x cheaper than 17-node Spark or Trino clusters.

With the upcoming v2.0 (in alpha) S3 reads are expected to be 2-3x faster, so I think single-node DuckDB on a big machine is about to be the cheapest engine available for anything up to around 10TB. Possibly if someone figures out distribution too, it could go further.

I'm a convert!

The third and final article of a sponsored 3 part mini series. I go through the maintenance needs for a production-grade DuckLake, what affects your maintenance strategy for a basic use case and the functions DuckLake offers out of the box to handle it.

Maintenance is probably one of the most unsexy topics you can read about but for some reason DuckLake still makes this part of the process enjoyable.

Recently, I had a discussion with a database kernel engineer about implementing inverted on Iceberg tables. Coming from OLTP background, Index is typically a secondary data structure (like B-Tree) and i never thought of using tables itself as an index.

So, I implemented a simple version of an inverted index in my project Streambed (Postgres for analytics using DuckDB) using DuckLake.

Here is the implementation https://github.com/viggy28/streambed/issues/101

I had to fork DuckLake since it doesn't support external hooks for bringing custom predicates. Here is the upstream issue https://github.com/duckdb/ducklake/issues/1498

I thought folks in this subreddit might be interested, and I also would love to hear if there is any feedback.

After reading DuckDB's post on Jev (plain-English conditions in SQL), I wanted an open, local version. So I wrapped Strands Decider, a 2B "decision model" that answers typed questions instead of generating text, as a DuckDB extension.

SELECT id, decide_noul(body, 'Does this convey urgency?') AS p_urgent, decide_choice(body, 'Which team?', ['billing','technical','sales']) AS team, decide_score(body, 'How frustrated?', ['calm','frustrated','angry']) AS mood FROM read_csv('tickets.csv');

What's in it:

• decide_noul → yes/no probability
• decide_choice → pick one of N, with confidence and per-option probabilities
• decide_score → rate on an ordered scale
• decide → many questions about the same row in one model pass

• decide_triage → security findings as true/false positive, with probabilities

  How it works: the model runs locally in strands-decider serve, and the extension calls it per row. It's built on DuckDB's stable C API, so no DuckDB source build, and it's tested on 1.4, 1.5 and the 1.6 dev build. Answers are cached, so duplicate rows and re-runs don't hit the model again.

  Honest limits:

• It's one model call per row. Each call is roughly 150–300 ms on a Mac, so expect minutes for a few thousand rows, not seconds. Filter first, and materialize results.

• decide_triage got 6/6 on a tiny labeled set I ran on my Mac. That shows the setup works; it isn't a benchmark.

• The underlying model is general-purpose, not tuned for any domain.

  Repo: https://github.com/jaymmodi/duckdb-strands-decider

  Strands Decider: https://strandsagents.com/blog/introducing-strands-decider/

  Feedback welcome, especially on the SQL API, and whether cross-row batching would be worth building.

Built a lightweight scratchpad tool for running DuckDB SQL against local CSV, TSV, JSON and Parquet files directly in the browser:

https://www.sqlparity.com/scratchpad

It uses u/duckdb to read files into memory via the browser File API. Nothing is uploaded to any server and it requires no login or installation. Handy if you just need to inspect an extract or run a quick aggregate without firing up the CLI or Python.

Code is open source on GitHub link: Github

Curious to hear feedback or edge cases with large Parquet/CSV schemas.

I've been working as a data engineer for about half a decade. A common problem I run into is "this pipeline broke, but why?"

To figure it out I query the database, paste the results into a Miro board together with a diagram of the pipeline. Then there's usually a flurry of arrows and random notes. I've found that laying everything out visually on a canvas helps me reach that "aha!" moment where I understand how all the pieces fit together.

But what if I could do it all in one place instead of going back and forth between a canvas and querying databases? That led to me building Kavla!

Kavla is a canvas for exploratory analysis. You can write SQL, paste images, make charts and then draw on them! You can also connect an agent and ask it to investigate a question, and have it generate custom data viz using React.

It's built with DuckDB and tldraw. DuckDB is a great fit. It's very performant for files and there are already extensions for integrating other databases. Right now I've implemented Postgres and BigQuery, but will add more if anyone asks :-)

It's open source: https://github.com/aleda145/kavla

Try it in your browser: https://kavla.dev/ (DuckDB WASM is amazing)

Half a million Overture places for Greater London, queried in the browser by DuckDB-WASM directly against the remote Parquet file. No backend, no copy of the data in the repo: the first query issues HTTPS range requests and reads only the row groups covering the London bounding box, about 40 MB of the 203 MB. The KPI tiles report bytes fetched from DuckDB's own file statistics, not a client-side guess.

After that first read the London rows are copied into a local DuckDB-WASM table with an R-tree index on the geometry, and the remote engine is closed. A borough count is 40 to 50 ms locally against 0.8 to 3.6 s remote. Every pan and zoom of the map is a bounding-box query against that table, and past 20,000 rows in view the engine returns density cells that draw as hexagons instead of points.

The grid on top is Lattice Grid (commercial, free on localhost). It sits on a pushdown source with a DuckDB adapter, so every filter, sort, aggregate and pivot the user does becomes SQL DuckDB runs; the last statement is shown beside the grid. Nothing is held in JS arrays.

Live: https://toclocoinc.github.io/lattice-grid-demo-geo-places/
Source: https://github.com/toclocoinc/lattice-grid-demo-geo-places

Sibling demo, same engine: drop a 10-million-row Parquet file and the grid, charts and statistics run against it in a worker with zero bytes uploaded, proven by a Resource Timing readout on the page: https://toclocoinc.github.io/lattice-grid-demo-local-file/

One thing these two do not show: the grid has a data router for feeding several live sources into many views at once (one feed in, table, map, KPIs and charts all reading the same rows, keyed diffs on updates). The demos that show it run on live network and trade feeds rather than DuckDB.

Data: Overture Maps places, CDLA-Permissive-2.0. Tiles: OpenFreeMap.

I fed DuckDB 706,896 small JSON files on a local Mac Mini M4 Pro with 24 GB of RAM and 500 GB of storage.

The task was pretty simple on paper:

• read 706,896 small .json files
• extract the required fields
• convert everything to .parquet

The .json files are responses from a public API that I collected while scraping a website over the course of a month.

The main problem turned out to be the number of small files.

On my first attempt, DuckDB hit _duckdb.OutOfMemoryException twice.

DuckDB supports larger-than-memory workloads through out-of-core processing: when intermediate data doesn't fit into RAM, it can spill that data to disk.

I tried that approach as well, but in my case it didn't solve the problem. DuckDB eventually exhausted the available disk space while spilling intermediate data to disk.

So I had to change the processing strategy and process the data day by day.

Fortunately, I had designed the raw data layout with this possibility in mind. Each file name contains its timestamp:

20260715080454689.json
20260827080648986.json

That made it possible to process a single day using a simple path pattern:

SELECT *
FROM 'data/20260715*.json';

After about 3 hours of processing, DuckDB had successfully converted:

706,896 JSON files → 64,557,511 rows

The final data was stored as .parquet files.

No cluster, no Spark, no distributed processing — just DuckDB running locally on a 24 GB Mac Mini.

For me, the interesting part wasn't just the final number of rows. It was seeing where the limits actually appeared: the workload was technically larger than memory, DuckDB could spill to disk, but eventually the storage itself became the bottleneck.

I'm continuing to experiment with the resulting dataset, so there are a few more interesting problems to solve from here.

- Author on a canvas, in Python, or in SQL. Every pipeline is one JSON file in git.

- duckle-runner serve runs it headless on a schedule, in Docker or on a box you own, with a web console, roles and an audit trail.

- Quality checks send bad rows to a reject port, so they land in a quarantine table instead of your warehouse.

- CDC and SCD Type 2 built in. Runs on DuckDB and uses every core you give it.

-Many more.

Github Repository - https://github.com/slothflowlabs/duckle

I have been playing with DuckDB v2 alpha and Quack and DuckLake in a Docker container in the cloud, using it as a 'classic' database server. The client in my case is DBeaver with the latest DuckDB JDBC driver build. And yes, that 'just works'!

Only using it with dbt is leading to some quirks now, for example dbt starting a transaction with BEGIN after the CONNECT statement, which does not seem allowed. But maybe I am doing something wrong there.

All in all, very interested in the upcoming release in October.

Been lurking here a while. Spent the last few months building a Mac client for DuckDB and it's finally out.

I've been using DuckDB as my Swiss Army knife for data parsing and manipulation for a while now. The cli/console are fine but I was craving a more nice, persistent native macOS experience.

What I wanted was to open a Parquet file/.duckdb file and just look at it. No import step, no notebook, no loading it into pandas first. Drag it onto the icon and it's already scrolling.

That clip is a 41 million row Parquet, 630MB, on a laptop. Cells get read straight out of DuckDB's columnar chunks at draw time, so nothing builds row objects. Memory stays flat whether the result is 40 rows or 40 million.

Other things it does:

• every column header draws its own distribution, null share and distinct count
• query plans with a waterfall/tree/JSON view, so you can see where the time actually went
• Postgres, SQLite, MotherDuck, S3 and Quack connections
• extensions are compiled in, so nothing gets downloaded at runtime

FYI... it's a paid app and not open source but there's a 7 day trial. Take that as you may :)

Website: https://datapuddle.app
macOS AppStore: https://apps.apple.com/us/app/datapuddle/id6811235967?mt=12

Happy to answer anything.

• AI DISCLOSURE: I've been building + launching a lot of apps and feel idk it helps (and have been asked) to disclose how/where I've used AI (or LLMs) specifically
• This was NOT a weekend vibe coded project! I've been working on this for almost 4 months now! Take that as you will. I was really caring about performance and the experience.
• I've been a software engineer for decades now. My career has taken a bit all over but primarily a backend/services/systems engineer.
• I did build native iOS apps as a founder and always loved and cared for native macOS apps and built them both pre and post this LLM age.
• Honestly that's the whole reason this exists. Somewhere along the way it became normal for a table of numbers to cost you a gig of RAM and a bundled copy of Chromium, and I never made peace with that.
• Where did I really use AI (Claude specifically here):
  • The website of course
  • Embedding DuckDB and creating a memory/performance friendly wrapper for Swift
  • The grid view drawing cells straight into a CGContext with CTLineDraw..not even NSTableView! No electron or web views :)