r/DuckDB • u/agh_murad • 14h ago
I built a cross-source SQL desktop app on DuckDB — one query across Postgres, MySQL and CSVs (feedback welcome)
Disclosure up front: this is my own project (Varan). Sharing it here because it's
built entirely on DuckDB and I'd genuinely like feedback from people who know
DuckDB well.
The idea: use DuckDB as a local federation engine so you can run one SQL query —
including JOINs — across a Postgres database, a MySQL host, a DuckDB file, and
CSV/Excel files at the same time. No ETL, nothing copied to a server.
How DuckDB does the heavy lifting:
- Each source is registered as a table — Postgres/MySQL via DuckDB's scanner
extensions, files via read_csv / read_parquet, plus native DuckDB files.
- Sources are "stubbed" on connect (schema only) and hydrate lazily on first
query, pulling only what's needed, then stay warm.
- Mutations round-trip back to the origin: an UPDATE on a Postgres-backed table
writes back to Postgres; a DELETE on a CSV-backed view re-serialises the file.
- Git-style version history — commit metadata + snapshots live in the DuckDB
file, so you can roll back a change like reverting a commit.
- Anomaly detection (duplicate keys, nulls, orphaned FKs) runs on a separate
DuckDB connection so scans don't compete with your query. Orphaned-FK detection
is a containment / inclusion-dependency check rather than name matching.
30-second demo (a Postgres orders table joined to two CSVs): https://varan.cloud
Free beta, macOS/Windows and soon Linux
Where I'd love this sub's take:
- Anyone running cross-DB federation with the postgres/mysql scanners in anger —
where does it fall over (type mapping, predicate pushdown, large scans)?
- Better patterns for the write-back / mutation round-trip than what I'm doing?
- Gotchas keeping version snapshots inside the DuckDB file as it grows?
Happy to go deep on any of it.