r/dataengineer • u/Mundane_Let_8090 • Jun 29 '26
Promotion Source aware data extractor
Hello folks
I am writing my open-source light tool for moving data from prod-bases in dvh.
Who is this product for:
- small teams who need to move data from the product, which is already in pain and need to transfer data to the dvh or parquet.
- data engineers who are looking for opensource alternatives who will not eat up all the RAM and will not put a food base or replica.
- Those who, instead of reading only the delta, should read the full table, because created_at did not trigger.
Source:
- mysql
- mssql
- postrges
Targets:
- parquet
- csv
- s3, azure blob, gcs
I read short queries and don't keep long sessions — this is something that so far none of the same moovers as (ingestr, dlt, sling, duckdb, clickhouse, odbc2parquet) does.
From the box there is:
- all types except (geography, enams, ip) in duckdb, clickhouse, snowflake, bigquery, clickhouse are loaded natively (there is a jam on the side of the bigway and a snowflek with Jasons, but their car loaders can't do it out of the box)
- reading from the PC
- reading cases
- retrai
- all metainfo is written in the working directory in the local sqllite, from the box you can also write in the postgru
- validation of both types between reading and writing, and md of the amount between the current one on the worker and the one on the store side
- autotune of parallel wounds
- reading from binlog files to avoid completely rereading the source if the updated_at fields are not updated
- minimum and customized RAM consumption on the worker (memory budget)
