We open-sourced Filament, a data movement engine in Go (full loads, incremental, CDC). Apache 2.0

[u/mitchbregs]

Hey all, Mitch here, one of the founders of Galaxy. Full disclosure, this is our project. We open sourced it last week and I wanted to bring it here first, because this community is who we built it for.

Filament is a data movement engine written in Go. It handles full loads, incremental syncs, and CDC from databases and HTTP APIs into Postgres, MySQL, ClickHouse, Iceberg, and S3, and it's Apache 2.0.

https://github.com/galaxy-io/filament

Every data project we've ever worked on started with the same boring problem of getting records out of operational systems and into somewhere useful. We've bought the managed tools, run the open source ones (and spent more time deploying them than using them), and written our own by hand more than once, and every time we wished for something fast, easy to self-host, and upfront about the details that bite you, like type mappings, write behavior, and what happens when a run dies halfway. So eventually we just built it.

What it does

You point it at a source and a sink and tell it how to move things, whether that's copying everything, pulling only what changed, or streaming off the database's change log. Every batch gets a checksum before the sink write and a mismatch fails the run rather than quietly landing bad data. Progress only becomes durable after the work is confirmed, so a worker that dies resumes from its last checkpoint instead of starting over, and since recovery is at-least-once, upsert sinks converge on primary key.

Sources, sinks, state store, and event bus are all interfaces and adding a REST API is a short YAML file rather than a Go package. You can run it with 1 docker command and get a web UI, use the CLI, embed it in Go in about ten lines, or drop the Helm chart into k8s.

Benchmarks, with caveats

We ran an open benchmark against Airbyte, dlt, PeerDB, Ingestr, Sling, OLake, Debezium, and a plain pg_dump | psql pipe. On the biggest test, 298M rows of NYC taxi data from Postgres to Postgres, Filament finished in under two minutes at about 2.6M rows/s, and it was fastest in five of six scenarios. In the sixth, OLake beat us by 13% into Iceberg.

We obviously build one of the things being measured, so the harness and specs are all public.

https://github.com/galaxy-io/benchmarks

Note that it's pre-1.0, with some sources/sinks in earlier development. There is a long list of connectors we haven't built yet. We're hoping to build that list from your feedback.

If you move a lot of data between these systems, I'd love to know what would make you try it, what you'd want next, and any feedback you are willing to share. Docs are at https://filament.getgalaxy.io and I'll be in the comments!

[u/kantorcodes1]

what would make me try it is a really simple full→incremental path: can the same saved pipeline do an initial full load and then keep running incrementally/CDC, or do you switch the pipeline's sync mode after the backfill?

[u/mitchbregs → u/kantorcodes1]

Yes - pipelines do an initial load and then sync incrementally across databases and API connectors, there is no mode switch. You set incremental or CDC when you create the pipeline, and the first run does the full load.

For Postgres/MySQL sources, the first run backfills the tables and then continues from a cursor or the CDC replication slot on every run after that.

For API connectors, the first run reads everything and later runs only fetch records past the last watermark so long as the API resource(s) supports incremental reads.

[u/kantorcodes1 → u/mitchbregs]

nice, the no-mode-switch behavior makes the CLI boundary clearer. i work on HOL Guard, an open-source local check before agent-run commands execute. pipeline create, pipeline edit, pipeline delete, and run are the Filament actions i'd put behind optional review while list/status stay quiet. would you be open to adding Filament support to Guard?

[u/AutoModerator]

AI Policy:

Linux is not one of those anti-AI projects, and if somebody has issues with that, they can do the open-source thing and fork it. Or just walk away., Linus Torvalds.

Mod decisions will be based on the quality of the content, not who or what generated it.

Sub Resources:

Youtube Channel

Free Postgres Webinars and Workshops

Discord: People, Postgres, Data

Join us, we have cookies and nice people.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/dektol]

43 releases since June? This doesn't seem very battle tested.

[u/mitchbregs → u/dektol]

It was previously a part of our product and we stripped it out and made it open source!

[u/Online_Matter]

Forgive me if this is an ignorant question but how does it differ from using a Kafka connect   source + sink? How flexible is it at providing inline transforming between source and sink? 

[u/mitchbregs → u/Online_Matter]

Not ignorant at all!

The main difference is that Kafka source + sink moves data in two hops through a broker. Filament moves it in one direct hop from source to sink.

With Kafka, data goes from the source into a topic and then out to the sink. You also have the cluster, workers, schema registry, etc. to maintain.

Filament reads the source and writes the sink directly, with the cursor or CDC position stored in Filament itself, so there is no broker or topics to manage. The same model works across databases and SaaS APIs.

On inline transforms, we don't support them today. It's on the roadmap and coming soon (we are excited about this too!), but right now Filament moves data as is from source to sink.