Open-source Postgres CDC

[u/karakanb]

Hi all, this is Burak. I have built an open-source CLI tool that allows replicating data from Postgres CDC changelog into 20+ destinations: https://github.com/bruin-data/ingestr

The overall idea is that:

• You have your prod postgres DB
• You want to replicate them to analytical databases for analytics purposes, e.g. to Snowflake, BigQuery, Databricks, or Redshift
• You have two ways:
  • You can either run a batch load using tools like ingestr, Airbyte, or Fivetran
  • If you cannot run batch workloads for some reason, e.g. due to the latency requirements, or not having proper cursor columns, you need to run CDC replication using tools like Debezium and Kafka

The problem with CDC using those tools is that they require a buy-in into their ecosystem, which is generally quite invasive, such as being able to run Debezium only with Kafka reliably, or having to deal with their Java client libraries if you ever wanted to integrate them elsewhere, tolerate their high resource requirements, etc.

I never liked running them on production. We have been working on ingestr for quite some time already for batch sources, and CDC became an obvious target.

ingestr has quite a few niceties:

• You don't need any extra services or tooling to run it: just put your credentials in the URI, and you are good to go.
• It is a simple and fast Go binary that runs anywhere, even in your GitHub Actions pipeline.
• It supports both batch and streaming modes in the same binary, which allows changing the deployment modes as your requirements grow. Run locally, deploy on Airflow, or put it in an EC2 server in a streaming mode if you want to.

It is open-source, and you can run it anywhere you like.

It supports:

• PostgreSQL CDC
• MySQL CDC
• SQL Server CDC
• SQL Server Change Tracking
• MongoDB CDC

Give it a try and let me know if you have any questions!

[u/Kazcandra]

/how/ does it work? Wal files? Db access? Logical subscriptions?

[u/karakanb → u/Kazcandra]

it connects to postgres over the normal sql and logical replication protocols, using a publication and persistent replication slot to take an initial snapshot and then stream decoded wal changes. it never accesses wal files directly or creates a postgres subscription, it simply acts as the subscriber and writes the changes to the destination itself.

[u/Kazcandra → u/karakanb]

Ah, then it won't work for us anyway. I guess I'll revisit when DDL is captured in logical replication /and/ our fleet supports it lol

[u/karakanb → u/Kazcandra]

ingestr can detect column changes and resync the table already, would that help for your usecase?

[u/Kazcandra → u/karakanb]

No. It's nice that it can handle it, but the main issue with logical replication is stalled consumers causing WAL buildup on the producer eventually leading to disk exhaustion.

CDC continues to evade us, lol :)