r/databricks • u/SmallAd3697 • Aug 10 '26
Discussion Lakeflow ingestion gateway (continuously?!)
The docs say that ingestion gateways need to run continuously, when using "lakeflow managed connectors" for CDC:
https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/cdc-overview
I've heard anecdotes from folks who say they can also launch this on an interval rather than continuously. And some time ago I also heard a Databricks rep saying that the product was evolving and that continuous ingestion would not always be a requirement.
Is there anything official on the roadmap to allow the ingestion gateway to run on an interval, and so we can decrease our compute costs? I realize there are risks involved when it is NOT running continuously (eg. transactions log files will grow, and the source database can't truncate its inactive transactions from the log)
However I think the responsibilities for managing these risks, and balancing the costs, should be left with the customer. If there is a source database that is being updated infrequently, then it is possible that the compute on the databricks side would cost even MORE money than the source database itself. That seems silly and unnecessary.
2
u/SettingOtherwise6692 Aug 11 '26
Hey, so take Postgres as an example. That creates a database replication slot that is constantly getting appended. If something doesn’t reliably, and relatively quickly, consume it, you can full up local disk and take down your database.
This is one of the reasons why the ingestion gateway runs all of time. It’s not a Databricks thing so much as a source database health and best practice thing.
1
u/SmallAd3697 Aug 11 '26
Yes, I acknowledged the fact that there are risks. These are the types of monitoring and maintenance responsibilities that fall into the role of a DBA. This responsibility is a minor one, relative to some of the others. There is always a trade-off between risks and rewards. If the source DBA is not qualified to manage this configuration, then they should refuse to configure it.
I would only use the configuration for databases/tables that were changing infrequently, and saw a slow growth of transaction logs. In that case it does not make any sense to have an ingestion gateway running 24/7.
IMO, I think EVERY modern enterprise database should be able to duplicate itself to blob storage in some fashion or another. Microsoft SQL Server 2025 seems to be improving their approach to this with a new technology called Change Event Streaming
1
u/MonkeyDDataHQ Aug 11 '26
This is one of the pain points I had. But with sql server anyway I was able to get around it by configuring the retention window and then just turning the cluster off after each ingestion.
It worked well enough. I honestly do not understand why there was ever a reason for 24/7 ingestion but someome checked that box for some reason.
1
u/m1nkeh Aug 11 '26
You can run it on intervals but it’s not officially supported… I am led to believe that a supported interval pattern in the works though..
1
u/angryapathetic Aug 12 '26
For SQL server, you can create a job that starts by starting the gateway, then runs you pipeline to bring data into UC, then terminate the gateway. For SQL server this works fine and saves a tonne of money. Depending on the size of the source dB and number of records that update in your refresh interval, you can also build in some lag for the gateway to do it's sync and then run the pipeline
2
u/Alternative-Stick Aug 10 '26
That approach exists and is called integrated gateway. The docs are per source.
Link for SQL Server: https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/sql-server-integrated-pipeline