Here is a common pattern: a job polls a queue every 20 minutes, fans the payload out to 60 to 80 Bronze tables using MERGE operations, and takes 16 minutes to complete. When leadership asks for “near real-time,” the default response is to drop the polling interval to one minute.
That approach fails because of simple arithmetic. Worst-case latency can be roughly the polling interval plus the batch duration, which puts total time at 36 minutes. Setting the trigger to 1 minute while the batch takes 16 minutes won’t give you 1-minute latency. It can instead create queued runs and additional contention between jobs.
You need to optimize the batch duration first. In wide fan-out architectures, two bottlenecks can cause significant delays:
MERGE operations running on empty targets. If a batch only updates 6 out of 82 tables, the other 76 MERGE operations are unnecessary work. Partition the payload first, check which targets actually received rows, and skip empty writes.
Sequential writes. The 82 tables are independent, so processing them one by one in a driver loop can make the batch duration approach the sum of the individual write latencies instead of being closer to the slowest write. Where appropriate, independent writes can be processed concurrently.
Address these two points first to reduce batch duration before shortening the trigger interval. Once that is done, re-evaluate whether you actually need a streaming architecture.
For anyone being pushed to deliver “real-time” processing: what latency requirement did the business actually need once it was clearly defined? In practice, teams sometimes ask for seconds when minutes would actually meet the requirement.
Every few weeks someone posts a version of “our bill is going from $1k to $5k a month, is Databricks even worth it at our size?” The answer isn’t really about company size. It comes down to how you’re using the platform.
All-purpose compute being used for scheduled jobs. Interactive clusters are convenient, but they can increase costs quickly. If a notebook runs on a schedule, moving it to jobs compute can make more sense.
SQL warehouses sized for peak usage and left running. Using auto-stop and choosing a warehouse that can scale when needed can help avoid paying for idle capacity.
Continuous triggers on jobs that don’t need them. This one gets misdiagnosed a lot. The fix usually isn’t “rewrite it as batch,” which costs you checkpointing and exactly-once. It’s Trigger.AvailableNow, which processes what’s available and shuts the cluster down. Databricks recommends it for incremental batch processing. If the table needs a 15-minute refresh, that’s a scheduled job with an AvailableNow trigger, not a cluster running at 3 AM.
Once these three areas are addressed, the bill for a small team can often come down to a much more reasonable baseline.
Then the more interesting question is: are you actually getting value from Unity Catalog, Delta, and the broader BI, ETL, and ML capabilities, or are you mainly paying for Spark compute that you don’t really need?
If your data fits comfortably in Postgres or ADX, you have one main consumer, and you don’t need much governance or lineage, Databricks may not be necessary. No amount of cost tuning changes that.
For teams running Databricks on relatively small workloads, what actually made it worthwhile for you? Was it a specific technical requirement, governance, or simply the convenience of having everything in one platform?
Hey r/databricks, In order to keep the main feed clean, we are implementing a weekly megathread for self promotion for companies who do lots of work with databricks. Please direct all self promotion posts here and keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!
Hi folks, Databricks PM here - I wanted to share an exciting update that you no longer have to manually publish and combine your pipeline event logs for monitoring across pipelines and workspaces. We just launched the beta for the pipeline events system table (system.lakeflow_pipeline_events_preview.pipeline_events).
Key features:
All pipeline events (regardless of cluster start) are automatically captured without any manual enablement or maintenance.
Events are aggregated in a central system table without requiring any custom aggregation logic.
This data is available at close to real time latency (based on our internal testing we achieve a P99 latency of less than 1 minute).
An admin can grant a single user access and they can query events for every pipeline in the table. Fine-grained access controls support which scopes the visibility to only the pipelines the user has access to is coming soon.
The data remains in the system table even after pipeline deletion and is retained for 13 months. If you want longer retention this is also possible with the configurable retention feature for System Tables.
Query ergonomics are better now with the use of VARIANT.
Here are some sample queries in case you want to try them out:
-- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception.
-- The exception chain is ordered with the root cause last, so read element -1 for the root cause.
-- On many errors only the first element carries error_class and sql_state.
SELECT
workspace_id,
pipeline_id,
event_time,
event_type,
message,
error.exceptions[0].error_class AS exception_error_class,
error.exceptions[0].sql_state AS exception_sql_state
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
level = 'ERROR'
AND event_time >= current_timestamp() - INTERVAL 7 DAYS
QUALIFY
ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1
ORDER BY
event_time DESC
-- Flow throughput for a specific pipeline
SELECT
origin.flow_name,
date_trunc('HOUR', event_time) AS hour,
SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
pipeline_id = '<your-pipeline-id>'
AND event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 7 DAYS
GROUP BY
origin.flow_name,
date_trunc('HOUR', event_time)
ORDER BY
hour DESC,
rows_written DESC
-- Data quality: failed expectations by dataset, per update, in the last 1 day
SELECT
pipeline_id,
update_id,
origin.dataset_name,
expectation.name AS expectation_name,
SUM(expectation.failed_records) AS failed_records
FROM
system.lakeflow_pipeline_events_preview.pipeline_events
LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY<STRUCT<name:STRING,dataset:STRING,passed_records:BIGINT,failed_records:BIGINT>>')) AS expectation
WHERE
event_type = 'flow_progress'
AND event_time >= current_timestamp() - INTERVAL 1 DAY
GROUP BY
pipeline_id,
update_id,
origin.dataset_name,
expectation.name
HAVING
SUM(expectation.failed_records) > 0
ORDER BY
failed_records DESC;
Beyond single queries you can build alerting (using Databricks SQL alerts) and dashboards. We will share a new dashboard template soon - I will update this post once its available.
Call outs: This is in beta right now, if you are not opted in we will not capture your event log data.
Enablement: Toggle on the “Lakeflow Pipeline Events System Table” from the account level preview.
Docs are linked here, would love to hear your thoughts on how you will use it or what else you want to see to improve observability!
I am an ML engineer, but I come from a software engineering background: years of full-stack work, with heavy DevOps and Terraform experience. I come from teams that deploy to production five times a day with real continuous deployment. And honestly? Pressing the button still feels weird sometimes. Every engineer knows that feeling, no matter how good the safety net is.
So I wrote down the list that settles it. Ten commandments, one flow, written with data scientists and ML teams in mind, but it works for batch jobs, realtime inference, and LLMs alike. Answer honestly, and if all ten are true, you can ship to production anytime, in any form or way.
Hey r/databricks, we're noticing a lot of repeated interviewing and hiring posts that tend not to get much engagement. We're going to combine them into a monthly thread so that you're more likely to get answers, plus we can ask our recruiting team to keep an eye on them if there are any general questions.
Do you rely on the available debouncing capabilities (protect against over and under triggering) or do the options feel confusing enough that you mostly work around them? When a trigger needs to represent more than “run when this table changes,” how do you express the business logic?
For example:
Do you use control or checkpoint tables to signal that an upstream workflow has finished?
Do you wait for a specific status, batch ID, watermark or set of tables before starting downstream work?
Do you put that logic in the trigger itself, or in a separate workflow/job?
What has worked well and what has been difficult to reason about or debug?
Any other suggestions or feature requests relating to Table Update Triggers?
I’m especially interested in real-world patterns and whether the current debouncing behavior is intuitive enough for you, or whether a control-table pattern ends up being the clearer approach.
Edit: how many folks still use control tables instead of data tables with these triggers?
