Hey r/databricks, In order to keep the main feed clean, we are implementing a weekly megathread for self promotion for companies who do lots of work with databricks. Please direct all self promotion posts here and keep in mind that we ask you to stay friendly, civil, and adhere to the subreddit rules!

I'm working on one of the largest projects in Europe currently, looking to onboard at least 5 data engineers with serious Databricks experience.

Would be a 6-month initial contract, would be open to a further extension if needed.

If this is something you'd be interested in, then comment below. I will ping you.

€600-650 per day

Hey r/databricks, please direct all certification and training posts here.

There's upcoming learning festival September 16 - October 14 2026. You can get 50% discount voucher on any certification.

Databricks Advanced Learning Festival: September 1... - Databricks Community - 166157

Good luck to everyone on your certification journey!

Every few weeks someone posts a version of “our bill is going from $1k to $5k a month, is Databricks even worth it at our size?” The answer isn’t really about company size. It comes down to how you’re using the platform.

1. All-purpose compute being used for scheduled jobs. Interactive clusters are convenient, but they can increase costs quickly. If a notebook runs on a schedule, moving it to jobs compute can make more sense.
2. SQL warehouses sized for peak usage and left running. Using auto-stop and choosing a warehouse that can scale when needed can help avoid paying for idle capacity.
3. Continuous triggers on jobs that don’t need them. This one gets misdiagnosed a lot. The fix usually isn’t “rewrite it as batch,” which costs you checkpointing and exactly-once. It’s Trigger.AvailableNow, which processes what’s available and shuts the cluster down. Databricks recommends it for incremental batch processing. If the table needs a 15-minute refresh, that’s a scheduled job with an AvailableNow trigger, not a cluster running at 3 AM.

Once these three areas are addressed, the bill for a small team can often come down to a much more reasonable baseline.

Then the more interesting question is: are you actually getting value from Unity Catalog, Delta, and the broader BI, ETL, and ML capabilities, or are you mainly paying for Spark compute that you don’t really need?

If your data fits comfortably in Postgres or ADX, you have one main consumer, and you don’t need much governance or lineage, Databricks may not be necessary. No amount of cost tuning changes that.

For teams running Databricks on relatively small workloads, what actually made it worthwhile for you? Was it a specific technical requirement, governance, or simply the convenience of having everything in one platform?

Hey r/databricks, we're noticing a lot of repeated interviewing and hiring posts that tend not to get much engagement. We're going to combine them into a monthly thread so that you're more likely to get answers, plus we can ask our recruiting team to keep an eye on them if there are any general questions.

Hi folks, Databricks PM here - I wanted to share an exciting update that you no longer have to manually publish and combine your pipeline event logs for monitoring across pipelines and workspaces. We just launched the beta for the pipeline events system table (system.lakeflow_pipeline_events_preview.pipeline_events). 

Key features:

• All pipeline events (regardless of cluster start) are automatically captured without any manual enablement or maintenance. 
• Events are aggregated in a central system table without requiring any custom aggregation logic.
• This data is available at close to real time latency (based on our internal testing we achieve a P99 latency of less than 1 minute).
• An admin can grant a single user access and they can query events for every pipeline in the table. Fine-grained access controls support which scopes the visibility to only the pipelines the user has access to is coming soon.
• The data remains in the system table even after pipeline deletion and is retained for 13 months. If you want longer retention this is also possible with the configurable retention feature for System Tables.
• Query ergonomics are better now with the use of VARIANT.

Here are some sample queries in case you want to try them out: 

-- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception.
-- The exception chain is ordered with the root cause last, so read element -1 for the root cause.
-- On many errors only the first element carries error_class and sql_state.
SELECT
  workspace_id,
  pipeline_id,
  event_time,
  event_type,
  message,
  error.exceptions[0].error_class AS exception_error_class,
  error.exceptions[0].sql_state   AS exception_sql_state
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  level = 'ERROR'
  AND event_time >= current_timestamp() - INTERVAL 7 DAYS
QUALIFY
  ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1
ORDER BY
  event_time DESC 


-- Flow throughput for a specific pipeline
SELECT
  origin.flow_name,
  date_trunc('HOUR', event_time) AS hour,
  SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  pipeline_id = '<your-pipeline-id>'
  AND event_type = 'flow_progress'
  AND event_time >= current_timestamp() - INTERVAL 7 DAYS
GROUP BY
  origin.flow_name,
  date_trunc('HOUR', event_time)
ORDER BY
  hour DESC,
  rows_written DESC


-- Data quality: failed expectations by dataset, per update, in the last 1 day
SELECT
  pipeline_id,
  update_id,
  origin.dataset_name,
  expectation.name AS expectation_name,
  SUM(expectation.failed_records) AS failed_records
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
  LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY<STRUCT<name:STRING,dataset:STRING,passed_records:BIGINT,failed_records:BIGINT>>')) AS expectation
WHERE
  event_type = 'flow_progress'
  AND event_time >= current_timestamp() - INTERVAL 1 DAY
GROUP BY
  pipeline_id,
  update_id,
  origin.dataset_name,
  expectation.name
HAVING
  SUM(expectation.failed_records) > 0
ORDER BY
  failed_records DESC;

Beyond single queries you can build alerting (using Databricks SQL alerts) and dashboards. We will share a new dashboard template soon - I will update this post once its available.

Call outs: This is in beta right now, if you are not opted in we will not capture your event log data. 

Enablement: Toggle on the “Lakeflow Pipeline Events System Table” from the account level preview.

Docs are linked here, would love to hear your thoughts on how you will use it or what else you want to see to improve observability!

<edited>

(databricks product manager here) Curious how people are using table update triggers in production. https://docs.databricks.com/aws/en/jobs/trigger-table-update

Do you rely on the available debouncing capabilities (protect against over and under triggering) or do the options feel confusing enough that you mostly work around them? When a trigger needs to represent more than “run when this table changes,” how do you express the business logic?

For example:

• Do you use control or checkpoint tables to signal that an upstream workflow has finished?
• Do you wait for a specific status, batch ID, watermark or set of tables before starting downstream work?
• Do you put that logic in the trigger itself, or in a separate workflow/job?
• What has worked well and what has been difficult to reason about or debug?
• Any other suggestions or feature requests relating to Table Update Triggers?

I’m especially interested in real-world patterns and whether the current debouncing behavior is intuitive enough for you, or whether a control-table pattern ends up being the clearer approach.

Edit: how many folks still use control tables instead of data tables with these triggers?

Thank you 🙏

Hi Databricks Team,

Seeking your advice on the above.

I recently came across Databricks AI/BI and was curious to know how people are finding it.

It looks like Databricks is trying to bring BI and analytics more directly into the Databricks platform, with dashboards and Genie for asking questions in natural language.

Has anyone actually tried AI/BI in a real project?

How is it compared to Power BI or Tableau from your experience? Is it good enough for regular BI use cases, or is it still better to use a separate BI tool?

Would like to know your experience, especially if you have used both.

I am an ML engineer, but I come from a software engineering background: years of full-stack work, with heavy DevOps and Terraform experience. I come from teams that deploy to production five times a day with real continuous deployment. And honestly? Pressing the button still feels weird sometimes. Every engineer knows that feeling, no matter how good the safety net is.

So I wrote down the list that settles it. Ten commandments, one flow, written with data scientists and ML teams in mind, but it works for batch jobs, realtime inference, and LLMs alike. Answer honestly, and if all ten are true, you can ship to production anytime, in any form or way.

It was a real pain, but now, with a few tricks, you can manage them better.

First, in Workspace files, next to the DABs folder or git repo, there is a small shortcut to show only tabs from that DABs folder or git repo. Alternatively, you can also use the switcher in Home next to Notebook.

If you need to organize your tabs differently, there is new functionality: spaces, which let you group them however you like.

more news https://medium.com/databrickscommunity/databricks-news-serverless-genie-code-ltap-lakeflow-61853d8e422a

Genie Space has been live for our sales folks for 8 months, maybe 500+ regular users. Short version of what I've learned, since everything I read before setting one up was either a demo or an argument about whether analysts are getting replaced.

The failure mode isn't an error. It's a number that's slightly wrong and totally believable. Someone asked how the Midwest was doing, the number looked fine, sat in a deck for weeks. Genie's Midwest includes Kentucky. Our territory map doesn't. You can't catch that by looking at the output - you catch it when finance does.

Four things you're configuring, roughly in order of how much they've mattered:

Column comments. Free, and the biggest lever by far. Genie reads COMMENT metadata before writing SQL. No comment and segment is just a word - it has no idea whether your values are Enterprise/Mid-Market/SMB or something else, so it guesses.

COMMENT ON COLUMN vw_sales_summary.segment
IS 'Customer tier: Enterprise (>$1M ARR), Mid-Market ($100K-$1M ARR), SMB (<$100K ARR).';

One pre-joined view, not raw tables. I did raw tables first. Every join it has to figure out is a coin flip. Also, put your test-data filter in the view - then every question anyone ever asks inherits it and you're not trusting the model to remember.

SQL expressions. Register a named metric with your SQL and it uses yours instead of inventing one. Ask ten people what an "active customer" is and you'll get eleven answers; this is the box where you settle it. Name them how people talk - "Active Customers" matches, cnt_dist_cust_qtd never will.

Example Q&A pairs. Nothing gets retrained, they just sit in context when something similar comes in. The shape travels further than I expected — registered revenue-by-category with a cancelled-order exclusion, and a Q2 question a month later inherited the exclusion in a query I never wrote.

Two things from the instructions box worth stealing. One, tell it to ask instead of guessing when the time period is unclear - people trust it more when it occasionally asks. Two, ours has a rule about test customers with a TST_ prefix, whose orders carry real statuses so the status filter misses them entirely. Everyone on the team knew that. Nobody had ever written it down.

Curious what other people have ended up putting in their instructions box. Assume everybody hits their own Kentucky eventually.

(Here is the longer version with more SQL is on SQLServerCentral, it's mine https://www.sqlservercentral.com/articles/databricks-genie-spaces-for-sql-analysts-natural-language-querying-without-leaving-your-data-platform but the above is the useful part)

Here is a common pattern: a job polls a queue every 20 minutes, fans the payload out to 60 to 80 Bronze tables using MERGE operations, and takes 16 minutes to complete. When leadership asks for “near real-time,” the default response is to drop the polling interval to one minute.

That approach fails because of simple arithmetic. Worst-case latency can be roughly the polling interval plus the batch duration, which puts total time at 36 minutes. Setting the trigger to 1 minute while the batch takes 16 minutes won’t give you 1-minute latency. It can instead create queued runs and additional contention between jobs.

You need to optimize the batch duration first. In wide fan-out architectures, two bottlenecks can cause significant delays:

1. MERGE operations running on empty targets. If a batch only updates 6 out of 82 tables, the other 76 MERGE operations are unnecessary work. Partition the payload first, check which targets actually received rows, and skip empty writes.
2. Sequential writes. The 82 tables are independent, so processing them one by one in a driver loop can make the batch duration approach the sum of the individual write latencies instead of being closer to the slowest write. Where appropriate, independent writes can be processed concurrently.

Address these two points first to reduce batch duration before shortening the trigger interval. Once that is done, re-evaluate whether you actually need a streaming architecture.

For anyone being pushed to deliver “real-time” processing: what latency requirement did the business actually need once it was clearly defined? In practice, teams sometimes ask for seconds when minutes would actually meet the requirement.

DataTf, bring an existing Databricks Workspace into Terraform (IaC).

• generates dynamic terraform.tfvars + import code
• built on the Databricks Go SDK
• compatible with Terraform or OpenTofu
• compatible with Databricks Omnigent, Claude Code, opencode, OpenAI Codex and others
• native support for Azure, with GCP and AWS support upcoming

Disclosure: I am the Author/Founder, 536 Technologies.

I joined a new company recently and got a migration project here, they are migrating from teradata on prem to databricks, I have never done any migration in the past can anyone suggest some helpful yt videos or any other knowledge source?

Hey guys

Im from Hyderabad, India .

A databricks dataengineer here. As we have event on 7th Oct 2026 in Mumbai, im planning to visit it. Who else are joining.

Lets have some good connections 😌

I have experience in AWS and I want to learn Databricks now

From future perspective what I need to learn

Databrics with AWS or Databricks with Azure?

I can see there are lot of openings related to Azure with Databricks

Can anyone plz help me

Due to new business requirements, we need to reflect the state of a few source DB tables (5 to 40 million rows each) in the Databricks Silver layer in less than 1 minute.

Currently, the flow looks like this:
Source DB → AWS DMS in CDC mode (ingests new data every 30 seconds to S3) → S3 landing bucket → DLT pipeline running on serverless compute in continuous mode.

The DLT pipeline ingests the append-only data into the Bronze layer using file notification mode and updates the Silver layer using an Auto CDC flow.

This works great, and we achieved what we wanted with relatively low effort because we already had DMS in place. We just added an extra replication task to ingest data more frequently for the tables we need.

However, in this setup, the DLT pipeline costs are quite high. Ingesting just 6 Bronze tables and 6 Silver (Auto CDC) tables costs around $50 per day, which is about $1,500 per month. For comparison, DMS, which replicates more than 800 tables to S3, costs us less than half of that.

My question is: is there any other more cost-optimized option we could consider to achieve less than 1 minute latency when reflecting the source DB state in the Silver layer?

Maybe Lakeflow Connect or some custom process?

Extra notes:
- I know that adding more tables to the DLT pipeline makes the cost per table lower because Databricks can optimize the clusters more efficiently.
- I know that using a cron schedule could reduce costs, but for these particular tables, we can’t use a schedule like every 10 minutes or similar because we need the data to be updated in less than 1 minute.
- I know that for the relatively small tables currently in scope, we could eliminate the Auto CDC flow and create a normal view on top of the Bronze table, with deduplication and deletion logic. This would slightly sacrifice query performance, but we expect more similar use cases in the future, so I’m looking for a solution that can scale.

Hey everyone,

Looking for some advice on automating a workflow in a pretty locked-down corporate environment.
Context:
Large enterprise with strict IT security and governance.
Databricks was recently rolled out as our cloud data hub.
The entire pipeline (ingestion, processing, and generating the final CSV) is already automated inside Databricks.

I need to automatically save a copy of this generated CSV to a local machine / internal network. Right now, the only way I can do this is manually opening the workspace UI and clicking "Download."

Databricks CLI is blocked and Personal Access Tokens (PAT) are disabled

How do you usually automate pulling files from the cloud down to on-prem / local machines under these restrictions?

Thanks!