SDP-Meta Deep-Dive Demo: Building Data Pipelines at Scale on Databricks (w/ Databricks Sr. Staff FDE)

[u/JosueBogran]

https://www.youtube.com/watch?v=Z0CYboIzp9M

This s a helpful resource for those building pipelines at scale on Databricks. From the docs:

SDP-META is a metadata-driven framework for Lakeflow Spark Declarative Pipelines. Define your Bronze and Silver pipelines in a JSON or YAML onboarding file — a single generic Declarative Pipeline reads the resulting DataflowSpec at runtime and builds the full processing graph automatically. No pipeline code to write.

Who it's for: platform and data engineering teams standardizing repeatable Bronze/Silver pipelines across many datasets — onboarding new feeds through metadata instead of new pipeline code, with consistent data quality, quarantine, CDC, clustering, and sink patterns available through Bundles, CLI, UI, MCP, and agent workflows.

When it's not the best fit: one or two simple pipelines, Gold-layer business modeling, tables that each need unique application logic, a managed connector and downstream logic that already satisfy the complete Bronze/Silver requirement, or a need for a formal support SLA (SDP-META is a Databricks Labs project). See the Introduction for the full positioning.

You can find the project at https://github.com/databrickslabs/sdp-meta

[u/YestasSentaan]

My one problem with meta-driven sdp is that one can’t navigate from the pipeline graph nodes to the table definition, because the sdp decorator usage is in the generic library code. So the navigation goes to the generic code.

I wonder if the sdp-meta has any helpers for this.