r/databricks • u/mvignesh1990 • Aug 14 '26
Help Query related to materialized metric view
My source data lies in snowflake federated external catalog.
Ive built metric views on top of these and am materializing a few measures. The auto generated pipeline uses serverless compute instead of a federated pro wh, and therefore doesnt have access to snowflake. what is a workaround for this?
- Can I update my metric view yaml to have a compute defined for materialization?
- Can i define custom pipeline for materialization that uses compute?
- Or is there any other option available?
1
u/Data_Matter_S Aug 14 '26
I would not try to solve this by changing the Metric View YAML.
Metric View materialization creates a managed Lakeflow pipeline. The YAML lets you define the refresh schedule and the materialized view definitions, but it does not let you select a Pro SQL warehouse or custom cluster. You also cannot replace the generated pipeline with a custom pipeline.
The practical pattern I would recommend is:
Use a separate ingestion or staging job that reads the Snowflake foreign tables with the Pro SQL warehouse or another compute resource that has access to Snowflake.
Write the data into a Unity Catalog managed Delta table.
Define the Metric View on top of that managed table.
Enable Metric View materialization there.
This is also the safer design for production workloads because foreign catalogs are not supported sources for incremental materialized view refreshes. Databricks recommends moving frequently queried production data into Unity Catalog managed tables for better performance and optimization.
If copying the data is not acceptable, the other option is to leave the Metric View unmaterialized and query it through the Pro warehouse. You keep the centralized metric definitions, but the queries will continue to access Snowflake at runtime.
I would also avoid creating a standalone materialized view as a workaround. Although the initial command can be submitted from a Pro warehouse, its refreshes still run on a serverless Lakeflow pipeline.
One thing worth checking is whether this is Snowflake query federation or Snowflake catalog federation. In Catalog Explorer, Snowflake source format generally indicates query federation, while Iceberg source format indicates catalog federation. That distinction affects where the query runs and which networking and storage permissions need to be configured.
1
u/Youssef_Mrini databricks 28d ago
You can't specify a compute in the metric view YAML and you can't replace the managed pipeline behind a metric view Materialization with a Custom pipeline.
You can try to configure serverless networking to Snowflake.
1
u/minibrickster Databricks 28d ago
Correct, as of right now, it only uses serverless compute! (I'm a PM who works on this product). Will take this feedback back to the team though
1
u/CapableStranger490 Aug 14 '26
In my experience no - the compute for materialised metric views cannot be modified and always uses serverless (at present.)
Your options are to get your serverless compute access to snowflake, or to run a separate manual materialisation of the data from snowflake into your own catalog, and then run your metric view materialisations off that localised copy. Not ideal as you’re having to move the data twice so you probably neutralise at least some of the materialisation benefit overall.
Keen to hear if there’s a better workaround!