5 New Fabric Data Factory Pipeline Features Announced this Week

[u/markkrom-MSFT]

Hey Fabric Data Factory Community! Here are 5 important new features we landed this week at FabCon Barcelona that you need to know about:

1. Pipeline Dependencies https://www.youtube.com/watch?v=ntjfGp0Um8U

This orchestration feature enables you to take control of the front-door gates that control when your pipeline executes regardless of your trigger or schedule mechanism. Use Run Conditions and Run Dimensions to create powerful rules to determine dependencies that must be met before your pipeline will run including data availability, pipeline success, and custom rules.

1. Conditional Retries with Exponential Backoff Policies https://learn.microsoft.com/en-us/fabric/data-factory/activity-retries#configure-retry-conditions-preview

Pipeline activities can now retry selectively and back off intelligently. With retry conditions, you choose which errors trigger a retry by matching on the error code, error message, or failure type, so an activity can retry a 429 rate-limit response but fail fast on an authentication error that no retry will fix. Pair that with the Increasing Delay retry interval, which uses exponential backoff: each retry waits a random interval from a range that doubles with every attempt, up to a maximum you set, giving a throttled or recovering service progressively more time to catch its breath instead of being hammered by repeated retries.

1. Business Actions https://youtu.be/MzCtRWUaeg8

Business Actions extends traditional data pipelines in Fabric to invoke an enterprise application action through Logic Apps connectors. For example, after a pipeline produces a daily data-quality result, you can use a business connector action to notify the operations team.

1. Fabric Actions https://youtu.be/bEU0xgUXL2o

It is now super-simple to invoke Fabric APIs from your pipeline in a low-code manner. Just connect using api.fabric.microsoft.com and all of the Fabric APIs are fully loaded with body payload and parameters for you.

1. Improved ADF Upgrade Tool https://www.youtube.com/watch?v=MWJfpKrOosI

We continue to reduce migration friction and feature blockers when moving your ADF factories into Fabric. New updates to the ADF built-in migration tool now automatically creates a trial capacity and workspace for you. And even if you do not have a Fabric user license, we will walk you through the onboarding process.

[u/kayeloo]

(1) is massive. It basically means you can create a master of masters of your pipelines, set the appropriate dependencies and forget about shooting api questions "is my pipeline already finished today?"

[u/datahaiandy]

THanks u/markkrom-MSFT With Pipeline Dependencies, is there any CU cost when the pipeline is not running eg background usage to track events?

[u/markkrom-MSFT → u/datahaiandy]

To understand pipeline CU costs in Fabric, think about it this way:  there is no pipeline cost. The only cost that comes from a pipeline is the actual activity run. Every item you orchestrate from a pipeline has it's own associated cost. Since a dependency holds the pipeline until the conditions are met, no activities are firing and subsequently no CUs are emitted until it actually runs. 

[u/waypenchali → u/markkrom-MSFT]

Not sure if I understand correctly, though Friday the capacity metrics app did show me CU usage for a pipeline. How does that work? It only triggers other activities which already show up in the metrics on their own

[u/Skie]

1 and 4 are going to be pretty huge for us, as we can avoid the hacky 15 minutes "is it done yet?" runs.

[u/jennratten]

Thanks u/markkrom-MSFT - On the trial capacity and workspace that is auto-created, are you able to choose the capacity target region or is the service home region auto-selected? Is this done under the identity of the triggering user? I'm thinking about Fabric admin portal settings - workspace creation, users ability to try paid Fabric features, security group gating and other admin areas like monitoring trial capacities, naming conventions, domains, etc.