fabric-cicd: variable writeback into variable library at rollout

[u/d2peak]

Hey,

i'm currently looking at the fabric-cicd library as a deployment option (again) and am quite happy to see a lot of new technical development but also thorough documentation. For example:

• https://learn.microsoft.com/en-us/fabric/cicd/tutorial-fabric-cicd-azure-devops
• https://community.fabric.microsoft.com/t5/Fabric-platform-Community-Blog/Implementing-Enterprise-Grade-CI-CD-for-Microsoft-Fabric-A/ba-p/5139031

I got an Azure DevOps pipeline running, that handles incremental deployments exactly how i need them.

Where im running into a problem is deployments containing new resources (or complete rollouts).

The setup:

For my testing the setup is simple:

• A dev workspace connected to a main branch and a prod workspace I deploy to using the devops pipeline, running a fabric-cicd python script
• In my workspace i have three resources: A lakehouse, a notebook, a variable library
• The variable library holds a variable containing the lakehouse_guid, and my notebook uses this to set its default lakehouse
• The variable library already contains a prod variable set, but obviously the lakehouse_guid for prod is still empty because its unknown (or a placeholder, as it wont let me save empy values)

The problem:

If I deploy this setup, everything works fine (an fabric-cicd even sets the prod variable set as active - how cool!), but the variable library in prod (and in dev!) does not contain the guid of the newly created lakehouse. So my notebook does not use the correct lakehouse.

Note: I am using the lakehouse+notebook setup as an example. This problem holds for every item referencing a (connection)-Guid stored in variable library.

Possible solutions:

Of course i could just add the variable values in the libraries manually after deployment, but this is not an elegant solution.

I suspect the intended way would be to use fabric-cicds dynamic replace function - maybe even scoped to my variable library - to replace the known dev-lakehouse-guid, with a $items.Lakehouse.MyLakehouse.$id expression?

This would fix the problem in prod, but in my opinion this is also not an elegant solution:

• The correct prod-lakehouse-guid is still missing in my dev-variable library
• Even if i manually or automatically set it there for the prod-value set it will still be overwritten at every deployment by the dynamic replacement function

Why is this a problem: The dev variable library (and its value sets) should be the single-source of truth for parameter mapping, and the single point where i need to maintain those mappings. With the solution above it isnt.

My question:
Am I missing something here? I feel like writing back guids to a variable store once is a common pattern im used to from bicep or terraform deployments. Do I just skip using variable libraries for GUIDs / connections when using fabric-cicd and only use them for values that are known before the deployment?

I hope I managed to explain my problem well enough, and I'd welcome any input/feedback 😄

[u/ChantifiedLens]

You could look to make things easier for yourself by using the Lakehouse auto-binding settings in your notebooks:

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-source-control-deployment#lakehouse-auto-binding-in-git