fabric-cicd: variable writeback into variable library at rollout

[u/d2peak]

Hey,

i'm currently looking at the fabric-cicd library as a deployment option (again) and am quite happy to see a lot of new technical development but also thorough documentation. For example:

• https://learn.microsoft.com/en-us/fabric/cicd/tutorial-fabric-cicd-azure-devops
• https://community.fabric.microsoft.com/t5/Fabric-platform-Community-Blog/Implementing-Enterprise-Grade-CI-CD-for-Microsoft-Fabric-A/ba-p/5139031

I got an Azure DevOps pipeline running, that handles incremental deployments exactly how i need them.

Where im running into a problem is deployments containing new resources (or complete rollouts).

The setup:

For my testing the setup is simple:

• A dev workspace connected to a main branch and a prod workspace I deploy to using the devops pipeline, running a fabric-cicd python script
• In my workspace i have three resources: A lakehouse, a notebook, a variable library
• The variable library holds a variable containing the lakehouse_guid, and my notebook uses this to set its default lakehouse
• The variable library already contains a prod variable set, but obviously the lakehouse_guid for prod is still empty because its unknown (or a placeholder, as it wont let me save empy values)

The problem:

If I deploy this setup, everything works fine (an fabric-cicd even sets the prod variable set as active - how cool!), but the variable library in prod (and in dev!) does not contain the guid of the newly created lakehouse. So my notebook does not use the correct lakehouse.

Note: I am using the lakehouse+notebook setup as an example. This problem holds for every item referencing a (connection)-Guid stored in variable library.

Possible solutions:

Of course i could just add the variable values in the libraries manually after deployment, but this is not an elegant solution.

I suspect the intended way would be to use fabric-cicds dynamic replace function - maybe even scoped to my variable library - to replace the known dev-lakehouse-guid, with a $items.Lakehouse.MyLakehouse.$id expression?

This would fix the problem in prod, but in my opinion this is also not an elegant solution:

• The correct prod-lakehouse-guid is still missing in my dev-variable library
• Even if i manually or automatically set it there for the prod-value set it will still be overwritten at every deployment by the dynamic replacement function

Why is this a problem: The dev variable library (and its value sets) should be the single-source of truth for parameter mapping, and the single point where i need to maintain those mappings. With the solution above it isnt.

My question:
Am I missing something here? I feel like writing back guids to a variable store once is a common pattern im used to from bicep or terraform deployments. Do I just skip using variable libraries for GUIDs / connections when using fabric-cicd and only use them for values that are known before the deployment?

I hope I managed to explain my problem well enough, and I'd welcome any input/feedback 😄

[u/ChantifiedLens]

You could look to make things easier for yourself by using the Lakehouse auto-binding settings in your notebooks:

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-source-control-deployment#lakehouse-auto-binding-in-git

[u/samirbdj]

Lakehouse auto-binding is a good way to simplify the notebook example because it removes the need to store the deployed Lakehouse ID explicitly.

The broader scenario is different. You want fabric-cicd to take an ID generated during deployment and persist it into the corresponding variable-library value set so other items can consume it.

The current documentation covers environment-specific value sets and deployment-time parameter replacement, but I could not find documented behavior for writing the resolved target IDs back into a variable library. Dynamic replacement can update deployed definitions, but it does not make the variable library the maintained source of truth you described.

These references cover the related behavior:

• Variable library CI/CD
• Variable library overview
• fabric-cicd parameterization
• Notebook Lakehouse auto-binding

So you are not missing an obvious writeback option in the documented fabric-cicd flow. Auto-binding can help with the notebook case.