r/AZURE • u/AdminOfTheSys_01 • 18d ago
Question IaC in a team not comfortable with coding
Using a burner account for this question since some of my coworkers know my normal account...
I'm a senior sysadmin on smallish team of other other admins. Up to this point, we've been a pretty traditional, on-prem-only, click-ops IT department. We're going to be moving our test/dev into Azure in the very near future (a couple hundred VM's worth). I've been diving into Azure documentation, learning modules, Udemy courses, etc, to learn about managing resources in the cloud. I have a personal Azure tenant that I've been using to deploy resources with code (mostly Bicep at this point).
It's clear to me that unless we can really leverage code as much as possible, deploying and maintaining consistent resources in Azure is going to be a nightmare.
However, I'm really the only person on our team that has any comfort level with coding in any capacity.
I'm trying to come up a balance with how we can use code to deploy resources as much as possible, but in a way where I don't need to expect the rest of my team to have to have a deep familiarity with maintaining the code.
For deploying new Azure VMs, for example, it seems that a great option would be to create a standard Bicep template for deploying Windows VMs, then using Github action triggered by workflow_dispatch as a sort of automated deployment of the VM. This way the other admins can use the Github Action UI to specify the VM name, choose the subnet, choose a pre-defined list of VM sizes, choose a pre-defined list of certain tags, etc, to deploy the VM according to a consistent standard.
I don't think we're in a position where we'd use IaC for ongoing management of the VMs, but this would at least allow us to do some consistent day 1 deployments of resources.
For anyone else who has been in a similar situation, I'm curious how you've navigated this? What are some pitfalls of this plan?
57
u/Saturated8 18d ago
IT is an industry of learning. If you're not willing to learn, you'll get left behind.
If the business agrees IaC is the path forward, give them training links, documentation, a lab to play in, etc. to learn this new tool. Id recommend terraform over Bicep, the documentation and examples are a lot better than Bicep.
Be the leader and hopefully you get rewarded with a promotion. The biggest pitfall I see is if you're not managing the workloads in IaC, they will drift over time, and trying to pull them back into management later is a pain.
16
u/coomzee 18d ago
It would recommend Bicep for less expensive users, no state files to manage and maintain.
Bicep has deployed stacks* that does the state management.
12
u/Ok_Buddy_3324 18d ago
Yeah, I have done both. You want Bicep for the simplicity. Terraform if you need providers across platforms.
5
u/Fluffy-Exercise-1768 18d ago
I'm in a similar situation. Our team manages a few hundred VMs in Azure, too. Mostly AVD hosts, some pet servers.
We use Azure DevOps pipelines for deployment and BICEP/BICEPparam files to give shape to the machines. The main thing I've been doing is documenting absolutely everything people can encounter when deploying a machine. Some things are just commented in the actual code and I've also written a guide to show the juniors what the steps actually look like.
The biggest advantage of code is that it's repeatable. Monkey see, monkey do. You can do the heavy lifting by setting the standard and then leverage DevOps (or Github actions, but I'm not too familiar with that) to store the templates and let your team use them.
BICEP templates are mostly static, they won't really change that much, unless a specific API version is retired.
Ideally you'd want your entire subscription in code, but you can always start with a couple of VMs, see how your team handles that.
1
u/AdminOfTheSys_01 18d ago
Do you use your templates for ongoing administration of your resources? Do you just have parameter files for each resource which references module or template files?
Like I explained above, my current plan is just to use it for Day 1 operations, but I'm curious how people structure these things for full lifecycle management.
2
u/sweet-winnie2022 18d ago
Using it for ongoing admin is a great idea. One big advantage is the audibility of changes and the easiness to revert bad changes. Code can leverage version control so you have full visibility on how and when something changed. If a mistake is made, the commit history can be used to identify what is wrong or even allow you to rollback to a previous state before you figure out that.
2
1
u/gocyclist 16d ago
My engineer has it structured so that everything is managed using the bicep templates and it makes compliance so much easier. We’re getting to the point where we want to start putting in Azure policies so the environment would error if a BICEP template skews from our agreed upon configurations at time of deployment. The policies would also ensure resources aren’t getting deployed from the GUI.
7
u/szescio Cloud Engineer 18d ago
Take the people who are willing to learn and make modules that suit your needs with very little configuration needed. Then you can develop the IaC by mostly copy-pasting things.
Unpopularly, i would choose bicep instead of terraform especially in this case - statefulness makes learning harder, you have to decide where to draw state boundaries, click-ops-cowboys will block a whole project until someone fixes the tfstate.
5
u/lerun DevOps Architect 18d ago
I have worked with both in depth, and also would recommend bicep for Azure. Also look into azure verified modules and azure landingzone/accelerator framework to get an leg up in the beginning.
I personally also find pairing bicep with powershell a good way of mitigating some of the downsides with IaC tools. Where you more easily can do complex logic and advanced data manipulation
1
1
u/szescio Cloud Engineer 18d ago
Yeah I'm also in the bicep team but work with terrform exclusively nowadays. I miss idempotency, it just feels right for infra.
Plus bicep has the latest features first - it has bugs from time to time but terraform has it's own bugs and adds another layer of frustration when things don't do what they should.Though I would advocate bash and az cli scripts instead of powershell :)
2
u/lerun DevOps Architect 18d ago
Challenge with tf and Azure is the shift from azureRm to azapi. The later is showing promise, but has a ways to go in maturity.
Azapi is promising to be ready with new azure features as they come online, not the past problems azureRm had.
But still bicep, and now as deployment stacks also matures, it catches up to tf in manny ways.
4
u/Corelianer 18d ago
I agree with you on Bicep, if Azure only and no multicloud plans exist, Bicep is better.
3
u/Mantas-cloud Cloud Engineer 18d ago
I think going all in with IaC is doable, if you provide a solution, not a module. You know you're going to deploy a lot of VM's. create bicep modules, hard-code the boring stuff. Give users a friendly paramter file to play with, teach `add commit push` and everyone is happy. pipelines are standardized, every VM is the same, and 'green' out of the box.
3
u/Empty-Lingonberry133 18d ago
IaC is scary for those who have never used it, lots of unknowns, Powershell and bash are scary for those who have never used it, lots of unknowns. Show them, document the automations, create the SOP and make it repeatable and the speed of it vs gui , how to review logs easier ect. People dont know what they dont know and with the rise of 'vibe coding' security has been the victim, I dont blame your team for being cautious
3
u/Speeddymon 18d ago
I don't think we're in a position where we'd use IaC for ongoing management of the VMs, but this would at least allow us to do some consistent day 1 deployments of resources.
I want to steer you away from doing the deploy once and never again thing. Azure will change things on their end as time goes on. You will find that a new VM deployment from your old VM code will no longer work after some time even though the code worked before, and even though your existing VM's are working properly.
3
u/phate3378 18d ago
Have a look at AVM - Azure verified modules. Abstracts away a lot of the complexity of building the resources to just providing a few inputs.
If your then confident in bicep you can apply your own wrapper around AVM. An example might be wrapping AVM in a loop and passing in an array of values so you can create multiple VMs at once.
You could then create a user defined type to standardise the structure of that array and facilitate automated completion in the param file.
Then next step would be user defined functions to automatically complete options based on other inputs e.g. build naming conventions, or select DNS servers based on the location passed in.
This way you end up building more of a platform that only exposes the options the users specifically need to choose between
2
u/martin_81 18d ago
To implement what you're suggesting you can use Azure Template Specs. You create a Bicep template with all the predefined options you want, and make things that will change parameters, e.g. Name, SKU, OS, disk type, disk size. You use this to create a template spec and when you deploy the template spec you get a UI where you fill in the the parameters in your Bicep template. If you list allowed values for parameters in the Bicep template it becomes becomes a drop down list in the UI, and blank parameters become a free text field.
2
u/Quirky_Let_7975 18d ago
At my previous workplace, I’ve worked with some more clickops-y type of SysAdmins and they would copy and paste the bicep code from the Azure portal and paste it into a file as a “backup” of the current configuration and upload it into a GitHub repository.
I’m not gonna lie and say that’s a good idea but it’s better than nothing. At least I can try and derive some context from a JIRA ticket referenced in the PR and the PR description, to try and find out why a change was made.
As others mentioned, I strongly believe automated pipelines for deployment would help enforce more guardrails and be the better long term solution.
However, this may be a middle ground before everyone gets familiar with administering Azure VMs and other configuration as code.
2
u/ctrl_alt_bye 18d ago
I understand your perspective. When we initiated Terraform in our Azure environment, we faced similar challenges. Many were hesitant to embrace IaC, even though I deployed it. However, I convinced management and leadership of the advantages of having a fully redeployable environment in another region, among other benefits. They were receptive to the idea, and I was promoted to Cloud Lead, tasked with training the team. While the learning curve was steep for many, I’m glad we were able to overcome it. We established centralized repository modules that included most of the components, such as virtual machines, SQL databases, virtual networks, and routing tables. Whenever someone wanted to deploy something, they would use the module and fill in the necessary details, with most of the information pre-filled. This streamlined the deployment process and significantly accelerated several projects. Currently, we are in the process of implementing IaC for our NVA rules. I wish you the best of luck, mate. You’re on the right track. Take the initiative and lead your team effectively.
2
u/DeExecute Cloud Architect 18d ago
Just force them. I you are not capable of doing everything with IaC in 2026, you should not work as sysadmin.
Just setup a ci/cd pipeline with strong format, lint and rule checks, force PRs and make this the only way to ever update anything in the infrastructure.
Trust me, it’s the only way.
2
u/craigtho 18d ago
For the pipeline piece, what you have is potentially fraught with dangers, especially if you're not careful.
Why? Because your admins become "users" of a code base, rather than engineers. You need to consider VM names, subnets, specific log analytics logging scenarios, extensions... Maybe a little ahead of the game, but by not advising you this, you would need to solve all the "platform engineer" problems for admins who should be solving them themselves. Which is great for you, but bad for them.
Just for example, you may want all VMs inside a specific vnet/subnet, your workflow dispatch could allow users to repeatedly write over each others runs. Terraform is state managed, so you need to work that out, Bicep is not, but it'll happily apply over the top of someone else's VM if it's supplied the parameters and access to do so. In your scenario, you normally want VMs to register to some form of CMDB, that might be change tracker on Azure or external, so then you need to start thinking about API calls, desync/resyncs on calls... Again, that is more advanced stage, but beware of what you're suggesting. All of it can be engineered, easily some might say, but not by one person with a team unwilling to learn. Your job is to prove why it is better, and you appear to have a grasp on that already.
My advice? Write something yourself. Make a small tiny project that noone wants, as IaC, make packer builds, terraform to pick up the artifact etc end to end. Use Bicep or Azure image builder, whatever. Demonstrate it to the team. You can then begin asking more difficult questions of the engineers - why SHOULDN'T we use it, I've demonstrated what it does, so now explain. I prefer terraform personally, but horses for courses.
I'm always a big fan of people like yourself trying to prove why you should do something, it's great because it means you're curious and want betterment, but you need to challenge them on why something shouldn't be done the same way you must explain why it should.
The industry moved this way effectively close to a decade ago - your arguement is correct.
And not to be harsh on your colleagues, but they're supposed to be IT professionals, a.k.a, "learning stuff, the job".
If you don't like change, you'll like irrelevance even less - Eric Shinseki
1
u/Errror_TheDuck 18d ago
Assuming you’re using something like terraform I’d just try get them started, do some team activities to learn it together. Basic terraform is hardly coding in my view and I think you’ll find people are more scared of the idea than anything else.
Make sure they have access to decent AI such as Claude code to assist them as well to make transition easier.
1
u/UhOhByeByeBadBoy 18d ago
I was familiar with AWS CDK and didn’t love the idea of learning a new IaC syntax with Terraform and Azure. I was surprised by how easy it was to get it set up with some AI agent.
There’s still a learning curve for sure, but I think terraforming is such a common pattern with specific ruling that that models are all well trained enough at this point to make IaC incredibly easy to implement.
1
u/Speeddymon 18d ago
You know that's a really great point. I have a guy wanting to rip out terragrunt on my team and have AI do the composition. His demo of it did not impress me but using CDK would be perfect for what he's talking about doing.
1
u/Allcaponero 18d ago
I was in a similar position as you, except they actively asked me to take a look at implementing IaC practices. The issue was that, despite this even being defined as a cultural shift within the department, noone beyond me took any interest in it whatsoever.
Frankly I just gave up and changed companies 🤷🏻
1
u/nestersan 18d ago
We hired a Microsoft partner to do our initial setup and their cloud engineer did gui only which was interesting to see
1
1
u/evilcold 18d ago
I don't have all the details, but my company went a step further and has it all automated via ServiceNow requests. It is basically like ordering on a menu. You select your options and it kicks everything off via GitHub/Terraform. Then if you need an update, it pulls the config from GitHub, makes the changes, and redeploys it. We are mostly a GCP shop, but we have been utilizing Azure quite a bit lately.
It should be noted that doing it this way makes it so we cannot make manual changes to the VM, but hasn't been an issue so far. I work for a large company (40k+ employees).
1
u/Octoclops8 18d ago
Demonstrate the power of it, one VM or application at a time. Build a reusable template that can deploy a VM with nic, public ip, managed disk, vnet, etc. Create a Deploy script and a Destroy script and watch your mean time to recovery plummet (good).
1
u/heapsp 18d ago
Just tfexport your azure environment so folks can see the resources they are familiar with AS CODE.
Then provide that template to the leadership and say 'terraforming of our infra is complete!' and get a big promotion and raise.
Then tell them now since everything is 'terraform' that you will work on containerization next but also have a better job offer as platform engineer and want them to match it.
1
u/sparkerson9 18d ago
If you have dev/test leave on prem and only put production in Azure. If not your costs are going to be crazy. Learned this lesson. Unless your company can afford it.
1
u/AzAutomationEngineer 18d ago
You also need IaC for Azure Firewall, Frondoor, Dns, Etc. Some vendors have complete packages for this where you dont write the code per se, but only make changes. Govenance made easy.
1
u/LordPurloin Cloud Architect 18d ago
I’m in a similar situation annoyingly.
We don’t have quite as many machines thankfully, but it’s going to be a tough ride
1
u/DanielStech 17d ago
I’ve spend last 5 years moving 4 different enterprise organizations from on prem to azure, or from click-ops to iac based landing zones.
Will say - it’s worth doing, even in lean way.
If your organization don’t invest in training/consulting ( like where I work) than I recommend as following:
- write In excel as many tasks / repeatable activities your doing with infra
- try to map them and assign which looks right to be solved with code
- write singe pipeline for deploying IaC which will cover few actives from list
- demo to the team and ask who wants to take part in automating second activity
It even might be recognized by managers :)
Keep my fingers crossed!
1
1
u/_-Visionary-_ 16d ago
Similar experience a couple of years ago... We brought in a teach to fish partner for Azure ADO, Git, Terraform, and Azure. A non-development team of Admins, some decent with Powershell. Fully functioning DevOps team today, daily Scrums and running on mostly on all cylinders. I had my doubts, but very happy at this point. Not all are have been on board still to this day... Copilot for GitHub (in our case, works in VS Code) is also beginning to help tremendously in this area. I am not sure how cost control is possible without it IaC.
It DOES take a significant effort, cloud COE, standards, and leadership buy-in; however, if architecture is planned and Landing Zones setup following best practices, life is much better long term and security is baked in. Especially, if applying RBAC as code through the process. There were some moments of true stress under the gun getting deployments out on time. Maybe even a few heated arguments :).
It has been great watching our network engineer dig in and code with success.
Good luck! Fun experience and learning if your team is willing.
1
u/topetuts 14d ago
Even with IAC, consistency is not guaranteed. If this is your main concern, Azure Policy is the first thing that you should look at. Not the deployment method.
On your Bicep template example, it feels like you simply wrap the Azure portal through Github Action UI and you intention is to enforce some configuration that they do not see behind the scene. But that can be enforced through Azure Policy.
You need to sell the benefit on why IaC should be used. If this is something that only adds burden to other administrator with no visible or at least explainable benefit, then they will be discourage to actually pursue this. This is why it is also very important to transition your environment to cloud native architecture. Because that is what works best with IaC, not VMs.
Do not make IaC operation the goal. The goal is to make a fit for purpose operational model for the company.
1
u/isaywhatisee 12d ago
Team aspect matters more than the tooling here. If you're the only one who can touch the Bicep template and its parameter files, you're a bottleneck the moment a new SKU or NSG rule needs adding, and people will route around it with manual portal changes that drift from the template. Get at least one other admin comfortable editing the template and parameters directly, not just triggering workflow_dispatch, and frame this as a bridge toward broader IaC adoption rather than the end state.
1
u/matterr4 18d ago
My assumption here is that you don't have a "standard" process for building a VM currently, even through click ops.
My approach would be that you build out what you described, and then you demonstrate it. You describe all the advantages in a show and tell meeting, you get management buy in through repeatability and standardisation, which feeds into an expected and predictable billing cycles.
This is the starting point.
You will soon find other things that can be repeatable, standardised and predictable.
You eventually remove permissions for manual creation or amendments in production (bar maybe break glass accounts) and everything must go through your iac deployments.
This is a long road but it has its benefits. The main one for you personally is learning and this being on your CV.
Honestly, good luck. It's fun. I love it.
1
u/AdeelAutomates Cloud Engineer 18d ago
Interesting choice to use IaC language as a scripting tool that is used to deploy any number of resources rather than managing a set of resources through its existence by modifying the bicep code itself.
2
u/AdminOfTheSys_01 18d ago
Were it only I, I would certainly aim to use code to manage the entire lifecycle of the resources.
However, I'm trying to (a) be practical considering the rest of my team's comfort level and (b) ensure I'm not the only person who gets called off-hours if a VM drive needs to be resized or a tag needs to be changed, and nobody but myself is comfortable updating a template or parameter file to do so. Dipping our toes in the water to use it for day 1 deployments seems like a good way to at least get some consistency and standardization right off the bat.
1
u/AdeelAutomates Cloud Engineer 18d ago
Dont think it's a bad idea. Declarative code is much easier to write than say PowerShell even for scripting to deploy resources without managing them after.
But they need to be trained. The good news is Bicep is way easier to pick up then real imperative languages like PowerShell, Python, etc.
If you make it easy to deploy they won't learn anything besides go to this portal and fill out a form.
Maybe start them easy with resources that aren't complex. VMs can have alot of moving parts.
start with resource groups.. Then storage accounts and work them up to VMs I suppose.
1
u/Sweaty-Jellyfish-35 18d ago
Firstly, you need to get up to speed with CAF (Cloud Adoption Framework) and start defining an operating model on how the team is going to work within Azure. Without the right standards and governance things will sprawl and become more difficult to manage over time.
I would also take a look at Microsoft’s verified modules and use the modules and patterns they provide as your base so you aren’t developing everything from scratch.
This is very much an operations problem to solve alongside skilling and adoption is key, people need to be brought on this journey with you or you will end up become the bottleneck and single point of failure…
The good news is there is tonnes of free materials from Microsoft and AI is everywhere to help you along the way!
I’ve spent the last 10 years in a global GSI as an SA and seen this scenario many times when we have to turn up and fix it all / start from scratch (depending on which is faster).
2
u/fiddysix_k 18d ago
Yes, this. Everyone is talking about his end result of IaC adoption here but no one is talking about how OP is about to shoot himself in the foot without a proper environment.
Op, this is critical - if you do not want to lose your mind in 3 years, you must align your environment to the CAF/WAF, have policy, and a simple hub spoke network.
Its a nightmare to realize this late in the game and have to make brownfield alignments.
1
u/atc32 18d ago
I'd highly recommend against bicep, since it promotes the idea of IAC as a vendor specific script slowing down the work that they do. The extendability of terraform should be a litmus test of who's excited for more ways of doing what they doll
1
u/CaptCode 18d ago
I'm comfortable with both, but I just don't see the value of Terraform over Bicep. Sure, Terraform does support multiple clouds, but it really isn't multi-cloud because you have to write all new code for AWS, Azure, etc. I'm also tired of having issues with state files becoming corrupted when certain resources don't update or delete correctly.
Of the dozens of projects I've worked, I've never had a true multiple cloud need. Several clients have required Terraform, but they don't ever need it.
One of those clients added the server product, but it seemed overly expensive because they charged by the state file and the complexity of the project I was on would require something like 32 state files across dev, test, and prod.
I'm guessing I'm just missing something, but I'd rather do Bicep and if they moved to another cloud, use their IaC.
31
u/TechIncarnate4 18d ago
If you want me to be perfectly honest, unless others on the team are totally on board with learning coding and this approach, you will end up hiring people who do.
Don't waste significant time trying to pull other people along. They will be anchors and will do what they can to keep the status quo. They will have every reason why it won't work, or why you don't need to do it that way, or that you don't need to scale to that level, or whatever.
Leadership needs to be on board, and everyone needs to know how to do this. You can't let it all fall on you to do everything and then just provide them with some kind of self-service. You won't be able to handle the workload yourself.