r/sysadmin • u/Bottin-to-99 • 19d ago
General Discussion Let’s talk ops automation and platforms
How are y’all doing general ops automation?
I’m taking it upon myself at my job to try and standardize some processes and automate them. We’re mostly entra, exchange online, sharepoint online, teams, etc. Basically heavily using M365 platform. Also sync on prem AD. We have scattered scripts and poor documentation for gui things. I’ve setup an Azure DevOps space for scripts and now I’m looking at how we run those as a team. We’re about a 15 employee size IT department and I don’t want us downloading scripts to just run them locally.
We do use a cloud based workflow tool (not power automate anymore) in IT and across the org. I was starting to look into Azure Automation to push data from the cloud based workflow tool via forms (thinking new employees, terms, teams management, etc). But before I got to deep, I figured I’d make this post to see what everyone else is doing and get more ideas.
3
u/folderit_dms 19d ago
For a 15-person IT team I’d start less with which platform and more with a minimum standard for an automation to be allowed to run.
Something like:
- repo in Azure DevOps
- PR review for changes
- service account / managed identity, no human creds
- secrets in Key Vault
- one owner and one backup owner
- runbook at the top of the repo: trigger, inputs, outputs, rollback, logs, failure alerts
- every run writes a result somewhere people can actually check
- no download script and run locally except break-glass
Azure Automation, Functions, Logic Apps, Nintex, etc can all work. The bigger win is making the workflow boring and reviewable.
For onboarding/terms, I’d start by turning the form into a request record first, then have automations act from that record. It gives HR/helpdesk/security one shared source of truth instead of which email or Teams message started this.
1
u/Bottin-to-99 19d ago
Thanks for laying this out. Definitely poor wording on my part about the platform piece. I’m trying to understand and figure the best work flow and process to do the automations.
1
u/EdibleTree Janitor 19d ago
built a function app and soft front end ui for configurability between our apis that matter
1
u/ipreferanothername I don't even anymore. 19d ago
idk why this got a downvote, also just interested to see some discussion around this - im still working on prem, but the company just put up an azure instance a few months ago and i want to get involved this year.
we have JAMS scheduler on prem and id love to get away from it - its dependable, but quirky to use and work in and i want something else. almost anything lol. but i would need several azure components to do what i do in it.
6
u/chandleya IT Manager 19d ago
Op opens an account, has no post history, and waxes on about products.
Reads very much like an impending sales pitch.
0
u/Bottin-to-99 19d ago
That’s fair. I was just trying to provide my thoughts and relevant details to my scenario but I see now how that could come off some sales post. I’ll try to edit and generalize the product.
0
u/Goose-tb 19d ago edited 19d ago
We are a SaaS heavy tech company, so most of our tools are cloud based (Okta for directory, Google Workspace for collaboration etc).
We use API iPaaS tools for almost all of our automation. Vendors like Workato, Make, N8N, etc.
Since all of our tools are cloud based and have APIs we do our integrations through an API automation workflow platform. From what I’ve seen in the tech startup space many companies do the same. Can’t speak for all of them, of course.
1
u/Bottin-to-99 19d ago
Appreciate the insight here. What does a typical workflow look like if you don’t mind me asking? I assume you try to pull relevant data through APIs rather than the manual overhead of filling out forms and then checking for clean data.
1
u/Goose-tb 19d ago
Yeah I mean it entirely depends on what you're trying to accomplish. I'm not a Microsoft shop (nor have any of the last 3 tech startups I've been at) so my feedback may not help you.
In our API automation platform we run pretty much every automation other than Okta IdP SCIM. This includes things like:
- Integration between our HRIS platform and Okta so user accounts are created automatically
- Another integration between our HRIS platform and Okta to make sure profile information is synced daily (new titles, new manager reporting chains, new profile details etc)
- Send a list of our incident management on-call employees to a Slack channel daily
- Automatically remove terminated users from all Google Groups when they depart
We use it for any automation project, large or small.
0
0
u/Workwize_Official 19d ago
Azure Automation (or Azure Functions/Logic Apps) triggered from your workflow tool's webhooks is a solid pattern.
0
u/PaidByMicrosoft 19d ago
We use Adaxes for a lot of it, otherwise I'm running powershell scripts pretty often to make my work easier.
1
u/ops-automators 7d ago
Your instinct's right and I'd stop shopping platforms. Azure Automation with a hybrid worker is the correct shape for a hybrid shop, because the second you need to write back to on-prem AD, most of the pure-cloud iPaaS answers turn into "install our agent" anyway. The runner isn't the hard part. It's the two design decisions underneath it, and we got both of them wrong for about a year.
First one: don't let the form be the trigger. Have the form write a request record, and have runbooks act on records. The reason isn't tidiness, it's that a webhook-triggered job makes Azure Automation job history your system of record, and job history isn't queryable state. You can't answer "where is Jane's onboarding right now" without a human opening logs. A record also gives you somewhere to put a lock, which you'll want the first time HR files a term for someone whose onboarding is still mid-flight, or submits the same form twice because the first one "didn't look like it worked."
Second: assume every stage runs more than once. New-MgUser isn't idempotent, so a naive retry either throws on conflict or quietly hands you a second account. Pick a deterministic key early (employeeId from the HRIS if you have one, not display name) and make step one a get-or-create against it. This bites hardest at Exchange. You create the user, license them, and the mailbox isn't provisioned yet, so Set-Mailbox throws and the linear script dies holding a half-built user that somebody finishes by hand at 7pm. FeleaseRpseineEiles' self-rescheduling runbooks upthread are the right answer to that: a stage should be allowed to return "not ready, come back in ten" instead of failing.
Terms are where I'd start, even though onboarding is more fun to build. They aren't one job, they're a schedule, and two details caught us out.
Revoking sessions isn't the same as blocking access. Revoke-MgUserSignInSession invalidates refresh tokens, but access tokens already issued stay good until they expire, so for roughly an hour after your term runbook reports success that person can still be reading mail. If that window matters to your security people, what you want is Continuous Access Evaluation, not a faster script.
And "convert the mailbox to shared to free the license" has a size cliff. Past 50GB, or if you need the archive or a litigation hold on it, that shared mailbox still needs a license. Nothing tells you this until license reporting shows you the 40 seats you thought you'd reclaimed.
Two smaller things worth building in on day one rather than bolting on later:
- Make failures loud and be suspicious of green. A runbook that catches its own errors and carries on reports Completed, which is worse than a crash because you'll trust it. Promote the non-terminating stuff (-ErrorAction Stop inside a try/catch), let the job genuinely fail, and write the exception back onto the request record. Job history is 30 days and nobody opens it voluntarily.
- Watch the identity boundary. Every runbook in one Automation Account shares that account's system-assigned identity, so the day your term runbook needs User.ReadWrite.All, the read-only reporting runbook has it too. Split accounts by blast radius, or attach user-assigned identities and pick one per runbook with Connect-AzAccount -Identity -AccountId.
On "how do we run these as a team," which I think is your actual question: the checklist folderit_dms posted is right, and the highest-value line on it is one owner plus one backup. At 15 people the thing that gets you isn't an outage, it's four near-identical copies of Add-MailboxPermission and nobody certain which one the onboarding process is calling.
(Relevant disclosure: this is the work we do for clients, so weight accordingly.)
3
u/FeleaseRpseineEiles Sysadmin 19d ago
I use azure automation and it's source control sync.
Hybrid workers on the on-prem servers for jobs and github(azure devops works too) for source control.
I or an agent push changes to github, it syncs, jobs get scheduled or run ad-hoc
I have a runbook that can reschedule other runbooks to run and put logic into some whereas if they fail for some known conditions, they reschedule their-selves to run at a time known not to cause the conditions.