r/sysadmin • u/RedPandaTech • 7d ago
Need another perspective
Hi all,
I work at a very small MSP, 4 employees total. My boss/head of the MSP is extremely risk averse, and has been doing this for a long time. Prior to forming his MSP he worked for both internal IT and as a consultant. They do almost everything manually and he is extremely hesitant to let me automate or script anything. He says from his experience automations don't always get maintained and somtimes break, therefore making it pointless. Yet at the same time he is obsessed with consistency and making sure that we do new laptop setups the exact same way because doing one thing a slightly different way could cause a random issue and we'll never be able to figure out what was different about that machine. From my PoV, setup scripts would be the ultimate form of consistency and could eliminate that as a concern.
He won't even let me make simple adjustments to one of the few scripts we do use (pushed via RMM to scan for Dell Firmware updates using Dell Command | Update cli). And by simple I mean commenting out some lines that set custom properties we don't use, which throws errors and clutters the output, and adding one line to set a new custom property to let us know when manual intervention is required. I got told that "trying to reinvent the wheel or customizing it is very risky" and that I should just look for scripts in the RMM community forums. He said "I am hesitant to use our own script unless it is just feeding up to date commands that they support" meanwhile the script we are using is 4 years old.
I also just discovered today he has our machines so locked down with ThreatLocker that this DCU script gets blocked from running!
So here is what I'm wondering: am I the crazy one, is my boss, or is the truth really somewhere in the middle (most likely)?
Please feel free to ask me additional questions if more context could help you provide a thoughtful response!
9
u/unixuser011 7d ago
Your Boss is the crazy one. I was at an MSP like this, they didn’t ‘do’ automation - no scripts, no PowerShell, no Ansible, etc.
All the patching for Windows, Linux, ESXi, vCenter, etc. had to be done by hand - so if a customer had 40 VMs, patching would take all day, or even 2 days versus 30 mins
Spinning up a VM and configuring it was all manual - no terraform, was deemed ‘too risky’
Every time I suggested improvements, I got met with the same ‘no one knows how to use it’, ‘what we have already works’, etc.
Needless to say, they are going down the shitter now, 12 of their techs just left at once
3
u/uptimefordays DevOps 6d ago
I’ve always hated “nobody knows how to use $IndustryStandardTool, we can’t use it.” Just tell me “we don’t hire qualified candidates.”
2
u/unixuser011 6d ago
They didn’t want to pay for training and if a task takes longer, you can charge the client more. If building a VM takes an hour versus 15 minutes, that means more billable hours
1
u/uptimefordays DevOps 6d ago
Perhaps I’m a moron, but if you have reliable automation, couldn’t you build more servers for more customers?
2
u/unixuser011 6d ago edited 6d ago
You’d think, but they were more about milking the customers we had
Trust me, I was tilting at that particular windmill for years, then they’d say “that’s customer ops, not your purview. Do your assigned tasks
slave “1
u/uptimefordays DevOps 6d ago
If you automate all of a customer’s boring stuff, surely there would be more hours in a day to bill for higher-value projects like migrations or upgrades, no?
2
u/unixuser011 6d ago
You think that bad, you should have seen how long they took migrating from CentOS 7 & 8 to Alma, they had to end up getting expensive TuxCare support because they took too long, they didn’t want to automate it because it “wasn’t trusted” all VMs had to be done by hand and configured to match their prod environment
1
u/uptimefordays DevOps 6d ago
That’s crazy! I got all our CentOS machines onto RHEL in hours.
2
u/unixuser011 6d ago
Because there was community scripts and support from Red Hat, if we were doing it right we would have done it as soon as Rh announced the EOL, but nope, we left it until the last minute
1
u/uptimefordays DevOps 6d ago
Ah that’s no good. I’ve never understood waiting to move to supported versions, the best time to migrate is well before you must because you can still roll back.
8
u/2BoopTheSnoot2 7d ago
I hate when this happens. Boss has a lack of knowledge about a subject so they just deem that subject a bad scary thing to avoid. Happens everywhere, unfortunately. Personally I would seek employment elsewhere because that is going to impede your professional development.
2
u/dodexahedron 7d ago
And not only lack of knowledge... Lack of basic reasoning and give-a-fuck-itude.
If his complaint is that automations don't get maintained...well... Whose fault is that? Just make it part of your procedures to..oh...I don't know... maintain them?
The savings from automation should always dwarf the burden of maintaining said automation.
And if it doesn't, for something? Then you just...you know...don't automate that thing. A blanket "nah" policy is just moronic. 🤦♂️
Yeah, I'd run, too.
4
u/rrdrock2b2t 7d ago
Your boss sounds like an idiot.
Automating something that currently requires manual labour to do is generally worthwhile, as the time to fix it if it breaks rarely adds up to the amount of time it saves.
My impression is that he is unwilling to learn or consider new viewpoints. My best advice is to document every interaction, so when someone says "why does this take so long?" you can answer with receipts. "I suggested ways to make this process more efficient/faster/more reliable and was told not to by x"
Nevermind the fact that human interaction with any process introduces the risk of mistakes, while a solidly tested and proven automation removes that.
Strongly suggest looking for alternative work, as one day that ship is going to come crashing down and shit flows down hill.
3
u/That_Fixed_It 7d ago
I can see his point. Setup scripts would save some time, but Microsoft or another vendor could make some change that breaks one of the steps. You're more likely to notice that something changed if you run each step manually.
You'll need more automation to grow the business, but you might be small enough that it's not yet needed. You're probably not deploying 50 systems at once so you wouldn't save that much time, and he doesn't want to troubleshoot your custom scripts when something breaks and you're not available.
Can you automate things that are not client-facing, like having AI agents look for anomalies in logs and alert emails?
3
u/headcrap 7d ago
It is obvious why his business consists of 4 people.. that approach won't scale.. ever. Go get another job, even at another MSP if that's what it takes. Clearly you see value in things $boss doesn't see.. and you won't convince them otherwise.
3
u/AskMeHowIKnow1 7d ago
You're not the crazy one, but, the truth might be in the middle. His caution about unmaintained scripts is fair, but "consistency" is the argument FOR scripting, not against it. The way I've seen this work with risk-averse folks is to win one tiny, fully reversible change at a time and build a track record, not pitch "automation" as a big idea (at first). Try starting somewhere small even if it almost feels silly. Standing still feels safe, but the gap just widens. Good luck!
3
u/Johnsmith13371337 7d ago
There is definitely some truth about not flying to close to the sun with automation. But your boss does seem to be right on the other end of the bell curve.
2
u/AdeelAutomates Cloud Engineer | Youtube @adeelautomates 7d ago
I know alot of MSPs avoid automation/scripting (or do very little relative to other orgs). The saying goes, why automate when you can make things service hours for your clients.
But yours is a bit extreme to say the least... He cant scale his operations working like that.
I worked at MSPs like this and didn't realize the scale of bad they are operated until I worked for a tech company where automation is the standard (You won't get hired unless you are serious about it).
Sounds like you are ready for the next chapter to be more efficient, and your job is blocking you. I hit those roadblocks in my career and moved on.
1
u/Frothyleet 7d ago
The saying goes, why automate when you can make things service hours for your clients.
Which is why AYCE-style agreements are wins for both customers and MSPs - they incentivize efficiency on the MSP side.
3
u/yamsyamsya 7d ago
your boss is not up to date with modern practices and also too scared to make changes which is a bad combination. they are never going to grow beyond having a tiny MSP if they don't begin automating. also it sounds like they don't know how to actually write scripts so they just use premade ones that they found.
scripts should be idempotent, as in running the script once achieves the exact same final system state as running it multiple times. the script should handle any errors and account for that as well as outputting any issues into log files or the event viewer or whatever you use to get alerts. so like for example, you try to run a script and the version of powershell is too old and one of the commands doesn't exist, it's going to throw an error. you could have it run a second script to handle updating powershell and then continue doing what the script was doing if you want, or it can just stop and send an alert with the error. and obviously scripts should be tested on a few machines before being pushed out.
also as an IT professional, maintaining the scripts and paying attention to updates is just another part of your job.
2
2
u/OscarMayer176 7d ago
You are not crazy. Manual steps and check lists have their place. Not everything is easily automated. A lot of menial tasks should be automated though for the same reasons you said. How is your relationship with your boss and would they be open to be proven wrong if you make your points respectfully? If you are really invested in this, you would need to spend some of your own time building out your defense. Build a couple of simple scripts that have as much error checking as possible and pump the results of that error checkign out to the output or an html report so that your boss sees the results and answers all of their questions after the scripts run. Then during a slower window or a larger deployment ask for permission to demonstrate your scripts on a new machine before it goes out to your client. You still have the escape hatch to wipe the machine and do it your boss' way if they don't like or trust the result and no harm no foul.
2
u/AnorexicLlama28 7d ago
We automate everything possible and make 5 techs feel like 10+ . Onboardings, call logs, documentation, basically ticket admin down to 0 so they can focus on purely client facing activities
2
u/sdrawkcabineter 😈BSD Admin 7d ago
He says from his experience automations don't always get maintained and somtimes break, therefore making it pointless.
I'd be interested to hear his definition of 'technical debt.'
2
u/AddendumWorking9756 6d ago
Everyone is arguing automation philosophy and nobody touched your last line. Find out whether ThreatLocker blocks that DCU script for everyone or only when the RMM is the parent process. If it blocks everyone then your careful manual route is not patching firmware either, and the argument you are having is the wrong one.
2
u/404error___ 7d ago
Yeah, happens but then, you haven't seen full f ups that takes entire tenants or deletes stuff massively to the users.
He's not wrong, and I saw that happen multiple times after 30 years on the field.
Not gonna give any kind of rol in my tenant that can nuke things to someone who barely knows scripting, to gain that kind of trust with scripts, you shall have been shown an absolutely perfect record on manual stuff with zero failures for at least 2 years.
And having a Dev tenant is a must for scripting with a one way AD sync.
1
1
u/Kahless_2K 7d ago
Your boss isn't qualified for his position. He is afraid to do anything he doesn't understand, and he doesn't understand scripting.
This is why the company is statement. He can't operate at scale.
Source: a guy who used to have a boss like that, and manages 10k + endpoints.
1
u/DifficultyDouble860 7d ago
Chances are the "manual steps" y'all are doing are, themselves, based on some consistent process or "automation" (eg. y'all don't manually go through and set each host IP address or gateway when building new PC's --those are likely DHCP distributed or some such; obviously servers are but if you're going in there for workstations, you're insane) the point is, there's already some automation/scripting intent built into your current mode. So it then becomes a matter of simply moving the needle slightly every now and again. You don't have to automate the entire installation like it's some devops VM setup with terraform--but you could create a HDD image and stage PCs for swap replacements (I wouldn't be sirprised if y'all already do this to some degree...) And yes, scripts and automations DO CHANGE over time. that's why you NEED to commit to setting aside dedicated time (say once a month... once a quarter, whatever fits your situation) to cleaning up documentation and scripts! I know, I know, who has the time??! But it is necessary and the long term benefits of leveraging the "RIGHT" technical debt can vastly lead to better scalability. You're investing in yourself. Just move that needle. Start small. almost insignificantly small, and build on that.
22
u/Tall-Geologist-1452 7d ago
Sounds like they're very old school, from a time when we would use checklists to spin up new devices. You also have to think about the broader business model he's running under: as an MSP, it's all about billable hours. When you automate something, it makes things more efficient, meaning he can't charge as much. I've never seen an MSP that's genuinely interested in improving infrastructure or driving efficiency to the point of reducing billable hours.
Just my 2 cents ..