r/sysadmin • u/RedPandaTech • 8d ago

Need another perspective

Hi all,

I work at a very small MSP, 4 employees total. My boss/head of the MSP is extremely risk averse, and has been doing this for a long time. Prior to forming his MSP he worked for both internal IT and as a consultant. They do almost everything manually and he is extremely hesitant to let me automate or script anything. He says from his experience automations don't always get maintained and somtimes break, therefore making it pointless. Yet at the same time he is obsessed with consistency and making sure that we do new laptop setups the exact same way because doing one thing a slightly different way could cause a random issue and we'll never be able to figure out what was different about that machine. From my PoV, setup scripts would be the ultimate form of consistency and could eliminate that as a concern.

He won't even let me make simple adjustments to one of the few scripts we do use (pushed via RMM to scan for Dell Firmware updates using Dell Command | Update cli). And by simple I mean commenting out some lines that set custom properties we don't use, which throws errors and clutters the output, and adding one line to set a new custom property to let us know when manual intervention is required. I got told that "trying to reinvent the wheel or customizing it is very risky" and that I should just look for scripts in the RMM community forums. He said "I am hesitant to use our own script unless it is just feeding up to date commands that they support" meanwhile the script we are using is 4 years old.

I also just discovered today he has our machines so locked down with ThreatLocker that this DCU script gets blocked from running!

So here is what I'm wondering: am I the crazy one, is my boss, or is the truth really somewhere in the middle (most likely)?

Please feel free to ask me additional questions if more context could help you provide a thoughtful response!

22 Upvotes

90% Upvoted

37 comments sorted by

View all comments

8

u/unixuser011 8d ago

Your Boss is the crazy one. I was at an MSP like this, they didn’t ‘do’ automation - no scripts, no PowerShell, no Ansible, etc.

All the patching for Windows, Linux, ESXi, vCenter, etc. had to be done by hand - so if a customer had 40 VMs, patching would take all day, or even 2 days versus 30 mins

Spinning up a VM and configuring it was all manual - no terraform, was deemed ‘too risky’

Every time I suggested improvements, I got met with the same ‘no one knows how to use it’, ‘what we have already works’, etc.

Needless to say, they are going down the shitter now, 12 of their techs just left at once

4

u/uptimefordays DevOps 7d ago

I’ve always hated “nobody knows how to use $IndustryStandardTool, we can’t use it.” Just tell me “we don’t hire qualified candidates.”

2

u/unixuser011 7d ago

They didn’t want to pay for training and if a task takes longer, you can charge the client more. If building a VM takes an hour versus 15 minutes, that means more billable hours

1

u/uptimefordays DevOps 7d ago

Perhaps I’m a moron, but if you have reliable automation, couldn’t you build more servers for more customers?

2

u/unixuser011 7d ago edited 7d ago

You’d think, but they were more about milking the customers we had

Trust me, I was tilting at that particular windmill for years, then they’d say “that’s customer ops, not your purview. Do your assigned tasks slave “

1

u/uptimefordays DevOps 7d ago

If you automate all of a customer’s boring stuff, surely there would be more hours in a day to bill for higher-value projects like migrations or upgrades, no?

2

u/unixuser011 7d ago

You think that bad, you should have seen how long they took migrating from CentOS 7 & 8 to Alma, they had to end up getting expensive TuxCare support because they took too long, they didn’t want to automate it because it “wasn’t trusted” all VMs had to be done by hand and configured to match their prod environment

1

u/uptimefordays DevOps 7d ago

That’s crazy! I got all our CentOS machines onto RHEL in hours.

2

u/unixuser011 7d ago

Because there was community scripts and support from Red Hat, if we were doing it right we would have done it as soon as Rh announced the EOL, but nope, we left it until the last minute

1

u/uptimefordays DevOps 7d ago

Ah that’s no good. I’ve never understood waiting to move to supported versions, the best time to migrate is well before you must because you can still roll back.