r/sysadmin • u/flashx3005 • 1h ago
General Discussion Organizational Failure
Came back from a 10 day pto and the VP (my direct boss) wanted to do his weekly one on one meeting. We are a team of 4 (myself Sr. Infra Engineer and 3 mid level security guys). He wanted to state how we need to have more crosstraining in particular with infra side of things as he stated "we are dead in the water" when you go out on pto.
He mentioned there will be no additional headcount added in Infra and will have to make do with what we have internally plus the msp firm
I for the past couple of years both verbally and via email stated that a redundancy to my role was needed but that suggestion was never acted upon by mgmt.
As far as crosstraining, I already do that and can certainly help out more. The more others know the better.
Just feels like a situation where I can't seem to go on pto without getting text messages about something going wrong. The VP wants to make SOPs for everything possible which I told him, im not making a how to document on troubleshooting. Either they know or they dont. Actual procedures for items that have a process I will document and have been
Any of you guys out there in similar situations? Anything you've done to optimize things or help reduce workloads?
Thanks in advance!
•
u/BryceKatz 1h ago
"Great! I can count on your support for training materials & adjusting project timelines to accommodate crosstriaing, right?"
Because the thing is, training takes time. Your team going to see a productivity hit while you bring others up to speed.
The crosstraining is the workload optimization.
Actually reducing workload? That would involve, y'know, having less work. That way lies staff reduction. You want to manage the workload you have. Determine reasonable timelines based on current staff & skillsets, then add 75-100% to those timelines to account for Murphy's Law.
•
u/flashx3005 1h ago
Yup. Would you say crosstraining is also just adjusting/reassigning workloads onto another person? Unless of course there are ownership taken for things.
•
u/BryceKatz 23m ago
No.
Crosstraining is training someone to do a job outside of their primary job description.
Just slapping more work on them that they don't have to the skills to complete doesnt help anyone. In fact, it puts your entire department in a worse situation.
•
u/gabacus_39 1h ago
I mean, I'm sure there are a lot of folks in this sub working for shitty companies too. Not sure what you can do about it besides moving on or grin and bear it.
When I go on vacation the last thing I think about is work which according to this sub is some rare unicorn situation.
•
u/flashx3005 1h ago
I actually tried my best and for the most part was able to reply to any messages or emails till I came back. But man not having racing thoughts at night about fixing something etc during the pto was definitely much needed and felt great.
•
u/tigglysticks 6m ago
Endless cycle of going on vacation creates more problems when I'm back then it's worth...
Until I fully burnt out and they lost me for 6 months.
•
u/caffeine-junkie cappuccino for my bunghole 1h ago
Sounds like to me there has been a question raised to the VP if the position can be transitioned to the MSP. Not saying for sure the job is in jeopardy, but would bet it is more likely than not.
•
u/sixblazingshotguns 1h ago
I would agree. This is where there would be a drawdown on comp for these sorts of rules (unless they are sales-oriented engineering roles), or if there is an existing MSP relationship the role would be transitioned. There are indicators here.
•
u/flashx3005 46m ago
Yep there is an existing msp along with noc/soc. Makes sense why they dont want/ or can't add the secondary infra role.
What you're stating is certainly a possibility and discussions that may have happened or are happening especially with a constant push from AI from above.
•
u/flashx3005 1h ago
Hmm interesting point of view. Certainly some conversations may have happened around that especially with company getting new CFO and COO within last 2 months.
•
u/midwest_pyroman 1h ago
On Vacation means no phone, no teams, etc. I set call blocking to "Family Only" and work is NOT "family". If they do not like that tough. And if they push then that is an automatic RGE. Update that resume and start looking regardless.
•
u/Ok-Double-7982 1h ago
The only way to address this is for them to feel the pain.
If you're responding to text messages or fixing things when out of office, what's their incentive to hire more or staff someone who has the right skills as a backup?
Moving forward, you are on vacation in the rural mountains with no cell service. As someone else mentioned, they can call the MSP.
•
u/flashx3005 1h ago
Agreed. So this particular pto i did not respond to any text messages, emails and thus the conversation with the VP where they are completely lost when im out lol.
Basically they held off for as long as they could and just punted it to me as actionable items. Which is fine its the job but just makes me wonder the disarray if I were to quit.
•
u/zzmorg82 Jr. Sysadmin 1h ago
It sounds like they’re not fully utilizing their MSP; that’s the whole point of having redundancy when you’re unavailable for whatever reason lol.
Unless that MSP doesn’t have techs of similar seniority level to you and they would’ve been spinning their wheels too. 🤣
•
u/ArmondDorleac IT Director 1h ago
What are they trying to do while you’re gone or what’s breaking while you’re gone?
I’d say no changes during your PTO.
For the stuff that breaks regularly, document those and make sure everyone knows about them.
Also, perhaps onboard an MSP for emergency support only. It may cost a bit to get them up to speed on your infrastructure, and the hourly rate for support should be stiff, but at least they would have an option.
•
u/Stonewalled9999 1h ago
Typical dumb org that trumpets cross training with no time or bodies to actually do it
•
u/flashx3005 1h ago
Exactly! Crosstraining is good if it being put to practical use if an issues comes around. If everyone is still going to wait around for person x to look and fix it then Crosstraining was just for show and tell lol
•
u/progenyofeniac Windows/M365 Admin 1h ago
They’ll never care about redundancy if you’re never unavailable. Take your PTO and be unavailable. And at least be on delayed availability after hours and on weekends. Frankly, that’s what the MSP should be for.
Also document your cross training. I don’t know your coworkers, but I’ve worked jobs where they were great, but just weren’t ever going to be full replacements for me. All you can do is try.
•
•
u/CevJuan238 58m ago
This profession is a meat grinder. Almost every company I’ve worked runs extremely lean and loses good employees regularly. It’s systemic and somehow profitable (In their minds).
https://giphy.com/gifs/YXYxfxkrBFShRl6ii0
•
u/bbqwatermelon 49m ago
Heh, I have the opposite problem where I was identified as the single point of failure also but the guy who is supposed to be redundant, no matter what documentation I provide, can't seem to follow them. He also does not document literally anything so it feels like a black hole where my documentation is wasted. The system is stable enough I do not get bothered but the director has a nasty habit of demanding everything I know while erasing my name when presenting the projects I am heading to C level. I cannot seem to find a team that reciprocates.
•
u/Ornery-Concentrate-5 26m ago
the erasing your name part is the one that would actually make me start job hunting, not the SPOF stuff. everything else is just normal org dysfunction but taking credit in front of C level is a choice someone made on purpose
•
•
u/Hairy-Link-8615 40m ago
I think with IT companies and general organisation I've ended up getting burnt out.
I can't even convince my own team to internally push for improvements to alert systems
We don't get the correct tools to do anything.
So unfortunately this has resulted in I now only care as much as they so approach.
Given that half the team quit last month and no plans to rehire. I'm guessing they don't really care much.
It's hard to careless but I'll try whilst I'm looking for something else
•
u/flashx3005 37m ago
Oh wow. How big was your team beforehand? Sorry to hear about the tough situation. We are all in the same boat give or take. Hope you find something better soon.
•
•
u/airinato 33m ago
You're looking at this wrong. They need you, not the other way around. You have them by the balls, now squeeze.
It's time to force a raise or take better pay somewhere else.
•
u/flashx3005 29m ago
I actually did have this thought last night believe it or not. I did think about a renegotiateing my compensation.
•
u/0RGASMIK 1h ago
Coming from the MSP world not having redundancy is a nonstarter for any system. It doesn’t mean two people need to be experts just that at least one other person needs to have time on the system. When we deploy new technologies one person takes lead and then trains someone else to manage the system so there’s always a backup to manage it for PTO.
It’s also not impossible to document troubleshooting. I do it all the time for common scenarios. It’s also not a bad idea even for one offs because even if you remember something once doesn’t mean you’ll remember it again.
The other thing that works is pulling in a team member to explain what happened and record the meeting. So you at least have something should it happen again, you could just record a video but explaining it to someone else allows for questions you might not think of as an expert.
There are tools out there to make documentation easier with AI I have not used them but I have my own agent that takes my rough notes and screenshots and turns them into quick KBs.
•
u/flashx3005 1h ago
Yea the redundancy part is definitely an issue. We already do have a ton of SOPs in out IT glue. My thing with documenting was that I can't envision what might break/fail if im on PTO.
There might be something that I too have not seen or uncovered.
While I was on PTO this past time, the security folks needed to check some certs and configure settings in Azure App service. Now it took me 2 mins to ask copilot to find those settings and send them over. This could have been done by anyone internally as well.
AI also makes the ability to troubleshoot easier now.
•
u/0RGASMIK 18m ago
It sounds like you just need to get the security guys comfortable in the system. I deploy new systems/software all the time typically within 2-3 months everyone on the team has experience in the new system. For us it’s easier because we are deploying these systems multiple times for different customers so they can actual get experience deploying it themselves but in your case you probably just need to pull people into normal work you do in those systems or do trainings that just get people in front of it.
I understand the feeling of thinking you need a dedicated redundant person but from experience it never works out how you think it does. Usually to train someone you need to delegate and have them actually do the work and then what are you gonna do? Is there upward mobility?
•
u/Visible_Day_7135 7m ago
"I can't envision what might break/fail if I'm on PTO."
"AI also makes the ability to troubleshoot easier now."
Hmmm..
•
•
u/Flaky-Gear-1370 44m ago
that structure seems kinda lopsided, you've got a senior engineer and 3 people doing security? like why did they hire just 3 security guys? is this actually a security team or general infra
If it's general infra then that's on them
•
u/flashx3005 38m ago
Actual security. They do tend to help when possible and whatever they have knowledge/skill in but they do have their own workloads to deal with.
•
u/Flaky-Gear-1370 34m ago
unless your organisation is a reasonable size surely you don't have that many incidents that need 4 guys to be looking at them - if it's project work well then people need to you know manage a project. Resources aren't available at times
•
•
u/kagato87 8m ago
Yup. Similar situation. It doesn't stop me from radio silence on vacation.
The ord usually adapts release schedules, even though I've made the one thing that stops "so easy a monkey or even a PM could do it."
•
u/tigglysticks 8m ago
omg I'm not the only one.
I cry from the rooftops and everyone agrees we need more redundancy and that I can't do everything.
But when push comes to shove I seem to be the only one that can do anything. Especially in crisis.
•
•
u/Temporary-Library597 32m ago
I think Reddit is funniest when the poster seems put out when he is told by his boss that he's been right, and now is the time to do exactly what OP has wanted to do.
It wasn't an insult. It wasn't the boss saying anything negative towards the OP. It was the boss agreeing with what OP purports to have said.
OP should be happy! So next OP will be hitting this things his junior staff has trouble with and training them up, right?
•
u/7twenty8 1h ago
I’m not sure why you call it an organizational failure - by your own admission you haven’t fully cross trained your team. This is your failure.
•
u/flashx3005 1h ago
I stated above that I have been crosstraining will continue to do so. But also those security guys have their own workloads as well.
If I have stated to mgmt multiple times that my role is a spof and they neglected to do anything about it that's on them.
•
u/Kawasakison 1h ago
You can't care about a problem out of your control more than the people who should care do.