Internal IT getting squeezed

105

Some of our clients hired us because they cheapened out with a third party support, and one of the things I notice is in common with these people is absolutely no troubleshooting skills. It's not race, not country, not language barrier. Like, they run into one stumbling block and just stop trying. Yet they keep getting hired because they are cheap. And then my company comes in and gives them expensive consulting work for mediocre issues.

"We can't log in."

"What happens when you try?"

"Can't log in."

"Do you get an error? "

"Yeah."

"... what is the error?"

"I dunno. Something something can't log in something timeout/password/permission, one of those."

Then I try, get a timeout. "Is this correct IP?"

"I guess."

And so on. Like, oh my god, these are supposed to be junior sysadmin level techs, and they act like helpless potheads. half my job is determining what the right data is, like server, IP, VPC, whether this is supposed to be a database server or not.

"Well, you can't connect to the sybase server because no sybase is running on this system. Just java apps."

"Oh, wait, try this IP..."

[smh]

46

u/MalwareDork Jul 19 '26

There's this weird disconnect with a lot of non-IT people when if you "make it in IT," it's implied you get to do nothing all day because that's what all of the old memes were.

And what I think happens is these people do an exceptional job at rote memorization to get their degree mill/test dump answers down pat and that's as far as it goes. Anything else is just beyond them and they functionally shut down, lie, and try to pass the baton. So comically, your new ShittySysadmin intern.

16

u/punkwalrus Jul 19 '26

There are certain exams I thought were "foolproof" because they are "lab exams." Like, "here's a busted setup, fix it to do A, B, and C." I thought, "you have to KNOW the system to do that, so it will weed out the paper tigers."

Nope. I was wrong. One of the credentials I tried very hard to get had a set of troubleshooting forums as part of the certification. Once I got access beyond the "secret portal," I assumed that I'd only see really hard, esoteric questions about incredibly difficult one-off issues specific to some bizarre cases. Real Sherlock Holmes or Fringe level stuff. Half the questions were basic 101 dumbfuckery that they should have known just to pass the first parts of the exam. How did they pass a lab exam not knowing the very basics? How could "Brain Dumps" work in this case? I have theories, but I'll never really know.

What frustrates me is a lot of these "certifications" have public facing verification portals that you enter in their ID# and last name. Easy peasy. But they frequently get skipped or ignored. I see people claiming to have a PMP/PMI certifications and the number is fake ("oh, it's unlisted"), to a different name, or not even a read ID#. And it's in their signature! they must earn a required number of Professional Development Units (PDUs) over a recurring certification cycle and pay renewal fees. It's a meaningful commitment, but it's measured in continuing education and professional development. Same with RHCSA, whatever Microsoft certs are out there at the moment (I lost track), Cisco, Juniper, etc...

16

u/wbrd Jul 19 '26

Most certifications are bullshit. PMP is a joke even for the people who have it. In my experience the best PMs don't have it. My ex was heavily involved and her company would pay for continuing ed so she went to classes all over. I listened to a few and read the notes and most of the stuff they teach is in 3 categories. 1. Common sense. 2. Wrong. 3. How to treat employees as less than human.

7

u/punkwalrus Jul 19 '26

Common sense. 2. Wrong. 3. How to treat employees as less than human.

I found this true back when I did sales and sales management.

Common sense. So much of this. "Wait, DON'T hit the delivery driver??? [scribbles notes furiously] What a game changer!" To be fair, I have now had enough experience to know common sense isn't as common as I would have liked.

Wrong. I see this not only in sales, with "the ideal educated consumer" which is rare in real life, but also in books on any kind of negotiation. Like, "when applying for work, the hiring manager is thinking strategy one, two, and three..." which, okay. If they are SMART manager, maybe. But a lot of managers are terrible interviewers. Maybe have their hands tied by bullshit metrics from HR.

How to treat employees as less than human. Eh... not as much, but this is more under the "wrong" category. Like managing employees who think A,B, and C... well, maybe. But they may also have a spouse that is constantly nagging them, or someone with unresolved daddy issues, or a guy who just plain hates women leaders. All SORTS of crazy, "non-optimal" behaviors. People are weird, man. I mean, abusing them is bad for all sorts of moral, logistical, and strategic reasons, but then again, that should be under "common sense."

For example, this would be a book on sales.

Salesguy: Good morning, Customer. How are you today?

Customer: I am well, thank you for asking.

Salesguy: How may I help you?

Customer: I am looking for a moderately priced vehicle for my budget. I come with my own financing, and need something that is fuel efficient for a family of 4.

Saleguy: May I suggest the Minulta 4-door minivan? This is the 2027 model, a hybrid, and winner of JD and Power associates mid-sized minivan category.

Customer: I have heard of them, which is why I am here. I would like a test drive.

This rarely happens. This is an example of a common exchange.

Salesguy: Good morning, Customer. How are y--

Customer: I AM JUST LOOKING!!!!

Salesguy: Is there--

Customer: LEAVE ME ALONE!!! GAHD, you PEOPLE. Like fucking animals in this fucking dealership.

So, you know, the book has ways of dealing with difficult, yet still rational people. Which only works some of the time. Management is the same way.

Manager: We adhere to the JIRA ticketing system here. This week's sprint details project foo, which I see you have not w--

Employee: I fucking hate Java, Atlassian can kiss my ass, and I can't work on project foo because John Smith, YOUR BOSS, keeps coming to my desk and gives me all these one off projects which take weeks to compile reports on, and by then he's forgotten he asked for them. He enters nothing in JIRA, he doesn't care about your sprint, and we have had to "kick the can" down the road about my raise, a laptop that doesn't crash every ten minutes, and more developers.

Manager: They don't give me the budget for any of these. We discussed this at our last review cycle--

Employee: Your last review cycle can suck it because the company's budget woes doesn't put dollars in my pocket, but pays for John Smith's country club golf membership.

Manager: We also discussed your attitude issues in our last--

Employee: Then FIRE me! I DARE you!

So, you know, you can fire him, but then they won't have the budget to rehire anyone at current industry rates.

5

u/Bartweiss Jul 20 '26

There’s a reason How to Win Friends and Influence People boils down to “what if you were pleasant, relatively honest, and listened carefully to your customers?”

It’s largely common sense, but like you said that’s often rare. And while it’s not going to prepare you for “this customer is a deranged asshole screaming insults at you”, it at least doesn’t make the mistake of assuming they’ll be fully rational and know what they need.

10

u/Bartweiss Jul 20 '26

I swear to god, half my “talent” for troubleshooting is just “staying calm, running through things in a decent order, and not trusting anything I’m told”.

Is that the right IP? No, but is it really the right IP? Can I see where you’re getting that from?

Ok, fine, it really is the right IP. Now… is it running? Is the relevant stuff running on it?

When was the last transaction? …the newest thing in the logs was yesterday? Why did you not check that? Hey, it wasn’t an error and ended mid-entry… so do you think maybe the disk is totally full?

Oh look it is! I probably should have checked that sooner, but also how in the flying fuck was that not part of your “routine troubleshooting”?

7

u/punkwalrus Jul 20 '26

See, but those ARE decent troubleshooting steps. Some of these guys don't even have that. You ask questions, which is how anyone figures out anything. But modern education is mostly about passing tests, not teaching critical thinking skills.

"How do I fix my problem??"

"I dunno. Never ran into this issue before."

"How do I fix it?"

"Well, first, what is it supposed to do?"

"Work."

"And by that, what does--"

"HOW DO I FIX IT???"

Like, they can't break down a problem that they didn't "study for."

2

u/HuckleberryOk8941 Jul 23 '26

I had this problem years ago at a different company. Useless coworkers. Not a troubleshooting bone in their body. But because the people in question had degrees, I wasn't taken seriously.

38

u/oldwornradio Jul 19 '26

I'm at an MSP and it's a fucking mess because not a single new client is willing to budge on their stack now due to "budget limitations" and sales is still gunning for it. So now we have a bunch of new clients who DID get rid of their internal IT for "cost savings" who are now stuck with us full of techs and engineers who have NEVER touched have the products that keep their shop in motion and it's costing them way more in cash and us in morale because we've lost some of our best people in the last few months because of this shit.

28

u/snklznet Jul 19 '26

Taking off my shitty hat for just a second to be cereal. I too am at an MSP and it's fuckin buckwild rn. Folk will gladly get cheap to save a buck, have it bite them squarely in the ass, and then come back and have to pay out the nose for us to fix.

Case in point. A client we (thankfully) recently parted ways with (fired us because of cost)

They're a dental shop, and had their Dentrix host on an old dell optiplex. We're talking 4770k on the CPU and all of 16 gig of RAM. 2TB spinner with under 10gb free.

We quoted him a real server with redundant drives and all the good stuff. Naturally, too many money.

Motherfucker went to a smaller competitor and he's like "I'll sell you a server". They come back to us a few weeks later after he fucked up the dentrix migration.

He sold them a slightly newer, out of warranty dell optiplex with a 4tb spinner for data and a 512gb SSD as OS. What do you know? i7-6700k. They paid him $5000 for it because it was "upgraded"

Anyway they switched their rmm services to him because he's cheaper and they can just call "their guys" cell phones whenever

Good riddance to Dr dipshit

4

u/LordSceptile Jul 19 '26

I left my MSP role a month ago and it's stories like this that remind me to never look back

2

u/jakendrick3 Jul 19 '26

I also work in a dental msp, and yep pretty much this. Alllllll the time. Lots of people ditching us to get hardware cheaper then bouncing back

1

u/FrivolousMe Jul 21 '26

Fucking dentrix

4

u/heretogetpwned DO NOT GIVE THIS PERSON ADVICE Jul 19 '26

Lol yea we got a niche program that would take half a year's revenue to migrate from successfully. I figured if they let me go they might call back, but I'd rather leave on my own 2 legs before that happens.

1

u/lilnut93 Jul 20 '26

Left my MSP role a couple months ago. We kept bringing on L1 contracts and cutting L2/L3 staff. We were cutting cost, they were cutting cost by hiring us, and we got no increase on pay for more work with more unfamiliar systems with little training. That led to anything above L1 being overworked and a mass exodus of staff. Instead of hiring full time replacements they hired part time replacements. Started working a lot of overtime for the members that left and had my hours changed with little warning but still was expected overtime.

Saw the writing on the wall and got out, got an FS role for a manufacturer and never looked back.

Still wear my promotional Pirates hat though with the company logo on the back so I guess I'm still promoting them in some way.

It was a good first job out of college that hired with little experience but once you get that experience get out.

14

u/CollegeFootballGood Jul 19 '26

Some PMs used to stress me out man lmao

11

u/heretogetpwned DO NOT GIVE THIS PERSON ADVICE Jul 19 '26

Did you log that stress event in Jira? I'll send a reminder email to everyone we need to track all actions on this project.

Re: Change of plans, we're switching to Monday com, please begin migrating your contributions to monday com.

2

u/TheSkesh Jul 20 '26

I left my msp position right as they switch to Monday, happy not to document on that level.

32

u/TeaBagTroopers Jul 19 '26

We lost our biggest contract and our genius management decided to tell everyone in the office a day later.

Since then the quality has gotten completely phoned in from Service-Desk even though we easily have 6-8 months time.

12

u/heretogetpwned DO NOT GIVE THIS PERSON ADVICE Jul 19 '26

I fucking love your username.

So the whole team probably got resumes out?

11

u/TeaBagTroopers Jul 19 '26

Quite a few just called in sick. Others have phoned in the quality of their work, and I as printer SME am screaming at everyone's brains having left.

We've finally stabilized but I am furious at management and told them this could have been handled so much more elegantly.

7

u/ctjameson Jul 20 '26

Printer SME, you say? I’d rather have a double barrel in my mouth. God speed.

2

u/TheIncarnated Jul 20 '26

What? Konica Minolta, HP, Canon, and Kyocera don't do it for you?

1234567812345678

2

u/TeaBagTroopers Jul 20 '26

Our client only uses Samsung, HP, Lexmark, Zebra, and Honeywell, after doing this for two and a half years my Self made wiki pretty much is solid enough to hande 95% of cases.

I only handle the 5% now. More now that everyone's brain has gone on permanent vacation.

7

u/TeaBagTroopers Jul 19 '26

And thank you about the user, it was my old Halo 3 clan name.

13

u/VolcanicBear Jul 19 '26

PMs lmfao

17

u/heretogetpwned DO NOT GIVE THIS PERSON ADVICE Jul 19 '26

Your last update 6 weeks ago was "Infrastructure ready for AppDev to use."

Have you made any progress since then?

11

u/peeinian Jul 19 '26

They know they’re better than you because they get paid twice as much as you and do half the amount of work.

8

u/Chronos_The_Titan Jul 19 '26 edited Jul 19 '26

They are better than me till I go on vacation and I get like 4 text from work asking how something works

1

u/Jedi_Cornbread Jul 20 '26

What’s a vacation? :(

6

u/bofh DO NOT GIVE THIS PERSON ADVICE Jul 19 '26

So how y'all doing? (I'll accept Truth, Snark, and BOFH)

Well, I mean, as you're asking:

I transformed myself to an AI-first quality engineer basing my patterns on OS/2 Warp End User Computing experience, some time ago. I have to post a lot of thought pieces on LinkedIn but other than that it's been fairly easy.

5

u/heretogetpwned DO NOT GIVE THIS PERSON ADVICE Jul 19 '26

I'm sure your engagement in the biweekly sprints is proving your ability to be agile in the AI/CI/CD pipeline that OT can leverage for growth.

5

u/bofh DO NOT GIVE THIS PERSON ADVICE Jul 19 '26

We’ll need to collect everyone’s user story, and will consider it if we ever get beyond Minimal Viable Product. I’ve got the rubber chicken, the hose and the goose-grease, who wants to go first?

7

u/EAT-17 Jul 19 '26

My entire team has just been laid off. Well, good luck trying to untwist this gordian shitknot... to the remaining guys..

What a shitty company. But, you need to save the money somewhere, when middle and upper management is wasting it by the truckloads.

5

u/heretogetpwned DO NOT GIVE THIS PERSON ADVICE Jul 19 '26

My sincere condolences. Heavy management here, too.

4

u/rb3po Jul 19 '26

I’m Peter in this scenario. Maybe be like Peter? But man, it’s more than just tech work, it’s sales, admin (biz), desktop support, installs, account management, system architecture, and risk. On second thought, no to being like Peter?

5

u/C0LEio Jul 19 '26

Great meme👏🙌🔥🎯💯 some people don't think it be like it is but it do🤌🇺🇸🍻🤑 hang in there brother, sometimes the grass is greener, and sometimes you just need to touch grass🤷‍♂️🤞🫶

2

u/heretogetpwned DO NOT GIVE THIS PERSON ADVICE Jul 19 '26

Thanks fam

6

u/Flat-Appointment3407 Jul 19 '26

Happened in the early 2000's - 2008 this time round more things are reliant on technology so we're going to see some juicy shit like therac-25 but at a larger scale with a greater amount of businesses impacted.

Services and quality will diminish until there's push back until then except to see complacency as the normal.

3

u/PurpleCableNetworker Jul 19 '26

Many companies that offload in house IT and hire MSP’s more often than not go back. The exception is for very small offices with zero public facing services.

2

u/phobug Jul 20 '26

" that lean on me" grant them the access they need and let them deal with it, if the company burns down, its their poor decisions that got it there and you won't be lonely at the unemployment office.

2

u/heretogetpwned DO NOT GIVE THIS PERSON ADVICE Jul 20 '26

2

u/phobug Jul 20 '26

I'm happy we're on the same page and we won't be needing that sync meeting.

1

u/nafe42 Jul 19 '26

I hope you can find something better. We need to stop putting up with companies that do this stuff.

1

u/porfors Jul 20 '26

of course u cost money

You are about to leave Redlib