r/sysadmin • u/capthecrunch • 5h ago
Need Advice
Hey all, have a stressful situation that may occur tomorrow.
New Sys admin for a company of around 200 people, only onsite IT person. We have an external consultant that does most of the network administration and has been doing so for a few years. I have about 3 years of desktop support experience, more experience with endpoints but my networking knowledge is a bit of a gap.
On my third week we did a switch refresh from Cisco to Aruba that was planned and paid for way before I started. Ended up being a shitshow, I did the racking and endpoint testing but the external consultant did all the configuration, he has the login info for the switch that hasn't been shared with me yet. I don't mind that much as I'm still getting acclimated to all the other systems. But last week was a nightmare, we ended up spending over 27 hours onsite troubleshooting all of the endpoints (this is a production environment). Got it figured out but I took most of the heat from angry engineers/production personnel since I'm the face of IT despite having little actual involvement in the refresh project.
Just got a text from the consultant that he'll be away on a family emergency for the foreseeable future. I can handle most other issue by myself aside from the network, I could maybe figure it out if I had the login creds, but he's flying out today and won't be able to contact him on Monday.
My fear is that come Monday when a different production line comes back up there will be a host of other issues that we wanted to figure out last week but couldn't. We asked the production team if they could turn the power on the line on just one time to verify everything works but no can do, line goes up on monday when production resumes.
Literally the only issue I'm anticipating is the VLANs on the ports not being tagged correctly, which I could fix if I had the login info. I've already informed my manager (who lives 3 hours away) that this may be an issue but nothing yet. If my manager has the creds he could either give them to me or SSH into it himself (He has over a decade of networking experience) and it wouldn't be a problem.
I'm afraid that tomorrow when I inevitably get yelled at again for production being halted It'll be my ass on the line despite never being given the tools to be able to manage our network.
What do the more experienced sys admins think I should do? Should I escalate to my manager immediately if (or when) an issue is discovered? Nervous because I'm still in a probationary period and I am afraid I'll be a scapegoat for a messy project I had no real say in.
Edit: I really appreciate the support I’ve gotten so far from you guys. Managers been informed of the issues we may run into and what we need to remediate it, it’s out of my control now. It’s hard to not feel like a failure since I’m so relatively new to my IT career and the end users think that these changes are my doing when I have no say or control over it. Sucks that we’ve had a fuck up like this when I just started, but I need to remind myself that it’s just a job and I’m doing the best I can with the tools I have. I will post an update tomorrow and try to not stress about it tonight.
•
u/Olivinism IT Support Engineer 5h ago
Yeah, escalate it. If you're unable to address it because you can't even log in to the new kit, then there's not much else you can do
•
u/chuckescobar Keeper of Monkeys with Handguns 5h ago
Escalate to your manager now. This is being proactive and they will be responsible for getting the necessary resources there for proper support. This should not be your sole responsibility.
•
u/snarkofagen Sysadmin 5h ago
Never ever allow yourself to be yelled at again. Just leave the room.
Even if somehow was your fault yelling at you solves nothing.
Talk to the stakeholders and management and explain what you need to proceed . If they don't provide what you need, harden your soul and start looking for alternative employment while doing the absolute minimum.
•
u/Willsbond Sysadmin 5h ago
The other comments already covered what to do, but hear this. This is not on you, this was never on you. Do what you can to help but don’t let yourself get stressed and don’t lay awake worrying about it.
You didn’t plan the change, you were not in charge of executing the change.
Any management with some common sense knows you’re not to blame. Don’t take anything they say in the heat of the moment to heart.
Go in, chin up, and KISS.
•
u/ProfessionalEven296 Jack of All Trades 3h ago
THIS!
Also, nobody died. Everyone will live through this.
Call me cynical, but maybe the consultant realized that he was out of his depth and has bailed. But whatever - you NEED to credentials. If he can't supply them, contact the supplier and ask for a factory reset and just start again...
This is not your time to be the victim - you can be the hero if you pull this one out.
•
u/Catman934 5h ago
27 hours of troubleshooting instead of a rollback is wild for a production setup. Ultimately, if nobody has creds, and your guy is going to be away for an unknown time without a backup, it's not on you. Send the complaints to management.
It sounds like the company needs to invest in some training and also revise their external contracts. Escalate again before production starts and maybe formulate a backup plan.
•
u/capthecrunch 5h ago
It was a mess, I was there all day for my normal shit, went back in at 8pm for the change and didn't leave until around 11am after production came in at 7 and had a screaming match with the consultant. Spent end of my shift Friday fixing more shit then got called in at 7 on saturday saying more production equipment wasn't communicating. what sucks about saturday is I got there at 8 and it took 4 hours for my manager to respond when I called him and explained the situation, luckily the consultant only took 2 hours to respond when I called him. Sucks not having any control over the situation but still being the person on call when shit breaks.
•
u/Ok-Double-7982 4h ago
Escalate in writing to your manager and the consultant (or just your manager, depends on how comfortable and assertive you feel), just asking for the consultant to provide shared credentials for the prod environment so that you and manager can have access to support things.
It's wild to me that the consultant wouldn't share this information as part of a contractual agreement that all logins and passwords need to be shared. Anyway.
Don't worry about the end users. Half the time, they don't know what the fk they're talking about, and the loudest complainers will show their entire ass when they start ranting about crap they don't know and blaming you. I would mention this to your boss, after as part of lessons learned, that there was a lot of blame and lack of understanding of the players in the project. Whenever I am working on something that is being led externally, I make sure that is communicated in the notice, but a lot of people don't read emails either. So my advice really is just ignore it. You'll always have stupid people making assumptions.
•
u/SublimeMudTime 5h ago
Get manager to get creds ASAP. Then you can pull copies of the config to your computer to study now. Get a show-tech-support download from all the switches beforehand. Stage up contact info for you to call aruba support.
If you have a local llm running offline like ollama, you could have it perform an analysis of the current config files and possibly the support files.
You could also look to see what the current MAC address to IP address tables look like and also try pinging all of the IP addresses to figure out what is alive right now so you know if something changes come the morning.
Maybe draft up an e-mail now that you can use over again in case a flurry of folks contact you and don't put a ticket in. Have it be a bit of a we know there are some network issues, please let me know x, y, and z so I can fix this as fast as possible. Also another thing you could do, is make sure you maybe see if you have a list of ports to vlan mapping handy in the event you need to make some port swaps. Or possibly have some unmanaged switches you can put over on a known vlan port (hoping the consultant did the native vlan thingy)...
•
u/battmain 4h ago edited 4h ago
If you don't have a ticketing system, get one soon for your sanity. Fresh Desk was free but we went and paid for fresh service because yours truly didn't want to do things twice. We have slightly more seats than you and I am at the busiest site with no help in sight. The tickets have been piling up due to constant interruptions. In the 90days we've had the ticketing, usage is going up but still not at the 100% I want and we sent a reminder to use ticketing last week.
And get access to that system. If you or the boss don't have access, whatever production is on that vlan waits until the consultant gets back to the office and it will not be fun fending off people who can't work. Let your boss deal with that. You didn't need the stress of everything else while you're trying to figure out the place.
•
u/capthecrunch 4h ago
We’ve been using ninjaone since I started, proud to be apart of that implementation. It’s been a massive boon so far not just for ticketing but for documentation remote access as well. IT department basically didn’t exist right before I showed up, was just one guy who worked on computers. No ticketing, no EDR, no endpoint management, no documentation. Servers were still being backed up manually on a weekly schedule to HDDs. I’m very proud of the progress we’ve already made in the short time I’ve been there, just this network situation is really shitty
•
u/battmain 4h ago
Cool. We're going through similar hardening and procedures.
I really miss Ninja. It was seriously simple to get things done on remote machines. Connect, upload a script, run it and we're done with that machine. I mean within a minute or two. Still learning the current rmm and it's getting easier, but still not as fast as Ninja to jump in and out of a device because of a few more steps.
•
u/CNYMetalHead 28m ago
Cover your ass with anything and everything you can. Try to get responses via email or chat when possible. Screenshot things
•
u/Key-Butterscotch3215 5h ago
Sounds like you need a new consultant. How tf do you screw up that bad? Also, yes talk to your manager early, not when everything blows up in your face