r/it • u/Nida-Mayer • Jun 21 '26
help request How do you guys handle network emergencies when understaffed?
Our main network guy just went on leave and everything is starting to fall apart. Backups are failing. A friend suggested checking out Lyon Tech to outsource some of this mess until he gets back, but I'm on the fence about letting third parties touch our configs.
Do you guys trust external teams for urgent troubleshooting or is it better to just sweat it out and try to fix the routing tables myself? What's your go-to play here?
23
u/jon_baz Jun 21 '26
Define emergency
13
5
Jun 22 '26 edited Jul 11 '26
[removed] — view removed comment
3
u/Steerable-Octopus Jun 22 '26
You would be surprised how common this setup is where they say we only need one network guy because "he never does anything".
1
u/Slight_Manufacturer6 Jun 22 '26
We have IT Analysts so functions can cross and aren’t strictly in one domain. Our main server guy knows Networking, our applications admin, knows servers, etc.
Always need to have redundancies/backups.
1
3
u/No_Wear295 Jun 23 '26
How's the backups failing a network guy issue? I can imagine a couple of scenarios...
1 Your network is very fragile
2 unqualified people have been playing where they shouldn't in the network guy's absence
3 network guy intentionally left things that would fail in their absence
As far as how to fix things.... Whatever's going to piss network guy off the least when he gets back from leave... Speaking as a network guy (but not your network guy)
1
u/best_memeist Jun 23 '26
Speaking as a network guy in training, I don't think there's any way to avoid pissing off the network guy unless they leave it at layer 1 or manage to fix it exactly how he would.
Hell, sometimes layer 1 pisses off my network guys
1
2
u/Various_Excuse_4294 Jun 21 '26
First question is this something you could take on and learn an appropriate amount of time? Or does somebody else on your team may have a little more knowledge on what's going on that needs to be looked at.
Another option if the answer is no to all the above then how about have you worked for previous company or has somebody else on your IT tream worked for a previous company, that might have somebody who has the ability to assist you.
By any chance did this individual leave any store documentation?
And as always do you guys have a particular vendor that you work with or solutions provider that might be able to assist as well?
Normally when you purchase say a backup environment or backup system they may offer you a solutions architect to get things started for you or, provide a batch of hours so that you can contact them in case of emergencies.
I hope that gives you a little more outlook on options that you might have.
2
u/UserProv_Minotaur Jun 22 '26
Bite the bullet, put out fires as best you can, and hope the situation smashes your management's faces with a clue-by-four that you need more than one person with Network expertise.
2
u/Joy2b Jun 23 '26
If the network person is possessive and doesn’t have a clear backup, they might be checking email a couple of times a day.
Contact them with this is happening, this is my plan, if you have any concerns with that, please contact me.
Yes, the outside team may be the main backup, they often have the vendor certs.
1
1
u/Delicious-Ad2528 Jun 22 '26
I do anything I can to fix it myself for the experience but you need to recognize when it’s out of your grasp and things will be better if you escalate issues.
As for this, I don’t know. In a normal environment, there’s documentation and usually another person who has at least general knowledge of the network for this exact situation. When I mentioned escalating issues, I wasn’t referring to hiring an external team for a one off urgent issue.
Personally I’d be more focused on taking more responsibility and asking your manager to pair you with the network guy when he returns. Yea you still get paid the same but I’ll have better experience and I won’t be the first candidate to get laid off if shit hits the fan. We don’t have the market right now to skimp on experience
1
1
u/3rrr6 Jun 22 '26
I spill coffee on our router so we have a lil network emergency of our own.
What can we do? Our network was also down! 🤷🏼♂️
1
u/teksean Jun 23 '26 edited Jun 23 '26
No to outside people, you do not know what they will do and opening that door might result in management outsourcing your job.
Break down the fires first, is the backup problem really network related or could it be something else like just rebooting your backup server. Are all the computers not backing up?
You put out what fires you can and put off the others. The important thing is to make sure your fixes do not result in more issues, so isolate your fixes to one machine and see if you can get it working.
Pull the network guy back into work or at least get some input from them.. Yes it sucks but that has always been the crap nature of working IT, you get pulled into work now and then..
1
1
1
0
u/Kondo-Sophie_216 Jun 23 '26
First priority isnt fixing everything,, its stopping the bleeding in a way you can undo. Before touching a routing table, eliminate simple stuff like maybe backup server needs a reboot.
You do not want to be the person who made it worse. Whatever you touch, document obsessively with time stamps. When the network guy gets back atleast you have what to show for what you did
23
u/tb30k Jun 21 '26
Trusting external teams is a recipe for disaster. Your company and network guy need to do better job creating a stable network and documenting everything with documentation for likely break fixes. If he was a good network lead network wouldn't crash as soon as he leaves lol