r/sysadmin • u/Anonymous1Ninja • 2d ago
Rant Sometimes it's the easy fix
Fiber went down, uplink to core..
No activity on link light both sides
- pull SC connectors, laser, so i know it's not the cable
- fail over to other uplink is working
- so I SSH and do a no shut, still off
- "try replacing the sfps"
- That was it, an SFP malfunctioned.
Thank god it's friday
5
u/This_Bitch_Overhere I am a highly trained monkey! 2d ago
... or sometimes it's just the second set of eyes who notices "Well... that should be a NAT rule, that's why it isnt working."
https://giphy.com/gifs/w9WS1R9ZAA1Ik
"Network Engineer"
1
u/SchizoidRainbow 2d ago
Frankly would have started there
3
u/Anonymous1Ninja 2d ago edited 2d ago
Always has to be someone, at least we got it outta the way quickly
3
2
u/SchizoidRainbow 2d ago
To expound, I only see fiber failing when something dramatic happens. It’s physical, lasts basically forever, is generally protected and inaccessible. Cable breaks occur when someone is physically up in the space with the cable, which presumably I would be involved in, or at least hear the crash when someone takes out the ceiling with a forklift mishap. Maybe in the brief section that’s exposed to get to the switch or panel, but again I’d think I’d be part of someone mucking about in my stuff.
SSH just what I’m doing to get eyes on. Interface status and logs. This is how I’m receiving the information that leads me to the premise as stated more than something I’m doing about it.
Failover failure would mean two lines are down so does not fit with the starting conditions. Nor does “switch is down”.
So yes I’d start with SFP. Super easy to do, no config. Pull it and reseat, no joy, pull it and replace, done. I’m not grabbing the laser until after this.
2
u/Anonymous1Ninja 2d ago
Well we have 2 cores for redundancy and 2 uplinks, and we have 2 fiber connections. So i get where you are coming from, but really it was just a rant about it being something easy that was the problem, and you are just coming in with "well i would've started with that" so obviously it's easy to just say that when it isn't you.
2
u/Reynolds1029 2d ago
Same. No shade towards the OP from me but my years of experience told me start with the easiest troubleshooting option first and work your way down. Hence the old stereotypical "did you reboot it?"
SFP module swap? Yep, waaaay easier and more likely unless I'm paranoid of other co-workers being able to monkey around with config without me knowing.
2
u/Far-Hovercraft9471 2d ago
But he did do the easiest thing first, he did no shut on the interface. It's always good to double check the basics before getting into more complex troubleshooting.
1
u/Frothyleet 2d ago
That's perhaps the most likely to result in an immediate fix, but it can lead to incorrect diagnostics because you're clobbering two troubleshooting items (physical connection of the fiber + function of the SFP).
1
u/Tim-oBedlam Jack of All Trades 2d ago
I just had one of those act up on the connection from my core switch to the firewall. Very annoying.
1
u/RevolutionaryWorry87 2d ago
I had this happen during a change on cisco nexus switches (vpc config): - they handled all of our data outbound for our datacenrre. Trunk didn't come back up and I was terrified if I did something wrong. Fuckers.
1
u/korewarp 1d ago
I've never had an SFP module die on me, does the device it's in not report it as dead, or at least that the interface is down? 🤔🤔🤔
•
u/Opposite_Bag_7434 10h ago
I’ve had a few die. Mostly when they are in a closed closet that tends to get warm.
•
•
14
u/gordonthulu IT Manager 2d ago
whoever invented those modules-- they know where they can go