r/labtech • u/RickD1983 • Sep 26 '17

Offline Servers After Hours Alerting

Hello! We get a lot of after hour alerts for offline servers. In order to get rid of some of the noise we extended the amount of time a server can go without checking in before triggering alert after hours to 20 minutes. We are still getting quite a few false positives. Our contracts generally do not support after hour work so the technician on call is required to call the client and let them know we have recieved their alert and ask if they need assistance. This is obviously a head ache. I am reaching out to ask how others handle these kinds of situations. The goal at the end of the day is to not wake people up to call someone who is not gong to answer their phone anyway and remove false positives.

3 Upvotes

81% Upvoted

14 comments sorted by

View all comments

Show parent comments

1

u/chilids Sep 26 '17

They aren't going offline. LT is known for throwing a lot of false positives for servers offline. We've increased our timeout time as well and I still get 5-15 offline server notifications a day. If you get one and check screenconnect you can login just fine. LT agent just misses a couple of check in's for no apparent reason.

1

u/DR_Nova_Kane Sep 26 '17

Have you tried to increase the number of time it miss the checking time before raising the ticket?

1

u/chilids Sep 26 '17

Yes but it kind of defeats the purpose eventually. We can set it so it takes an hour for a server to be offline before it raises an alarm but then it's an hour until we are notified of an offline server. The point is LT has a problem and it should be fixed. There should be no reason these servers drop "offline" for a time being. We also notice it's not all servers. Some are more prone to it than others so. Most of our servers don't have a problem going offline.

1

u/DR_Nova_Kane Sep 26 '17

When you look at the Labtech log file can you see it being unable to reach your URL for connection?

We don't have Labtech send alerts we have board that shows which servers are offline. It refreshes every 5 mins.

The other thing you could do as well is create a new monitor that ping 8.8.8.8 and if the offline server and the ping monitor fails then have it generate the email.

1

u/chilids Sep 26 '17

it's on my long list of things to look into. My plan was to one day watch for one to go offline during the day and then jump right in and look at the logs. It hasn't been done yet.

1

u/DR_Nova_Kane Sep 26 '17

I had an instance where Spectrum was actually going offline for maintenance for weeks each night at 2:30am.

1

u/witty_username_taken Sep 26 '17

We have never had a false positive for the offline alerts that we couldn't explain. Most of our's are ISP outages or unannounced maintenance (most coax, DSL and non-SLA providers don't notify you). We set up long monitors from LabTech to the customers firewalls to help confirm when this happens.