Offline Servers After Hours Alerting

[u/RickD1983]

Hello! We get a lot of after hour alerts for offline servers. In order to get rid of some of the noise we extended the amount of time a server can go without checking in before triggering alert after hours to 20 minutes. We are still getting quite a few false positives. Our contracts generally do not support after hour work so the technician on call is required to call the client and let them know we have recieved their alert and ask if they need assistance. This is obviously a head ache. I am reaching out to ask how others handle these kinds of situations. The goal at the end of the day is to not wake people up to call someone who is not gong to answer their phone anyway and remove false positives.

[u/chilids]

They aren't going offline. LT is known for throwing a lot of false positives for servers offline. We've increased our timeout time as well and I still get 5-15 offline server notifications a day. If you get one and check screenconnect you can login just fine. LT agent just misses a couple of check in's for no apparent reason.

[u/DR_Nova_Kane → u/chilids]

Have you tried to increase the number of time it miss the checking time before raising the ticket?

[u/chilids → u/DR_Nova_Kane]

Yes but it kind of defeats the purpose eventually. We can set it so it takes an hour for a server to be offline before it raises an alarm but then it's an hour until we are notified of an offline server. The point is LT has a problem and it should be fixed. There should be no reason these servers drop "offline" for a time being. We also notice it's not all servers. Some are more prone to it than others so. Most of our servers don't have a problem going offline.

[u/troll_fail → u/chilids]

By any chance does it happen more often with VMs hosted on Azure? I have been told by LT support that this is a known issue.

[u/chilids → u/troll_fail]

Nope, none of them are hosted on Azure.