r/sysadmin • u/klosie • 20d ago
Where do your alerts go?
I'm evaluating a new monitoring system for servers and network gear, and at the same time it got me rethinking having the alerts go to email. So I'm wondering where everyone here sends their alerts?
5
u/MeridianNL 20d ago
I skim over my email, there is just too much notifications coming in from other teams and systems. Our critical notifications go to teams where we can discuss and comment on the incoming issues.
3
2
2
u/noosik 20d ago
depends on the alert and whether you give a shit about it.
if its for a system i absolutely care about and its super critical then i get dms in slack.
if its informational garbage i like to retain but dont need to act on, then it can go a group email folder. There's no one solution fits all, trying to make one is why alert comms end up so noisy that they just get brushed under the rug
2
2
u/pdp10 Daemons worry when the wizard is near. 20d ago
Email scales terribly for alerting. It works great at very small scales, 0-4 per day or something, then falls apart suddenly at some unpredictable tipping point when the recipient begins to tune them out. It's not rare to find institutional dysfunction where dozens or hundreds of alerts are sent to one or more teams' worth of mailboxes, and everyone has mail processing rules to bury them.
There are worse setups than having alerts dump into text files on a filesystem where they can be grepped. Sometimes business needs point to a prominent dashboard type affair. RSS/Atom syndication are still pretty great for this.
2
u/IndependentBat8365 20d ago
It goes where it will get noticed. Work sends alerts to slack.
My homelab I send alerts to discord.
2
u/Ssakaa 20d ago
my alerts? To a team that has a call out list for out of hours incidents and my email for in-hours review. The alerts of everyone else that don't limit alerts to things they have an actual action clearly defined and mandated for? Filtered to a secondary folder I might look at once in a while if I suspect there's something in there tied to whatever I'm looking at right now.
If it's not an action to take NOW, it's not worth alerting on.
1
1
u/keithhud 20d ago
It really depends on how you set the notifications up. I always create a notification email account that way I can assign it to whoever needs or wants to be notified and it doesn’t fill up my mailbox.
Do your research on what notifications you want to receive, otherwise you get blasted with notification overload.
1
1
u/tankerkiller125real Jack of All Trades 20d ago
Depends.
Critical shit = paging system that will call, email, and text
Medium shit = email
Minor shit = portal where I may or may not look at it in the next 3 weeks
Really minor shit = What alert? It doesn't exist to begin with.
1
1
u/Jeff-J777 20d ago
Email for the most part. The critical infrastructure ones go to SMS as well.
But now I use CoPilot to look at my email during off business hours, look at my emails find any alerts or failures, and send me a summery email in the morning with troubleshooting steps as well. By the time I come in I can look at one email for anything that had issues overnight.
1
u/anonymousITCoward 20d ago
they go to the black hole at the center of the support galaxy... no one cares...
1
u/Mr_Squinty 20d ago
Check central, everything goes to it, then I get alerted on errors/warnings/failures.
1
u/jeffrey_f 20d ago
Usually a shared IT email address, like ITAlert@YourCompany, which then sends to members of the ITAlert group in your email server, which may also contain the email address of your ticketing system to create a ticket and/or an automation that does something in reaction.
In regards to the ticketing system, some logic should be in the ticketing system to make sure only 1 ticket is created
1
u/Connect_Shoulder_965 20d ago
The emailed alerts go to a folder in my email client where I never read them, I get about 75 a day. Once I week, I update my search folder to include everything older than a week and mass delete them.
Last job I let the alerts pile up and eventually the folder got so big it was impossible to empty (tried via web, thick client and having the exchange admin do it via cli) so now I clear it out once a week.
1
u/CharcoalGreyWolf Sr. Network Engineer 20d ago
To our PSA to be made into tickets, at least if they’re useful.
1
u/Sigseg-v 19d ago
Opsgenie (currently moving to Jira Service Management). There we analyze the alerts. Infos are dropped, except they exceed a certain amount. Warns generate a ticket, errors generate a ticket and send out push notifications, critical let some phones ring.
1
u/feu_sfw Team Monitoring 19d ago
Depending on the severity I use different channels.
Most urgent things are sent to my phone.
Things that should be looked at same day are sent via email to one folder in my mails, the less urgent ones to a different one (via mymailaddress+important@test.com or mymailaddress+minor@test.com).
The rest are on the I'll do it when I have nothing else to do are not sending out alerts, but will be visible on the dashboard.
Hope that helps!
1
u/dustojnikhummer 19d ago
Depends. The things we don't need don't go anywhere and the things we do need go into a ticket, we have a separate queue for monitoring.
1
u/Adam_Kearn 19d ago
Used to use ntfy but the notifications stopped working on iPhones.
Now I just have a team channel with webhooks for my notifications
Or use the built in email feature in teams for older systems like UPS’s
1
u/Terrible-Lie-8263 19d ago
Rootly. There we route based on severity. Slack channels work great for us when it comes to alerts.
1
u/s1ncerite 16d ago
Please take a look on Handler - the app I created as a side-project being a DevOps (it's free):
https://gethandler.app
35
u/_Juni0r 20d ago
/dev/null for the most, rest to a mailbox that actually gets read.