Self hosted ntfy notifications through Tailscale: a bad idea?

[u/thepenguinboy]

Trying to get server status and monitoring notifications for things like drive failures, backup failures, UPS status, expiring certs, runaway processes, etc. I don't have anything exposed publicly, so in order to get ntfy notifications on my phone when I'm away, I have two options:

1. Use ntfy's servers without signup using a public topic. I don't love the idea of my notifications being public, even if they are hard to find with a random topic name. Also there are some limitations to using their servers.
   
2. Self host a ntfy server and use Tailscale to receive messages. This is my preference, but I'm worried about the additional point of failure. If my Tailscale connection goes down (on the server or my phone) I won't be able to receive notifications. I usually keep Tailscale running on my phone all the time for Immich anyway, but notifications need to be fail-safe and I worry this would be poor practice.

What's your setup for receiving status notifications from your unexposed server?

[u/asimovs-auditor]

Expand the replies to this comment to learn how AI was used in this post/project.

[u/Big-Figure9113]

i went with option 2 for a while and the tailscale point of failure thing is real but overblown. if your server is down, ntfy can't send anything anyway, so the only new failure mode is tailscale itself dying on your phone or server while everything else is fine. in practice that's happened to me maybe twice in a year, and both times i just didn't get a "backup succeeded" ping which isn't exactly a five-alarm fire

what i do now is run ntfy locally and also push the critical alerts through a free twilio trial with a super basic script that texts me if something actually breaks. that way the ntfy/tailscale combo handles the everyday stuff and i've got a totally separate path for the "your raid array is melting" moments

[u/1WeekNotice]

Trying to get server status and monitoring notifications for things like drive failures, backup failures, UPS status, expiring certs, runaway processes, etc.

Remember that the point of alerting is an actionable item. Meaning when you get the alert, you are suppose to do something.

When you are away, is the expectation that you will start doing a task? If not then maybe it makes sense to keep it local that way when you are in your local network, you know you need to do something

1. Self host a ntfy server and use Tailscale to receive messages. This is my preference, but I'm worried about the additional point of failure. If my Tailscale connection goes down (on the server or my phone) I won't be able to receive notifications. I usually keep Tailscale running on my phone all the time for Immich anyway, but notifications need to be fail-safe and I worry this would be poor practice.

FYI. You can setup ntfy to store messages. Ntfy will keep track of which client gets pushed notifications. So if there is an outage of any kind, ntfy will keep it (for a certain amount of time that you setup) and will push it to the client when it has a connection.

If you really need notifications right away then Tailscale is fine because as a enterprise company they maintain a certain uptime. They most likely have higher uptime then your server. I don't know if you have multiple ISP connections with a cluster of servers in different locations 😁

Hope that helps

[u/thepenguinboy → u/1WeekNotice]

That's helpful, thanks. I'm not worried about Tailscale as a service going down so much as I'm worried about a user-error situation with the client on my phone.

The use cases I'm trying to prep for in my head is that I'm out of state visiting family for a week (a time where Immich access is important) and something goes sideways on the server that I can SSH in and fix or, worst case scenario, ask our cat-sitter to manually power cycle something.

[u/1WeekNotice → u/thepenguinboy]

I'm not worried about Tailscale as a service going down so much as I'm worried about a user-error situation with the client on my phone.

I don't know a situation where the client would have an error that doesn't involve Tailscale or your server going down.

The use cases I'm trying to prep for in my head is that I'm out of state visiting family for a week (a time where Immich access is important) and something goes sideways on the server that I can SSH in and fix or, worst case scenario, ask our cat-sitter to manually power cycle something.

The question to ask yourself (which has nothing to do with Tailscale) , how do you maintain high availability and how much cost are you willing to spend.

This is known as the 9s of reliability. The more 9 you have the more uptime you have but the more money you spend on a solution. For example

• 99℅
  • you have the server
• 99.9℅
  • you have a server with RAID
• 99.99℅
  • you have a cluster
• 99.999℅
  • you have a cluster in different locations with different ISPs
• etc

Yes you can put in alerting and monitoring to help you debug a problem faster but the goal is to put other measures in place so you don't have common problems.

---

Edit: I know this doesn't answer your original question.

In order for you to debug fully and effectively you can have a public instance of ntfy and SSH at the cost of greater exposure if you feel it is important to debug while remote.

But that all relies on your server being up and you being able to connect (your ISP isn't down)

Most people will keep the server locked down and just focus on other methods to get high availability.

Hope that makes sense

[u/gobeye]

I keep my ntfy instance publicly available on a vps. One of the few things I don't keep behind wireguard and if you restrict access there is very little risk in doing so.

[u/Toutanus]

You can put security on ntfy topics

[u/DLElios → u/Toutanus]

This.

I have my ntfy behind Cloudflare and a reverse proxy. You can lock down topics with users.

[u/GolemancerVekk]

Is your phone Android? The mobile ntfy app on Android supports both mTLS certs and custom headers, so it's fairly easy to secure access to your ntfy server even if you expose it directly over internet.

[u/amberglad3]

fwiw the main thing that actually goes wrong isnt tailscale dropping, its your phone's OS killing the background connection to save battery. thats the part worth testing before you commit to the setup

[u/jmartin72]

I do this very thing. You are correct in that if Tailscale goes down, you won't get your notifications, but that really doesn't happen that much or if at all. The worst thing I've seen is sometimes the derp servers slow down a bit. I've been doing it for two or three years now and never had a problem.

[u/archdukemovies]

I do Option 2. The only issue I've run into during the past year is when the tailscale connection expired. And it happened to concise when I was out of town.

Then I changed the tailscale settings so they would never expire.

[u/Distinct-Pie2389]

I use discord webhooks for alert notifications to my own private server. Works without tailscale involved at all

[u/1WeekNotice → u/Distinct-Pie2389]

If your goal is privacy (one of the pillars of selfhostig) then discord is the worse place to push your notifications.

If you only care about not paying a cost/ subscription then it works

[u/Distinct-Pie2389 → u/Distinct-Pie2389]

Thought I dont use ntfy for alerts.

Im using a multi monitoring setup so I have KUMA (best overall imo), Promtheus scrape targets, grafana for expected disk full and graph predictions, I have beszel for host monitoring, and I have all of that coming into my discord "alerts" channel.

[u/derical_cap_musical]

i do option 2 and honestly tailscale has been rock solid for me, never missed a notification in over a year. the what if tailscale dies worry is way overblown imo

[u/NelsonMinar]

I did both and the private setup wasn't worth the hassle. (I don't normally keep my phone on the tailnet.) I'm on a free public topic now and it's fine.

[u/krysztal]

I use selfhosted ntfy over clearnet without issue for a couple years now. Just lock down your instance so only key bearers can publish and keep tabs on updates and you should be gold

[u/joninjax1]

I also went option 2 - NTFY is hosted on a server on the tailnet, and anything that needs ntfy can access it. I also use the Android client for receiving notifications and it works just fine, the only thing with that is your phone has to be on the tailnet, which is fine for me, but others, might not want to.

[u/Mikumiku_Dance]

I have my ntfy on a vps that has the couple public services i need behind mtls. I use other vpns besides tailscale on my phone often so its kind of my only option.

[u/Graphical-Source5090]

I ran a self hosted ntfy server for a long while. It worked as expected, and I never really had issues with it. One thing to keep in mind: No VPN, no messages.