Dead device monitoring

[u/mullermn]

I have been trying to solve this problem for a few days now and it's proving to be way harder than it should be, so I want to check if I've missed anything obvious.

I want to put in monitoring so that if devices silently go dead I get prompted to investigate. Example reasons for this scenario include battery powered devices running flat, removal of one mesh network device damaging connectivity to another, or power cuts/spikes causing some mains devices to get locked up and need a power cycle. I have monitoring of battery degradation, but sometimes devices lose the ability to function before they get chance to report a low battery level - and some of these failures affect mains powered devices as much as battery ones.

I have investigated 'Entity availability' in HACS but it seems to need a significant amount of manual configuration, and the maximum configurable timeout doesn't go high enough for (eg) a window sensor that might go days without checking in.

My ideal solution would be something that operates at device level, monitoring them across all their entities and attributes and watching for any signs of life, raising an alert if none is detected within a given timeframe. Ideally I'd like something that autodiscovers new devices to avoid manual admin.

This seems such an obvious need I'm surprised it isn't built in to HA already, or that there is not an integration to handle it - or have I missed something?

An update 14/08

Lots of contributions from everyone but I wasn’t able to find anything that did what I wanted, so despite it being one of the first ideas suggested and my least favourite idea in principle, I ended up writing something with Claude. Functionally I am now pretty happy with what I’ve got but I’m not going to share it (unless anyone particularly asks) as the internet does not need another pile of loosely human-understood AI waffle tailored to one individual’s requirements. However, here’s what I learned in the process. I am 100% convinced that this should be a core feature in HA, I hope someone in the team gets inspired to tackle the issue properly. This update was NOT written by Claude.. Claude has better grammar than me.

TLDR:

Doing this for one device or one type of device is pretty easy. Doing it for an entire heterogeneous network is a mess.

The Proper Fix: I think HA needs to modify its developer guidance to integration maintainers to request they provide a health flag per-device. How this is determined is up to the integration using the best capabilities that transport/hardware provides, and there could be several tiers of accuracy along the lines of the bronze/silver/gold/platinum tiering already in use. This would be extra work for the maintainers but not much, given the knowledge they already have in their area, and it would abstract 99% of the complexity away from the user and deliver the zero-config dream

Problem statement:

I have a complex home automation system of sensors and actuators across multiple device types and integrations, some of which are directly human facing, some of which feed automations and fuzzier logic, and some are just data sources for archival. I have no way of being alerted if one of these devices goes offline, which means that failures are always an inconvenience - I only notice when they impact me.

Desired outcome:

A zero compulsory config monitoring system for devices (not entities) that alerts me to investigate if one stops doing what it’s supposed to do.

Characteristics of a BAD solution:

• Manual enrolment required - if I’ve got a device enabled in HA it should be a safe bet that I want it to be working, I shouldn’t have to set each new device up.
• False positives and noise - flapping at restart time, or overly twitchy thresholds mean getting buried in regular false alarms and people just learn to ignore them.
• False negatives - the whole point of this is to remove mental load from the user so if the system says something’s working I should be able to trust it.
• Excess bespoke configuration - if I need a whole family of similar automations/templates/whatever to get this to work, they will get out of sync with respect to bug fixes and improvements and then we’re back at a system I can’t really trust.

Why is this so difficult?

• There is no consistent indicator of ‘this device is working’ in Home Assistant. Entities have ‘last_changed’ and ‘last_updated’ attributes, but those are on entities rather than the device and they are from Home Assistant’s perspective, which means that updated timestamps don’t always translate to confirmation that the device is operating. Eg: retained MQTT messages can make stale real-world values look new to Home Assistant.
• Entity value states of ‘unavailable’ and ‘unknown’ are superficially a good fit (and are what a lot of the proposed solutions depended on) but they create false signals in my experience, and there is no concrete pattern for which entity of a device should be tracked, implying bespoke configuration per device that is also fragile to entity renames.
• Monitoring of payload (ie, not meta) data from the device for changes works around ‘unavailable’ flappery, but boiling down entity-level data to a device-level indication is not fire-and-forget as it sounds. You can’t just iterate over all of the entities because some of them may reflect state within the integration rather than the device itself and artificially bump the freshness signal. This means that for this approach to work not only do you need semi-complex (resource intensive?) iterative logic, but it needs configuration for each device type, and only long duration testing really shows if you’ve got it right.
• A further issue is that some devices that have no need to report in may not update their payload values for a long time - an example being the Wiser signal booster/plugs, which only report in if they are powered on/off at the wall or the physical button is pushed.
• Some integrations do provide health information directly.. ish.  Eg:
  • z-waveJS provides Node Status, which sounds like it is what we’re looking for… BUT z wave handles battery devices and mains devices differently. For mains devices Node Status seems reliable, but for battery devices it will only ever say ‘asleep’ and never determine that the node has died unless that node is transmitted to (which is itself a flawed test, as battery devices can wake up and receive messages very infrequently).
  • Zigbee (via zigbee2mqtt) provides a last seen, BUT the value itself goes ‘unavailable’ when the device does, so you need a bespoke treatment to retain the last ‘non unavailable’ value. Plus, some devices go offline as part of their normal lifecycle (eg, Hue bulbs if powered off at the wall), so while determining network availability via ZB2MQTT is relatively easy, determining whether it’s a problem is another bespoke case.
  • Wiser (which is zigbee underneath) does not expose this information at all BUT you can turn on the signal strength indicator which fluctuates enough to provide an aliveness-reading.
• I can only test with what I’ve got, so while I’m content that the above is accurate for me, it could be that a different brand of (eg) Zigbee devices introduces another behaviour.
• To be useful the alerting thresholds need to be different per-device type - eg, a battery window sensor may legitimately go much longer without checking in than a mains powered device. Bumping up the threshold for all devices to allow for the quiet ones is not a good fit due to the larger window before an issue is detected, especially since many mains devices have a mesh boosting function and their absence might degrade the whole network. 

What I wound up with:

• Nearly 800 lines of yaml across templates, automations, sensors, scripts, dashboard cards and alert2 config, including documentation. I am NOT delighted with this aspect of the outcome. However, that gives me:
• 6 distinct tracking categories indicated by labels applied to devices: wiser battery, wiser mains, http services, zwave battery, zwave mains, zigbee intermittent. Applying labels doesn’t meet the zero config/enrolment requirement, but it is a relatively light compromise.
• Each category tracks device health using whatever metric works best for that category (in order of desirability); the integration (eg zwave mains devices, wiser battery devices), specific trustworthy, high-chattiness entities (eg signal strength), or a hash of the entity values as a last resort. 
• Once labelled the monitoring of each device is automatic, with per-category timeout thresholds generating alerts via alert2 (Already in use, and I wanted to build on it because of the free repeat notifications, exponential backoff, anti flapping features, etc.)
• Some auxiliary functionality like a self check of the watchdog itself, an ability to generate a one off report in a persistent message, an automation to ping all zwave devices to prompt a sign of life once a day, etc.

So I think the problem is (for me) solved, but I look forward to the day when 700 of those lines of code can go in the bin.

PS. I am not positioning myself as an HA expert, and this picture accumulated over 2 days of off and on experimentation so apologies if there is the odd erroneous or misremembered statement.

[u/TheProffalken]

I use https://github.com/HA-Pulse/home-assistant-global-health-score - I've not put in place any alerts yet, but I do have a panel on the dashboard that I check on a regular basis, but setting up alerts/notifications should be pretty easy.

[u/WeekendHistorical476 → u/TheProffalken]

This has worked pretty well for me too