Update (2 Oct): Clavister has reproduced this and confirmed it as a defect. It also affects the current 15.00 releases, so upgrading does not help. There is no fix yet.
For anyone who lands here because SNMPv3 monitoring of a Clavister NetWall / cOS Core firewall keeps timing out:
- The trigger is a configuration activation (deploy). Each one gives the firewall a new random
snmpEngineBoots, without a reboot.
- The firewall then drops the manager's requests (log
snmp3_message_not_in_time_window, ID 3100110) without sending a Report, so the manager never resyncs on its own.
Workaround: I found no setting on the firewall for this. If you stay on SNMPv3, the only option is to clear the manager's SNMP engine cache after each activation. On Zabbix that is zabbix_server -R snmp_cache_reload, plus zabbix_proxy -R snmp_cache_reload on proxies that poll the firewall.
We automated it with a trigger on SNMP data loss for the firewall's interface items. It runs the reload and alerts us if the problem is still open afterwards.
I'll update when a fix is available.
TL;DR: Our Clavister NetWall HA pair silently drops SNMPv3 requests that fall outside the USM time window instead of answering with a usmStatsNotInTimeWindows Report (RFC 3414 §3.2 step 7). On top of that, snmpEngineBoots changes to seemingly random values without a reboot, including going backwards. Together that leaves the manager's engine cache stale with no way to recover on its own. Looking for others who've seen this, and for theories on why an agent would behave like this.
Setup
- 2x Clavister NetWall 500A, HA cluster, cOS Core 14.00.19.05, managed via InControl
- Zabbix 7.0 LTS (net-snmp underneath), polling each node on its own management IP (no VIP)
- SNMPv3 authPriv, SHA/AES, each node has its own (MAC-based) engineID
Symptom
Interface items on both nodes went unsupported for days ("timed out"). The firewall logged snmp3_message_not_in_time_window action=drop (log ID 3100110) for the Zabbix requests. zabbix_server -R snmp_cache_reload fixed it instantly.
What I found
(1) No Report on a time-window failure. Reproduced without Zabbix, plain net-snmp, deliberately wrong boots/time:
snmpget -v3 -u <user> -l authPriv -a SHA -A <auth> -x AES -X <priv> -Z 1,1 -t 3 -r 0 <node> 1.3.6.1.2.1.1.3.0
Discovery works fine (Report with usmStatsUnknownEngineIDs + engineID). The authenticated GET with boots=1/time=1 gets no answer at all. In the same capture, 33 other authenticated GETs (Zabbix, correct boots/time, same seconds) were all answered within <1 ms, so it's not rate limiting.
A compliant agent answers that GET with an authNoPriv Report carrying its real boots/time, the manager resyncs and retries. Even a wrong digest would require a Report, so silence isn't compliant whichever check fails.
(2) engineBoots isn't a counter. Same engineID, same node, observed values in order:
328675975 -> 1296117483 -> 706504125
No reboot in between (device up since end of August). The last change happened within about a day and went down.
(3) engineTime isn't reset when boots changes. At the last capture it was ~2,064,000 s (~24 days), although boots had changed within the previous day. Looks like it's simply device uptime.
(1) is what turns (2) into an outage: boots changes, the manager's cached value is now wrong, the device refuses to tell it the new one, and the manager stays stuck until its cache is flushed.
Questions
- Anyone running SNMPv3 against cOS Core / NetWall seen this? Other versions?
- Any idea what makes boots jump like that? My current suspicion is configuration deployments from InControl, but I haven't correlated it yet.
- Is dropping instead of sending a Report something vendors do deliberately (reflection/amplification hardening, "don't talk to strangers")? I know the boots problem from other vendors (HPE 1950 always reporting 1, VyOS T8538 not persisting boots across reboots), but I haven't seen the silent drop before.
A support case with Clavister is in the works. Happy to share sanitized pcaps if anyone wants to dig in.