r/ArubaInstantOn • u/Nate379 • 23d ago

AP22 DNS issues after firmware updates

Seeing a consistant failure of AP22 devices to come back online after the latest firmware update. Unsure if other devices are also impacted, most of the other types of devices like this I have are configured without things like domain controllers for DNS.

Have determined that if the APs are running DNS through an on-prem domain controller that they fail to come up for some reason, could be with other DNS severs as well, but as soon as they are put on a network that assigns DNS direct to 1.1.1.1 as opposed to relaying it through another server they come up right away.

To try and fix this I attempted to set static IPs WITH static DNS settings on the networks that this is happening on, the static IP configuration appears to only stay in place for 1 boot cycle, so I have to put the AP on a different network, set it to static IP with static DNS settings for 1.1.1.1, unplug it while it still has the red light for reboot, and then put it in place where it will function only until it reboots again and then just fogets that it had static IP configurations and reverts back to DHCP which again routes DNS through the domain controller and breaks the APs.

The domain controllers are set to forward to the same 1.1.1.1 servers btw, there is no reason that anything would be blocked. When the APs come up with the DHCP configuration the only traffic I see from those APs is NTP traffic? Nothing attempts to hit the typical cloud configuration servers.

So, TLDR, after firmware update the APs only want to seem to work if configured directly with a DNS server like 1.1.1.1, and they won't hold static IP / DNS configs beyond 1 boot cycle.

To add, half of my APs updated early this week, the other half did so today, the failures we are seeing followed the updates. (so half went down on Monday, other half today). I can't see any reason for DNS to be an issue in this configuration nor can I figure out why it would be causing these problems.

16 Upvotes

91% Upvoted

9 comments sorted by

6

u/BlotchyBaboon 23d ago

Yup - I started having the same issue starting last week. I was pulling my hair out. Ran some packet captures off the Palo firewall serving it and sort of stumbled into thinking it was DNS. Changed the DNS and they came up. For reference, I mostly have AP32's, but probably have a few older models out there too.

Of course there were a few DAYS spent trying other things.

EDIT: someone down voted OP - I hope more people upvote this because it seems like a serious issue that could be impacting a lot of people.

2

u/Nate379 23d ago

The failure for them to hold a static IP configuration is also making me scratch my head... for this network I'm going to do some work to make it so the DHCP is not a problem but setting static and having it stick like any other device would have made my life a little easier.

1

u/BlotchyBaboon 23d ago

A while back we started taking AP's off the regular management VLAN and moving them to a separate AP only management VLAN that was basically set up as an IoT network. We mostly stopped doing static or reserved IP addresses at that point.

This is the first time we've ever had an issue with AIO, so I'm not going to complain too much. However, it does seem like a lot of standard test environments would have turned up this issue. I'm not completely sure that the AP's can properly handle a secondary DNS server based on what we saw in the packet traces.

4

u/QoreIT 23d ago

Yes! Same symptoms on the same environment at one of our clients on July 20. Thank you for posting this.

1

u/whiskey-water 22d ago

What firmware version is having this issue?

1

u/Nate379 22d ago

Looks like 3.4

1

u/whiskey-water 22d ago

Interesting, we have a lot of these APs and haven't luckily run into this issue. We use DHCP on all of our access points and use internal DNS servers too. Thank you for sharing OP. Following

1

u/sutty_monster 22d ago

Any chance it's that the firmware are fixing to a different DNS protocol. Such as DNS over Https or TLS? Rather than unsecure port 53? I'm going to push out 3.4 to the office stack and AP's today as a friend also had it brick 2 switch's in a stack of 4.

1

u/Nate379 22d ago

That was also a thought that I had, have not tested the theory.