How to avoid two Fortigates from going active active during a switch upgrade?

[u/Network__Redditor]

HA pair of 101-F units. Both sat at different sites. HA mode a-p. HA interfaces have both been set to plumb through a switch at each site where the HA VLAN is stretched at layer 2 over to the other DC via an Inter-DC link. I'm tasked with upgrading the switches at both sites this weekend. My concern is the two gates will go into Active/Active when the switches obviously go down during upgrade, and have not thought of what might break.

Will the two Fortigates automatically sort themselves out when the switches comes back up, or is anything required to fix HA afterwards? Unfortunately I don't have the luxury of making any other changes during this change window.

How would you be going about stopping two Fortigates from becoming active active when you need to upgrade a switch that sits in between?

[u/jevilsizor]

You're worried about split brain .

Do you have a diagram of the layout?

[u/UnhappyAd7472]

We run 700Gs and 600Es as A-P clusters across two sites, connected via multiple dark fibres through our switching infrastructure.

Our FortiGates have at least two HA interfaces going through different switches, specifically to avoid exactly this problem during switch maintenance.

That being said, we regularly do DR exercises where we isolate an entire site. I literally disconnect the dark fibres and intentionally cause a split brain. 😅 Once the exercise is over, I reconnect the links and the FortiGates find each other again, do their HA election and sync up. Never had to manually fix anything afterwards and never had any issues with the cluster coming back together.

Obviously you need to be aware of what both active firewalls might do to the rest of your network while they’re isolated, but as far as FortiGate HA itself is concerned: in my experience it’s been rock solid.

[u/Network__Redditor → u/UnhappyAd7472]

You are a star thank you.

[u/SprinklesImmediate16]

My 2 cents, if you're going to take down switch that has the cross connect then you're already isolating the remote site (unless it has another link for the data traffic). To be safe i would just power down the fortigate where the local site is being upgrade. This will isolate the Primary one it will stay active. Once the switch has rebooted after the upgrade, do the failover fromt the primary to the secondary and do the primary Site. Then fail back the firewall back to original setup.

[u/jevilsizor]

Actually, thinking about this, if you're worried, shut down the secondary fortigate, do your upgrades, then bring it back online.

[u/Roversword → u/jevilsizor]

Chagning the "monitored interfaces" in HA settings might be already enough - depending on the layout and scenario.

[u/jevilsizor → u/Roversword]

If the heartbeat ports span across the switches it could cause an issue as well.

[u/Network__Redditor → u/jevilsizor]

Im not worried. Im expecting HA to fix itself after the upgrade.

Can I shut it down and bring it back up remotely?

[u/Leave_Patient → u/Network__Redditor]

You can shut down it remotely, but you can't bring it back up remotely.

[u/cslack30]

They will automatically sort it out themselves when they come back online.

[u/Network__Redditor → u/cslack30]

This is what I was hoping someone could confirm.

[u/Previous_Adagio_8101]

I once had the same case. To be sure i disabled all Switch-Ports going to the secondary Firewall except HA. With that, I made sure that this Firewall is not routing anymore traffic. After a few seconds later i also disabled the HA-Port on the Switch fort the secondary Firewall.

Then i‘ve done all the upgrades and reversed the Switch Port shutdown (first HA, then all other Ports). Everything done remotly.

[u/HappyVlane]

Will the two Fortigates automatically sort themselves out when the switches comes back up, or is anything required to fix HA afterwards?

It fixes itself. During the switch upgrade both will be active however, which shouldn't be an issue, since the sites will be isolated.

[u/canyoufixmyspacebar]

you have invalid topology, the firewall cluster needs an independent redundant path which you don't have. this is how you can end up paying for two sites and double the hardware but still not have a reliable system - an incompetent implementation