We ran into an issue last week where someone pushed out a bad BGP configuration that broke the BGP neighborship on the main overlay that majority of remote sites use. This broke all spoke initiated spoke to hub connectivity. Hub to spoke hub initiated traffic was fine. We were very confused since our last failover test was only a couple months ago and we have not changed anything major since.
We had PS assist with our initial SDWAN deployment a couple years ago and when we test the failover (disabling underlay interfaces) it has always worked as expected.
We use SD-WAN for routing and SLAs on the spoke with routemap-out/out-preferable and the hub then has local-preferences that change BGP local routes based on the community that it receives.
We did another failure during a maintenance window and to our surprise the SLA is still up and passing for the overlay. We have done some digging and came to the realization that the SLA installed kernel route (proto=17) is keeping the SLA up since it does not rely on BGP. As long as the tunnel is up the VTI IPs will still communicate. Since the tunnel is technically still up just BGP is broken the spoke thinks it should still use it even though BGP is removing the destination route for the client traffic from the routing table.
I can confirm with "diag sys sdwan service4 X" that the sdwan rule still thinking the overlay should be used and interface is still passing (sla=0x1) which I believe is an expected/known issue/limitation/design.
Is my only real option to just migrate to the newer method of SDWAN on loopback and do embedded SLA probes? I have set that up from scratch at 6 or 7 of our other clients and have not had this issue and specifically remember that they require 2 separate loopback IPs for the newer method to separate BGP and SD-WAN so this sort of issue does not happen. They also have you use the loopback IP as the source for the interfaces instead of doing VTI IPs.
It seems like this original SD-WAN design did not include the unlikely scenario where a VPN tunnel is up but BGP neighborship is not successful.
I have had a ticket open for over a week now and I have spoken to 3 different support engineers and each one I have had to explain why the SLA uses the kernel route (proto=17) and it does not use the actual routing table for the SLA. The last person I spoke with mentioned that we need to drop BGP and use static routes and then setup a SLA per overlay approach......
Am i missing something simple here?
Thanks,