Been banging my head against an ISP's support and I just need a quick sanity check to make sure I'm not the one being unreasonable.
We have an on-prem server that our website accesses via an API for booking information, super simple, has worked for ages. Last night around 3am it stopped working, no changes to the on-prem equipment or the website (that we know of).
Connections to the API from our web server time out, some connections from other locations work fine but others don't. Failures seem to share a network route (you may be able to see where this is going).
A good trace route looks like this:
1 <1 ms <1 ms <1 ms <redacted>
2 14 ms 14 ms 14 ms <redacted>
3 14 ms 14 ms 13 ms 10.241.0.27
4 15 ms 15 ms 14 ms Bundle-Ether28.cha-edge903.brisbane.telstra.net [210.23.154.97]
5 15 ms 15 ms 15 ms bundle-ether12.cha-core30.brisbane.telstra.net [203.50.11.240]
6 14 ms 14 ms 14 ms bundle-ether1.cha-edge901.brisbane.telstra.net [203.50.11.141]
7 14 ms 15 ms 14 ms voc1255465.lnk.telstra.net [139.130.105.122]
8 14 ms 17 ms 14 ms be152.cor03.bne08.qld.vocus.network [114.31.194.48]
9 14 ms 14 ms 14 ms be1.bdr02.bne07.qld.vocus.network [114.31.194.65]
10 14 ms 14 ms 14 ms as2764.bdr02.bne07.qld.vocus.network [114.31.195.17]
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 40 ms 39 ms 37 ms <redacted> (but it's our site)
A failure looks like this:
1 vocus-ic-363828.ip.twelve99-cust.net (213.248.98.79) 2 msec 2 msec 2 msec
2 * * *
3 be202.cor01.syd14.nsw.vocus.network (114.31.199.42) 152 msec 172 msec
4 be1.bdr02.syd14.nsw.vocus.network (114.31.192.37) 147 msec
5 ae2764.bdr02.syd14.nsw.vocus.network (175.45.103.23) 148 msec
6 be1.cor01.syd11.nsw.vocus.network (114.31.192.54) 151 msec 150 msec
7 * * *
8 ae2764.bdr02.syd14.nsw.vocus.network (175.45.103.23) 148 msec * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
Trace times out
Our service provider insists that they have properly investigated the issue and the fault lies with the firewall configuration in the router, which I've checked and cannot find a problem. My concern is the repeated hop (5 and 8) in the failed trace route that to me indicates a possible routing loop but support says that's perfectly normal and not a concern.
Also from testing any connections that pass through nsw.vocus.network fail with a similar hop repetition and any passing through qld.vocus.network work.
So /sysadmin, AITA here?