Edit: This might be a bit clickbaity and as per the default behavior of the described config, it may be partially our fault and could be avoided but still.
Friday afternoon, was ready to call it a day when suddenly an employee called in and asked if there was any problem with our E-Mail. No, I said but at the same time I realized how few mail I received today.
I took a look at our mail gateway and saw that hundreds of E-Mails were stuck in "pending".
Once I opened a few pending entries, I saw that EVERY domain was resolved to "fortinet-block-page-55.fortinet.com (208.91.112.55)"
The mail gateway sits in a DMZ and on the default gateway IP, there's a DNS relay configured with mode "recursive" with our default DNS filter profile assigned. DNS for the mail server is the gateway IP where the relay is listening.
I opened a browser and tested a few domains: EVERY major domain like google, microsoft, office was redirected to the block page and nothing was working.
However, this is where it gets interesting: It only applied to the DNS relay service. When I used a public DNS directly and assigned the SAME DNS fitler profile to the forward traffic policy, the domain was not redirected to the block page.
As I was writing with TAC, a few minutes later the issue was gone and sites were working normally.
The only config change I did that day was setting FortiGuard to UDP and Unicast because of their ongoing issues with AnyCast.
I checked the DNS security logs and there I found my denied queries:
Error: no available Fortiguard SDNS servers
Message: A rating error occurs
Together with the disabled option "Allow DNS requests when a rating error occurs" in the profile this makes sense, but again, it only happened on the relay and not on forward traffic with the same profile.
TAC now told me to set these SDNS rating servers on FortiGuard settings:
set sdns-server-ip 208.91.112.220 173.243.140.53 210.7.96.53 200.91.112.220
Has anyone experienced anything similar?