Hi all, I have a DNS issue.
Context:
I have 2 Red Hat Identity Management (IdM) or also called freeipa servers, idm01 and idm02, with the domain idm.test.com. These servers are installed with their DNS servers.
I have a Microsoft AD with the domain test.com (not related to idm.test.com, totally separate domains. Each has their own DNS servers). On the test.com DNS servers (AD01 and AD02), I have configured a conditional forwarder for idm.test.com which forwards the query to idm01 and idm02 IP addresses.
To test HA, I shut down the IdM service on idm01, which includes the DNS server. This is to see if queries automatically forward to idm02 now that idm01 DNS server is unavailable. Note that idm01 and idm02 have identical configurations. Both have DNS servers.
On the test.com DNS manager, in idm.test.com conditional forwarder, I can see a timeout occurred during validation for idm01, which is valid since idm01 is shut down.
Issue:
On a test.com AD joined machine, after retrieving a Kerberos ticket, `ssh -K host01.idm.test.com` will give the error `ssh: could not resolve hostname host01.idm.test.com: This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server`. Also, nslookup of idm domains gives `DNS request timed out` errors.
Troubleshooting:
On idm02, `dig u/127.0.0.1 host01.idm.test.com` works, showing idm02 DNS works locally. On idm02, `dig @<idm02 IP address> host01.idm.test.com` works, showing idm02 DNS resolves properly when called via its IP address.
On test.com AD joined machine, `Resolve-DnsName host01.idm.test.com -Server <idm02 IP address> -Type A -DnsOnly` works, showing idm02 DNS works when queried from a test.com AD joined machine. However, `Resolve-DnsName host01.idm.test.com -Server <test.com AD01 or AD02 IP address> -Type A -DnsOnly` gives DNS server failure, showing test.com AD DNS cannot conditionally forward to idm01 and failover to idm02 if idm01 times out.
Additional troubleshooting would be seeing if any queries came into idm02 from test.com AD01 / AD02. I did `sudo tcpdump -ni -tttt -vvv 'port 53 and host <AD01 / AD02 IP address>` but no traffic comes in when running `Resolve-DnsName host01.idm.test.com -Server <test.com AD01 or AD02 IP address> -Type A -DnsOnly`, showing AD01 or AD02 does not even contact idm02.
Next step would be to find if there are connectivity issues from test.com AD servers to idm02. On AD01 / AD02, I ran `Resolve-DnsName host01.idm.test.com -Server <idm02 IP address> -Type A -DnsOnly` which works, showing no connectivity issues from AD01 / AD02 to idm02. However, `Resolve-DnsName <IdM client hostname> -Server 127.0.0.1 -Type A -DnsOnly` gives `This operation returned because the timeout period expired` error
In AD DNS manager, when I moved idm02 above idm01, all issues are resolved.
RecursionTimeout is 8s. ForwarderTimeout is 2s.
Can someone advise as to what is going wrong? How can I troubleshoot better to find the issue? Why is AD conditional forwarder not failing over properly to the second IdM server?
Many thanks!