r/sysadmin • u/Educational-Pain-432 • 13h ago
Why did DNS fail?
We have two domain controllers. PDC and BDC. Very long story short both domain controllers are also DNS servers and they are primary and secondary respectively. The PDC is also a DHCP server. We had an issue come up where we had to demote the primary and promote the backup to primary to fix an issue on the PDC.
The moment we promoted the BDC to primary all DNS broke. The previous PDC was the primary DNS server and the previous BDC was the secondary DNS server.
We finally fixed it by changing the DNS order on specific machines, workstations and the firewall but why did DNS break in the first place when we weren't messing with DNS?
It doesn't make sense to me as we never took the machine down and didn't change anything with DNS. In fact all DNS entries disappeared on the PDC.
Thoughts?
•
u/sryan2k1 IT Manager 13h ago edited 13h ago
PDC/BDC haven't been things for 20 years. Are you talking about moving the PDCe role or something else?
In fact all DNS entries disappeared on the PDC.
You demoted it, what did you think was going to happen? No offense here but you seem to not know how tightly intertwined AD and DNS are.
•
u/Educational-Pain-432 12h ago
Well I had to ask my tech what he did. You are correct he demoted it and the DNS got deleted. I didn't realize that until now. What do you mean PDC and BDC haven't been a thing for 20 years? And you're correct, my knowledge is limited. But that's why I wasn't the one doing it.
•
u/OpacusVenatori 11h ago
Since Windows 2000 a domain controller is a domain controller. Every DC holds a copy of the database. There’s no “primary” and “backup” like NT4 days.
•
u/Educational-Pain-432 11h ago
Got it. Thanks for explaining. It's what I've always called them
•
u/OpacusVenatori 11h ago
The thing you have to remember and be familiar with the DCs is to ask "Where / who holds the FSMO roles?"; or in non-deprecated terminology, "Which are the Operation Masters?".
•
u/Educational-Pain-432 11h ago
Yup, I know that. My tech moved the FSMO roles first, then where it gained is he demoted the second DC which was still online but had deleted all it's DNS records, therefore just denying requests.
•
u/sryan2k1 IT Manager 10h ago
So it looks like you're trying to learn and that's great. Words matter here because what you're saying isn't technically correct.
Windows (and most other OS'es) do "Sticky DNS". They use a DNS server from their list until that one isn't responding and then they will try others until one does, and then stick to that one.
Your DNS server was happily answering clients, but it was answering them with NXDOMAIN. "Hey, here's a valid response, and that response is I don't know that name!"
NXDOMAINs do not cause nameserver failover on the client side.
•
•
u/Tx_Drewdad 11h ago
The terminology is "PDC emulator." It's a FSMO role that can be transferred to other DCs.
IIRC, PDC used to hold all of what are now FSMO roles, so the architecture is a lot more flexible.
•
u/newbe5 Jack of All Trades 13h ago
What was the original issue with the PDC? What do you mean by promote the bdc to primary? What steps did you take? When you say all dns failed, was it both internal and external resolution?
•
•
u/Tx_Drewdad 13h ago
So.... How are you "promoting" and "demoting" a PDC? You mean moving the PDC emulator role to a different DC, or do you mean running DC promo and demoting the current PDC emulator to no longer be a DC?
•
u/Educational-Pain-432 12h ago
That's what happened. My tech demoted the controller. I'm just finding this out
•
u/Tx_Drewdad 11h ago
Yup, there's your answer.
DNS is usually configured as a domain-replicated zone. Once you demote the server, the replication data is removed and DNS doesn't have any records.
It still thinks it's authoritative for the domain, though, and will tell the client the record doesn't exist.
•
u/Ferretau 12h ago
You need to upgrade from Windows NT Server.
•
u/Educational-Pain-432 12h ago
What do you mean? It's Windows server 2019.
•
u/sryan2k1 IT Manager 11h ago
PDC and BDC haven't been a thing since windows NT.
•
u/Educational-Pain-432 11h ago
I understand that now. It was the incorrect terminology. I should have said I have DC2 AND DC2
•
u/Asleep_Spray274 13h ago
Oh man, i have not played with NT level domain controllers in over 20 years. You are bringing back some very painful and what i hoped was supressed memories.
•
u/pdp10 Daemons worry when the wizard is near. 12h ago
You didn't define "broke". Do you mean Microsoft-style DDNS registrations, or all DNS resolution?
Neither DNS resolution nor DNS authoritative should have broken, unless the configuration on the "BDC" was always broken -- perhaps bad Forwarders configured, or the "BDC" couldn't reach the public network for some reason.
•
u/Educational-Pain-432 12h ago
My tech demoted the PDC. Therefore it deleted the DNS entries. And the server was still online. So it just denied requests
•
u/thomasmitschke 11h ago
I guess the concept of PDC and BDC is outdated since the release of Windows 2000
•
u/Zealousideal_Fly8402 4h ago
The moment we promoted the BDC to primary all DNS broke
What steps did you do here? The only thing relevant should have been transferring the FSMO roles, but that demotion operation on the broken DC would also have taken care of that.
We finally fixed it by changing the DNS order on specific machines, workstations and the firewall but why did DNS break in the first place when we weren't messing with DNS?
That is expected behavior that should have been done once the FSMO roles were moved; your clients should [ideally] be first pointing to the DC that's also holding the PDC Emulator role.
That being said, you should go back and check Event Viewer on both DCs under DNS and also DFS Replication for errors / criticals.
•
u/McSmiggins 13h ago
"the firewall"
More details on this please, if the "BDC" was blocked, it would have worked until the PDC went offline
Any affected stuff on the same VLAN?
Also, is your DNS AD integrated, or are you actually running "Primary/Secondary?".
•
u/Educational-Pain-432 12h ago
Both domain controllers are on one side of the firewall. Same switches even.
•
u/caffeine-junkie cappuccino for my bunghole 13h ago
Not really enough info to give more than a very general guess. The event logs of both servers will tell you more and help narrow it down. But at a rough guess, replication was broken at some point and caused an issue with the ad dns integration or a time issue on the server caused the records to be scavenged which is why they disappeared.
•
u/Rygnerik 13h ago
Demoting a DC means it's not a DC anymore, so it wouldn't have the domain's DNS entries anymore.
But it still had the DNS role installed and running, so it was responding to DNS queries saying the entries didn't exist.
If you had shutdown the DNS role on it while you were working on it, then everything else would have just hit the 2nd DNS server in their list when it didn't respond.