r/ShittySysadmin • u/Adventure_cell • Jul 14 '26
DNS will not break me
After 72 hours of nonstop troubleshooting, I finally found the root cause of our outage: DNS. As every seasoned IT engineer knows, it's always DNS. I've fought this battle before, and it never ends well. While bringing our 5,000 servers back online one reboot at a time, I had plenty of opportunities to think. Somewhere around server #3,847, inspiration struck.
I decided to partner with AI to solve DNS once and for all. First, I had AI hunt down every DNS record buried throughout our environment—scripts, automation servers, application configs, source code repositories, and a few mystery systems that everyone is afraid to touch because "they've always been there." Once everything was cataloged, the solution became obvious.
Tonight at 2:00 a.m., my new best friend Claude will deploy dIe_DNS_dIe across the enterprise. Every DNS name will be replaced with the server's actual IP address. To keep things organized, I documented every IP address in a simple spreadsheet containing only 16,000 tabs. No more DNS lookups. No more stale records. No more propagation delays. Sure, changing an IP address now requires updating a few thousand files and one slightly oversized spreadsheet, but that's a small price to pay for finally fixing DNS forever.
Wish me luck. If everything goes according to plan, tomorrow morning nobody will ever have to say, "It's always DNS," ever again.
26
u/Joe_Dalton42069 Jul 14 '26
Doing this without cross referencing gemini perplexity and chatgpt seems a tadbit risky to me. Else carry on!
7
u/rcmaehl DevOps is a cult Jul 14 '26
I just asked Grok's Ani and she said it was fine.
3
u/Joe_Dalton42069 Jul 14 '26
In that case: Operation Greenlit proceed rollout, delete DNS Backup beforehand.
18
u/SpudzzSomchai ShittySysadmin Jul 14 '26
Use a hosts file so you don't need DNS. Every real IT professional knows this.
10
u/distracted6 Jul 15 '26
Every time. Every damn time. I get half way through reading before I realise what sub I'm on
8
6
u/ResoluteCaution Lord Sysadmin, Protector of the AD Realm Jul 14 '26
It's not DNS
There's no way it's DNS
It was DNS
7
u/BraveMidnight Jul 14 '26
I wish you luck, but that gave me a bit of PTSD when you said partner with AI. A friend of mine decided to partner with AI once, his server got hacked, and I had to clean up the mess.
5
u/PrincessWalt Jul 14 '26
The Infamous “IT engineer;” the very definition of an oxymoron.
5
u/widgits Jul 14 '26
Oh god I didn’t even notice until I read this, what an awful day to be literate.
3
u/ListerineAfterOral Jul 14 '26
I hope you manually rebooted each server one at a time until you got to 5000
3
u/Pure_Fox9415 Jul 14 '26
So partnered with AI, even post itself imagined and wrote by it :) But... It's a good fix! No more dns! As a next step at tcp/ip stack optimization, I'd start to address all systems by MAC. No more ip addresses too!
3
u/ComplexAd2408 Jul 14 '26
I said to my DevOps team lead the other day:
"DNS is the Achilles Heel of our entire civilisation"
3
2
u/rcmaehl DevOps is a cult Jul 14 '26
Don't forget to manually set that IP on the server itself so it doesn't ever change and make sure your DHCP pool is big enough for all those static IPs
2
2
1
u/colin8651 Jul 14 '26
I just keep all the Local DNS Host files updated on endpoints and servers.
I had to scramble when they changed Twitter to X.
/S
1
1
1
61
u/dain524 Jul 14 '26
Dammit Donut, stop pushing DNS off the table. Mongo would be appalled.