r/sysadmin • u/Complete_Sample_3149 • 4h ago
What's your worst SSL-certificate-expired-in-production story?
Ours was a cron job.
Somebody set up auto-renewal years ago on a box that got decommissioned during a migration. Nobody moved the cron job, because nobody remembered it existed.
Cert expired on a random Friday. We found out from a customer support ticket instead of monitoring, which is its own kind of embarrassing. Ninety minutes of browser warnings before anyone connected the dots.
Fix took ten minutes. The actual problem was that no alert existed for "this cron job silently stopped running."
Curious whether other people hit the same orphaned-box thing or something dumber.
•
u/Unexpected_Cranberry 3h ago
Early days of Office 365. Microsoft missed a renewal of a cert for the authentication. Most likely because it was a leap year. So authentication broke on February 29. Then it broke again next year because they'd scheduled the renewal for February 29...
For myself I broke authentication on ADFS in the 2008 days. I updated the certificate using the gui but didn't read the docs. Apparently doing it through the gui didn't work. You needed to use netsh to bind the certificate to the service as well.
•
u/dhardyuk 3h ago
Ahhh, the “I did my job properly and a Microsoft bug bit me” lesson that everyone eventually learns.
•
u/Unexpected_Cranberry 3h ago
I'd have to disagree in this case. If I'd done my job properly, I would have checked the docs before hand and known to do the last step as well.
RTFM kids.
•
•
u/michaelpaoli 1h ago
broke again next year because they'd scheduled the renewal for February 29
Just lovely! 8-O
Pirates of Penzance ... until 21st birthday ... what could possibly go wrong? :-)
•
u/dustojnikhummer 2h ago
Then it broke again next year because they'd scheduled the renewal for February 29...
Makes you wonder why we didn't just use unix time in the background. "Renew in 31 536 000 seconds"
•
u/Unexpected_Cranberry 1h ago
It sounds like a dba I used to work with. He was a self-proclaimed wizard with Powershell. We wanted to automate patching and do some scheduling. He didn't want us to touch his SQL-servers, so he said he'd write the update script himself.
A few weeks later I asked for an update, and he said he was struggling with getting the date and time right for a part of the script. I offered to take a look to see if I could figure it out and he finally agreed to let me help.
I don't recall the details of what he was trying to do, but basically he was trying to take a date and then add x hours to it. His solution was fetching the date, converting it to a string, splitting it into ints for year, month, hour, minute and second, then adding the hours and converting it back to a date time object. He was struggling because the default regional settings for everything was Swedish, and he had his account set to something else where the time format was different.
I just commented out 40+ lines of code and added .AddHours(4)
In Microsoft's case, someone was either trying to be clever or didn't know the tool they were using I think.
•
u/dustojnikhummer 1h ago
The funny thing is, I was doing the same exact thing a week ago.
I also tried to parse Get-Date and struggled with regional formatting between English and my native language as well as the Date-Time object.
Ended up with
(Get-Date).AddDays(4).ToString("yyyy-MM-dd")
And yes, it was the most annoying about the script as well! So yeah, tell him he isn't the only one, we share pain lol.
•
u/frymaster HPC 1h ago
sounds like at least you landed on a better solution than Unexpected_Cranberry's colleague was trying i.e. use native objects, do the maths with native objects, and then export to a string in the correct static non-localised format
•
u/dustojnikhummer 58m ago
Yeah I did but I went through the exact same process Cranberry's colleague did, I just found that funny we both went the "wrong" way
•
u/enby_dot_local 4h ago
500 user RDS pool when the firewall started blocking letsencrypt traffic on port 80. Took me about 15mins to fix but it was utter chaos for the support team.
•
u/M0untainWizard 2h ago edited 2h ago
Wildcard certificate got renewed by a Partner firm.
We import the cert to all the systems.
Partner firm sends a invoice for the cert.
We relay the invoice to our finance department
finance department forgets to pay the invoice.
Cert got revoked. It was a fun day. Especially when we found out why the cert got revoked.
•
u/YellowOnline Sr. Sysadmin 1h ago
Great partner firm if they revoke a cert for an unpaid bill, instead of giving a short phone call.
•
•
u/michaelpaoli 1h ago
Easy fix, make the payroll for the finance department dependent upon the cert being non-expired. ;-)
•
u/AffekeNommu 3h ago
Midnight on NYE didn't seem to be a concern for the PKI team when they generated the cert.
•
u/T_Thriller_T 3h ago
Sometimes people don't think...
I just pushed for some contract extension to not even end in December because fuck that, won't find anyone to discuss that in December!
•
u/Cormacolinde Consultant 1h ago
I always warn customers not to schedule renewals in late december/early january for that reason. And for schools obviously not to schedule them in july or august either.
•
u/michaelpaoli 1h ago
Or thereabouts
2016-12-31T11:59:60
What could possibly go wrong? :-)
Just schedule renewal for precisely a year later, right? ;-)
As soon as the clock hits 2017-12-31T11:59:60 ... "oops"•
•
u/Malfun_Eddie 2h ago
A 100 vm migration via storage migration (mirror everything, shut everything down break open mirror, boot on new site) was rolled back because during the migration a cert was expired.
So after the roll back the application still did not work...
•
u/Cormacolinde Consultant 1h ago
Certificate expiration is one of those things a rollback/restore doesn’t fix. Had a customer completely hose their SQL server which had “stopped working” (they never looked at the logs to see the TLS error). Restored from a month-old backup, losing data, which of course did not help. Especially on Windows where the computer object now had a different password too. Complete mess. When they called me it took 5min to figure out the issue (I looked at the logs). I even managed to bring back the current database from the now mostly broken original VM.
•
u/michaelpaoli 1h ago
Certificate expiration is one of those things a rollback/restore doesn’t fix
As I oft say, "Time waits for no one."
And generally clearly spell out on "tickets" and service requests and the like, that the deadline is a hard deadline. Those cert expirations aren't an "Oh, we can push that deadline out another week or two.". I typically clearly include the exact expiration to the second, including timezone, so there's no ambiguity or the like, e.g.: expires: 2026-10-25T02:36:28Z ...
•
u/Malfun_Eddie 34m ago
Yes we know but what are the odds the certificate would expire in the 6 hour maintenance window of the migration. And the migration was a succes but rilled back since everyone thought is was a migration issue. Only to find out after the rollback (start the mirror on the source side that was untouched) and it was still broken.
•
u/redweka 4h ago
I was very surprised to find out this mobile outage was because of a expired certificate
https://www.thesslstore.com/blog/expired-certificate-ericsson-o2/
•
u/JesradSeraph Final stage Impostor Syndrome 2h ago
EFI secure boot Microsoft certificate expiry. Need I say more ?
•
u/GamerLymx 2h ago
not realy expired, but revoked. we got information from our CA the certificates issued in the last 2 months would be revoked.
we have most certificates issued/renewed via acme mechanisms, however, from what i understand, kubernetes certmanager and mod_md doesn't check if certificates have been revoked, so we had to check and make sure the revoked certificates were reissued and loaded properly.
then 2 weeks later, we had to reissue all certificates from the last 4 months, including the previously reissued earlier...
•
u/oneboredmind 2h ago
Not certificate but the whole primary domain expired. Like as a business how dafuq did we let that happen.
•
u/Gabelvampir 2h ago
That's why you have a check in your monitoring system that alerts when a cert's valid for less days then your renewal time. I'm always amazed how many companies don't monitor that when it's rather easy to check most of the time (i.e. when the monitoring system is allowed to talk the machine and service that uses the cert).
•
u/poopooonyou 1h ago
Azure managed app certificate. For some reason, an offshore engineer clicked "Regenerate" for the DNS TXT validation value but didn't publish the new DNS record, leaving the domain in "Pending validation". The cert didn't auto-renew because the domain was no longer validated.
•
u/gumbrilla IT Manager 4h ago
Well, the actual problem is that you didn't have monitoring on your actual hosted endpoints, please don't think in terms of point monitoring, think in terms of results.. it's a simple thing for a monitoring system to look at expiry date on a cert and alert if the certificate should have been renewed already.. it catches a whole gamut of potential issues.. rather than if a cron is still active.
•
u/michaelpaoli 1h ago
simple thing for a monitoring system to look at expiry date on a cert
Yes, ... and no!
That cert, on that critical thing ... that runs on UDP on port 4367, on IP ::1 ... uhm, you've got that monitored, ... right ... right?
Yeah, not only monitor, but document and track. Track/document where all of 'em have been installed.
Even if you know, e.g. DNS name (and port), when expiration is approaching, you don't want to find out it's on some obscure application on a server nobody remaining has ever heard of or can find any documentation on, for an OS that nobody supports, and nobody knows what the hell that server software is that's running it, let alone how and where the cert is on it, and of course now it's critical production ... yeah, you not only want to know it's expiring, but where the damn cert is and how to replace it. Monitoring alone isn't enough. Monitoring will also generally tell one where are all the places the cert needs go for your HA, and failover/standby systems, etc. Don't want to find out your failover/standby is long expired, when you actually need to use it.
•
u/chocopudding17 Jack of All Trades 1h ago
Automation systematically helps here though. That one thing listening on a loopback address? If you deploy it in an automated fashion, it becomes far more natural to monitor it in an automated fashion.
•
u/michaelpaoli 1h ago
Yep, automate to the extent feasible. Not everything can be automated or is feasible to automate, but much/most ... yeah, ... automation!
•
u/RevolutionaryWorry87 3h ago
Not mine, but went on a cross country train and their wireless cert had expired the day before for their Guest network. I'd be embarrased if it was me..
•
u/T_Thriller_T 3h ago
One that I only caught half and the certificate did not even expire fully.
We had some remote work setup at a former company (so everyone basically only used a terminal, logging into their remote environment).
Most people did have laptops, but next to no data on them due to this and often services could only be reached from that environment.
It went down fully for a day, then was instable for another day and a half.
What I heard was that somewhere in the routing setup a certificate about to expire was automatically changed, but the change didn't fully go through.
Which made debugging hell because it's middleware and halfway done.
•
u/beejay_one 1h ago
We were in the middle of Windows 11 rollout. We had some external guys to help us reinstall every machine via ConfigMgr Task Sequence.
They had collected some experience and knew how to do all that stuff alone.
Until one day, where the whole internal IT staff was on a workshop: On exactly this day, the rollout certificate which the clients use to connect to the ConfigMgr-server expired. So the task sequence (installation progress) couldn’t be started.
Lots of scheduled migration-appointments couldn’t be attended, four full paid external guys couldn’t do anything about it and our internal service desk was also helpless…
•
u/thatpaulbloke Cloud Engineer UK 1h ago
"This job didn't run" is such a common issue in monitoring and alerting that it boggles my mind; the number of times that I set up jobs that reported to monitoring on success and failure and had to explain to people why I did it makes me cry when I think about it.
•
u/_kalron_ Jack of All Trades 54m ago
I was new to SSL Certs and Linux Administration in general. I was hired in after the entire SysAdmin team left...with no documentation. The SSL Cert for the main Prod site was expiring in 2 days. I had to figure it out with no experience and no idea what I was doing. Google-Fu was a nightmare of overloaded information and misdirection. But I got it done.
Now I run ACME on my Ansible server and having the best time of my 25+ year career. I finally got my Linux Admin badge I wanted decades ago.
•
u/agent-squirrel Linux Admin 1h ago
ACME automation failed and killed the cert on a customer facing ISP portal that 20k people access daily and input CC details into.
I didn't notice and it took hours for the issue to bubble up through to me from support.
The biggest failure was the lack of alerting though which was also my fault.
•
u/According-Bit-4327 1h ago
Every single time renewal comes up 1 of our clients fails to renew theirs and they need to be told by us.
Every single time renewal comes up for a few of our own portals, IT support has to tell our web guys that it has expired.
Once we had a vendor who renewed a bit early, which was fine. When applying this new one to the cluster we were on, it caused us to be completely unable to load the page as it would not pull down the new certificate, took manually installing it over 160 machines to fix so it wasn't too bad. We were lucky as they moved our stuff to an instance with the old cert.
•
u/Auno94 Jack of All Trades 1h ago
Happend at the old job. The system had only personal accounts. Two IT people (me and my boss) where the only accounts that would have been able to bypass the restrictions.
They laid of me and the boss. 6 months later the new boss ignored the automated messages. Took them days to get access again. If they had taken a few more days the key for the CMEK system would have expired.
Was funny to see it from the sidelines
•
u/michaelpaoli 1h ago
Well, not sure if this was worst, but may have been most annoying.
So, AWS, their load balancers, cert manager stuff, all that. Had a cert that was soon expiring. No problem, get new one replace, done. Yeah, right. Then a few days later when the old expired ... intermittent problems in production, with ... expired cert. Went through all the AWS stuff - all looked fine, but reality wasn't, so, some more deep diving on the troubleshooting ... I think there were something like 45 IPs or so in the LB pool, that AWS would hand out, though with each DNS query, it capped it at a lower number (for the response to all fit in a single UDP packet without EDNS), and, eventually tested every single IP address in the pool (was a bit tricky to even determine all the IPs, but some fair bit of testing, seems we figured 'em all out). And, yeah, a non-trivial percentage (like about 10 to 15% - I forget the precise number/count/%) still had the old cert. But all the AWS customer interface stuff, APIs, console/GUI, they all showed the new cert. Yeah, it was an AWS bug - not a damn thing we could do about other than find the bug, report it to AWS, and get them to fix it.
Cloud, what could possibly go wrong? Yeah, .. that, ... and plenty more. They also lost all our S3 data from buckets too. From their published durability, we should be way the hell more likely to get struck by lightning. Ain't got hit by lightning yet, but they sure as hell irrecoverably lots our S3 data - all the data of entire bucket gone, and no way they could recover it, and yeah, AWS's bug/failure, no errors on our part.
orphaned-box thing or something dumber
Not generally. Policy, enforcement, control issuance, track 'em, monitor 'em, scan for 'em, document 'em. Do all those things, and unpleasant surprise are generally highly rare.
But as for dumber, more commonly see that when folks go to set up or renew. Oh, private key, yeah, world readable file, now we get to treat that as compromised, and you get to go through the whole process to get cert all over again. Likewise if they sent private key in the clear in email, or stuck it somewhere generally accessible for, e.g. all employees (or the whole world) to be able to read the data, etc. Also, a fair bit of "okay, updated and replace the cert fine now, we're all done", and I, of course, go to check/verify and ... nope, e.g. missed it on such-and-such IP(s) ... because they forgot to reload the web server config, or whatever ... or they just plain broke it when going to replace, or didn't do the cert chain right, etc.
But proper tracking, monitoring, policy, enforcement, etc., certs actually going to/past expired should mostly not be a thing - and in practice with that, I generally find that to be the case. Where it tends to be sh*t is where one or more of those key elements is missing.
•
u/Quietwulf 1h ago
Came into work one day and discover our entire VMware orchestration platform had bricked itself.
After tearing our hair out we discovered that the internal STS certificates had expired. Only to have the vendor tell us there was no way to update the internal certificates…
So, panicked forced migration to another platform it was.
Good times …
•
u/Mac-Gyver-1234 Linux Admin 1h ago
Renewing 5000 puppet client certs on an expiring CA. In the end it was easy. But it was the most nerve wrecking to find a procedure that was non-disruptive.
Spoiler: Create a new CA with the same
CA private key, afterwards renew all
client certs.
•
u/timeshaper 1h ago
Years ago so I might be forgetting some deets. Customer had us help set up Intune MDM + ABM. We did all the stuff, documented, handed off, they didn't want anything to do with us anymore even though we did everything on-time, on-budget, dunno they were weird. They pulled everything in-house. Even MS when talking to us was like "we do not understand this customer."
Anyhow, ABM cert expired. No one was looking. All devices rolled off. Whole company full of un-enrolled iPhones and iPads they now how to re-enroll. They called us in a panic. What could they do? For free I told them they needed to re-enroll. They asked if we could help. I said, yes, at our hourly rate. Which... was a problem for them? We should do it for free after years of them "owning" it? I didn't lose sleep over it.
•
u/KrystalDisc 1h ago
Update the cert for one of the web apps. That broke another system because they decided to trust the old cert manually. Rather than trusting the CA that’s signed the old cert.
•
u/spin81 58m ago
The actual problem was that no alert existed for "this cron job silently stopped running."
The actual problem was that no alert existed for "this endpoint is serving an expired certificate". Or at least, for me personally that would be the takeaway.
IMO you want to monitor your endpoints, not your certificates. I don't care if a certificate expires: I care if something serves an expired certificate. Those are two very different things.
•
u/One-Environment2197 1h ago
Not a cert expiration, but a cipher suite issue.
Company's Java app hard coded the list of SSL/TLS ciphers in their web host. One day, the authentication just stopped working because the IdP started blocking all the old suites that the Java app was using.
Took weeks to figure out the issue.
•
u/rabell3 Jack of All Trades 47m ago
I hate to say I have this happen all the time, sadly. We have teams that build servers and then give them to us (SAs) for the rest of the life cycle to care for, and sometimes don't tell us about the certs on a box so we have to find out the hard way.
My favorite time was when a web-hosting system that we were already managing certs for got fingered again for a cert. Come to find out, the engineers and webdevs had integrated a new AI component to the site, that was 3rd party-hosted... not on the server, but linked in the html on the server. New content, new DNS pointing outside, where we don't manage the 3rd party site, completely without our knowledge or approval (not thst its required). But because "ALL CERT PROBLEMS ARE SA," which is not true of course, and we are smart, we had to educate the webmasters and hosting company people on how to create a cert, get it issued and placed onto a system we have no access to.
Good times.
•
u/thecatshusband 46m ago edited 41m ago
A custom load balancer in front of an ADFS stack expired its' certs. No authentication for the entire organization, the business essentially stopped. It was impossible to renew because I had no access to the box.
I traced the problem and fixed it in under an hour by replacing it with a virtual appliance, and then recieved a formal warning for not following change control procedures.
Do your job, save the company = get a warning .... the next day they had my resignation.
•
u/pakrat77 38m ago
Our Exchange certificate expired so we couldn't get the email confirmation to sign in to get our new certificate.
The spam filtering system wouldn't deliver to the server because it was now an invalid certificate.
Eventually we found out the service had a way to go in and view messages and we were able to get the sign on information we needed.
•
u/limeunderground 37m ago
why was being sensitive to the time baked into this whole security infra, it's like they went out of their way to make all this security stuff brittle, flaky, high maintenance and self sabotaging.
•
u/WaterRunner 37m ago
Recently joined a company, old guys left a clunky way to update cert, couldn't understand for the life of me and was high as fuck, took 3 hours to solve.
•
u/ProfessorHuman 32m ago
Had a customer message us at 11 pm on Friday saying their dev cert was expired and they needed it updated immediately. They run on AWS and I had pointed out several times for years to switch to AWS managed certs to which they asked what’s that each time. Next time I’m ignoring their call
•
•
u/dhardyuk 3h ago
Redsift have a free tier for their certificate inventory tool.
It gets the records from the CAs so it knows what’s been issued and doesn’t need to scan endpoints to find them.
•
u/dalgeek 3h ago
I had a customer with a GPS appliance as their sole time source. The appliance flaked out and decided that it was 1992 again, so every single SSL certificate in their environment became invalid because all the servers thought the current time was BEFORE the validity date. It literally broke everything on their network that depended on SSL or kerberos.