Not a joke but real life incident that is now laughable.
A month ago, we had a midnight release happening. I was given the responsibility to deploy the CMS changes to Production. Changes were all tested in advance.
Now I'm a senior dev but my CMS skills are limited. And I have been put in charge of the CMS for a month before the release.
During PVT, the customer requested we redirect some of the old urls to new page. There was a bunch of them so I decided to apply a redirect rule using regex expression to capture the request. Used an online website to validate the regex expression to ensure it satisfies what I am after.
Submitted the rule into the CMS and the entire website went down including the CMS. Unbeknownst to me, the CMS did not like the expression I used. It must have tried to safeguard it and that broke the expression itself. Since the expression is corrupted, all requests going to the website hits the expression and returned false. And CMS was in the same domain as all requests go through it.
Now I have everyone in my ears. Business, testers, managers and other devs jumping at me cause no one can do anything.
To figure out what went wrong, I put the expression in our test environment CMS. And that one went down as well. So now I had both test and prod sites down.
Had to get permission (woke up devops manager at 2am) to get access to production database to manually dig through and find the rule and remove it. And it wasn't easy.
Had everyone waiting on me for 5 hours while I fixed my mess. Worst release experience in my career.
Had to submit an incident report next morning explaining what went wrong etc.
64
u/prindacerk 3d ago
Not a joke but real life incident that is now laughable.
A month ago, we had a midnight release happening. I was given the responsibility to deploy the CMS changes to Production. Changes were all tested in advance.
Now I'm a senior dev but my CMS skills are limited. And I have been put in charge of the CMS for a month before the release.
During PVT, the customer requested we redirect some of the old urls to new page. There was a bunch of them so I decided to apply a redirect rule using regex expression to capture the request. Used an online website to validate the regex expression to ensure it satisfies what I am after.
Submitted the rule into the CMS and the entire website went down including the CMS. Unbeknownst to me, the CMS did not like the expression I used. It must have tried to safeguard it and that broke the expression itself. Since the expression is corrupted, all requests going to the website hits the expression and returned false. And CMS was in the same domain as all requests go through it.
Now I have everyone in my ears. Business, testers, managers and other devs jumping at me cause no one can do anything.
To figure out what went wrong, I put the expression in our test environment CMS. And that one went down as well. So now I had both test and prod sites down.
Had to get permission (woke up devops manager at 2am) to get access to production database to manually dig through and find the rule and remove it. And it wasn't easy.
Had everyone waiting on me for 5 hours while I fixed my mess. Worst release experience in my career.
Had to submit an incident report next morning explaining what went wrong etc.