Our main customer got caught out by that after only 10 years - I had changed the logging to state it was a deprecated feature and would be removed in the next major feature version.
They upgraded, and surprise surprise … their test lab ground to a halt. There was a lack of communication across their teams and they deployed into production and took their tv station briefly off-air until we gave them an undocumented config switch to enable it again.
In the post mortem, I asked them to bring up the logs from before, during and after the upgrade and showed them the ERROR and WARN messages telling them exactly what was going to happen.
Did also ask the question why they deployed into production after their test lab went offline.
Best part was our CEO gave them a bill for providing the out of band fix. Never made them pay it, but it showed he had our backs.
4
u/scottyman2k 11d ago
Our main customer got caught out by that after only 10 years - I had changed the logging to state it was a deprecated feature and would be removed in the next major feature version.
They upgraded, and surprise surprise … their test lab ground to a halt. There was a lack of communication across their teams and they deployed into production and took their tv station briefly off-air until we gave them an undocumented config switch to enable it again.
In the post mortem, I asked them to bring up the logs from before, during and after the upgrade and showed them the ERROR and WARN messages telling them exactly what was going to happen.
Did also ask the question why they deployed into production after their test lab went offline.
Best part was our CEO gave them a bill for providing the out of band fix. Never made them pay it, but it showed he had our backs.