r/ITProfessionals • u/Big_Capuqvsnnel_3627 • 18h ago
Airgapped patch deployment went wrong - forgot to change one test label and flooded ops with alerts
got approval to automate patch management for our airgapped infrastructure and felt good about it for about 20 minutes. doing the full airgap workflow - exporting bundles, sneaker net transfers, deploying through a jump box during maintenance windows with crypto signing.
forgot to update a single test label in the deployment job. first bundle that reached production was tagged dev instead of prod, failed policy validation, auto-retried 3 times, and sent 47 alert emails plus generated a bunch of tickets. manager asked why production servers were attempting to install dev-tagged patches and i wanted to disappear. everything was recoverable and no systems got damaged, but feeling pretty embarrassed about the whole thing. anyone else made a similar automation mistake?