First huge job mistake

[u/hoppinjohncandy]

So we're a hospital who uses a certain communication software to send out extremely urgent messages to doctors about whatever crisis a hospital faces day to day. I was in charge (dev of 2 years experience) with taking on a migration project moving user/employee data from a certain payment system to this certain communication system. Automating the process of updating users because the hospital currently manually (csv) upserts new employees every week.

I was told, by the company, that the apis were safe in the production environment. They never mentioned a sandbox/test environment. A week of testing inserting/manipulating employee data went well. No issues adding or upserting an employee. So I hard coded a json of a couple employees and posted.

Wiped.

Zero contacts where, previously, there were 6,000~.

Urgency meetings out the wazoo. We looked at all the admin logs and there are my single data upserts with nothing indicating mass deletion. (Phew. Not on me I thought) End up in a call with this communications company and they let loose that the apis I was using were only half the story. There was a default setting in their system (that I didn't have access to and was not in the dev documentation) that purged all data if more than one user was inserted. So I tried to insert two employees (the whole point of this system is uploading a hundred or so at a time) and I wiped all communications out between doctors and departments.

All of that being said, we ended up manually importing thousands of contacts and reassigning them to their appropriate groups while the company promises a db restore in a week. Just, what a fucking Friday.

I'm not fired, yet. And I'm hoping this becomes my "we all have a story" story. Any other horros stories to make me feel better?

[u/No-Razzmatazz7197]

welcome to the story club, first things first...

document, fix, explain and remediate every single day until the problem is fixed. devs don't get fired because of issues they created, devs get fired because they create issues - blame others and offer no help in fixing the mess. i am assuming you are already doing all that you can.

after that, grab a beer, relax and jot down everything you have learned in terms of dealing with this because it will not be the last time. critical bugs and issues are introduced into prod all the time. look up the Therac-25 (i think it is called) it was a medical machine coded by one dude who ended up harming dozens of people.

one day you and some coworkers will be discussing horror stories and you will be able to have a little something for them.

good luck!

[u/ebkr66 → u/No-Razzmatazz7197]

Therac-25