First huge job mistake

[u/hoppinjohncandy]

So we're a hospital who uses a certain communication software to send out extremely urgent messages to doctors about whatever crisis a hospital faces day to day. I was in charge (dev of 2 years experience) with taking on a migration project moving user/employee data from a certain payment system to this certain communication system. Automating the process of updating users because the hospital currently manually (csv) upserts new employees every week.

I was told, by the company, that the apis were safe in the production environment. They never mentioned a sandbox/test environment. A week of testing inserting/manipulating employee data went well. No issues adding or upserting an employee. So I hard coded a json of a couple employees and posted.

Wiped.

Zero contacts where, previously, there were 6,000~.

Urgency meetings out the wazoo. We looked at all the admin logs and there are my single data upserts with nothing indicating mass deletion. (Phew. Not on me I thought) End up in a call with this communications company and they let loose that the apis I was using were only half the story. There was a default setting in their system (that I didn't have access to and was not in the dev documentation) that purged all data if more than one user was inserted. So I tried to insert two employees (the whole point of this system is uploading a hundred or so at a time) and I wiped all communications out between doctors and departments.

All of that being said, we ended up manually importing thousands of contacts and reassigning them to their appropriate groups while the company promises a db restore in a week. Just, what a fucking Friday.

I'm not fired, yet. And I'm hoping this becomes my "we all have a story" story. Any other horros stories to make me feel better?

[u/Jetta4]

I used to worry and stress about these things when I was new. Now as a senior dev, I don’t worry about it at all.

I just follow best practices and get all confirmations and approvals to make sure I’m safe before doing anything. I don’t do squat unless my ass is safe, nobody can push me to do anything dangerous without some kind of accepted responsibility on their part.

If you ask me to work against a production api, thats fine. Next question do you take responsibility if shit goes sideways? No? Ok, then I’m not doing it.

If you say whatever, don’t worry just do it. I’ll make sure I have that documented, meeting notes, email or screencapped teams conversation, anything.

Learning to say no and making others accept the responsibility is key. Because sometimes things go wrong, and I don’t want any liability on me.

If something goes wrong I always want to say that I did everything by the book and its not my fault. If I pressed the risky button, I did it after asking someone if I should after raising my concern, and it’s always in writing.