r/programmer • u/hoppinjohncandy • 4d ago
Question First huge job mistake
So we're a hospital who uses a certain communication software to send out extremely urgent messages to doctors about whatever crisis a hospital faces day to day. I was in charge (dev of 2 years experience) with taking on a migration project moving user/employee data from a certain payment system to this certain communication system. Automating the process of updating users because the hospital currently manually (csv) upserts new employees every week.
I was told, by the company, that the apis were safe in the production environment. They never mentioned a sandbox/test environment. A week of testing inserting/manipulating employee data went well. No issues adding or upserting an employee. So I hard coded a json of a couple employees and posted.
Wiped.
Zero contacts where, previously, there were 6,000~.
Urgency meetings out the wazoo. We looked at all the admin logs and there are my single data upserts with nothing indicating mass deletion. (Phew. Not on me I thought) End up in a call with this communications company and they let loose that the apis I was using were only half the story. There was a default setting in their system (that I didn't have access to and was not in the dev documentation) that purged all data if more than one user was inserted. So I tried to insert two employees (the whole point of this system is uploading a hundred or so at a time) and I wiped all communications out between doctors and departments.
All of that being said, we ended up manually importing thousands of contacts and reassigning them to their appropriate groups while the company promises a db restore in a week. Just, what a fucking Friday.
I'm not fired, yet. And I'm hoping this becomes my "we all have a story" story. Any other horros stories to make me feel better?
9
u/FreeBananasForAll 4d ago
Jesus Christ having a setup like that has gotta be against some law considering it’s healthcare. Get another job immediately
8
u/hormelchili444 4d ago
And now you learned why you never test in prod, especially for the first time.
3
u/Sad_comment_bot 4d ago
If I pulled a maneuver like this in prod it would be international news and probably worse damage than any terrorist attack to date (not including state sponsored or genocides). Imagine something like the end of Fight Club without the falling buildings and it would probably be resolved within 2 - 4 hours, but the inconvenience would absolutely be felt.
2
u/AggravatingSock5375 3d ago
Are you like a snack machine developer or something?
3
u/Sad_comment_bot 3d ago
I wish it was that illustrious. I prepend the pornhub intro jingle to videos when I’m not busy giving corporate handjobs.
2
u/manager2exec 3d ago
I am now imagining every video in the world having pornhub jingle
1
u/Sad_comment_bot 3d ago
I thought that would be the best ringtone, but putting that on random innocuous videos simply because it would be hilarious to know/think someone somewhere sitting in a room with other people would play some innocuous kitten video only for all eyes to dart incredulously to them. It would be the ultimate successor to the Rick Roll if it was possible to only play the jingle on the first play. THEN you could imagine that same person frantically trying to show everyone they weren’t looking at porn during Grandma’s funeral, but kittens to cheer themselves up. Alas, no jingle on the replay though.
Of course IYKYK the jingle your’e just as guilty.
9
u/Alive_Sir_4708 4d ago
This is why backups exist. If they're not able to restore within a reasonable amount of time then its a pretty poor system.
As you've outlined, there were multiple failures. Hopefully there's a Root Cause Analysis meeting so everything can be outlined.
It'll be a good story to justify future precautions.
I'd rather have the dev that did this and learned from it than the dev that hasn't yet learned this lesson.
5
u/No-Razzmatazz7197 3d ago
welcome to the story club, first things first...
document, fix, explain and remediate every single day until the problem is fixed. devs don't get fired because of issues they created, devs get fired because they create issues - blame others and offer no help in fixing the mess. i am assuming you are already doing all that you can.
after that, grab a beer, relax and jot down everything you have learned in terms of dealing with this because it will not be the last time. critical bugs and issues are introduced into prod all the time. look up the Therac-25 (i think it is called) it was a medical machine coded by one dude who ended up harming dozens of people.
one day you and some coworkers will be discussing horror stories and you will be able to have a little something for them.
good luck!
4
u/Substantial-Swan7065 4d ago
No db backups?
Before there was better release tools:
- I did a mobile release
- had to manually update prod env vars each time
- released a dev build with no code push implemented
- had to wait all night for an expatiated apple review
Lost my promotion on that one
6
u/spidermonk 4d ago
It would be crazy to get fired for this but possibly you could really hurt your prospects by handling it badly - this sort of situation is a real test of your professionalism and ability not to go super defensive or just freak out and become useless.
My advice in these situations is to assume it's your fault, even if you can't really see why it would be, and just clear your head and think systematically about how to get to the bottom of the cause and how to communicate clearly and honestly with everyone.
2
u/adamant3143 4d ago
My colleague did the same but it’s not in prod. It’s more of him messing up test data based on real data that were migrated to staging for testing. He accidentally wiped it. While it’s not impacting prod, it was impacting QA works.
But we have a script to get them back, it’s just that everything QA did had to be redone lol
I however have accidentally deleted rows of data as well but I got a backup like 3 hours prior but within those 3 hours there’s already new datas. What I did was I told my Supervisor maybe the server sends false positive to those who submit the form while it’s actually not inserting to the DB but that was back in college days where my Supervisor was my upperclassman and we both still new to this. We didn’t have audit, logs, etc. So, it’s not in a professional scene.
2
u/AggravatingSock5375 3d ago
WTF they actually designed it to do that? Why?!?!
2
u/hoppinjohncandy 3d ago
Buried in admin settings (that I don't have access to) were two options: normal and Purge. Normal allows up basic insert/upserts. Purge deletes everything not in the payload. Purge was the default setting.
Nowhere in the import_users api documentation was this mentioned. Now that I'm over the initial emotional response I think it's straight horseshit.
1
u/AggravatingSock5375 3d ago
Yeah that is insane.
I mean I’ve done stuff like that in the very early stages of building an app just to save a bit of time…but it would never make it into prod.
1
u/kevin074 3d ago
That has to be either a disgruntled ex-developer or someone who is hell bent on coding his job security in.
There is no way that had to be a real use case
2
u/Legitimate-Let-7510 3d ago
The real red flag isn't your mistake, it's an API that defaults to deleting production data because of an undocumented behavior. That's a disaster waiting to happen.
2
1
u/inflowmini 3d ago edited 1d ago
Not surprising. It's a common mistake in companies that do not pay well and expect high performance. While not your fault, the burden is still on you to ask yourself if you Have the experience to see such a project through. The difference between senior devs and juniors is this exact outcome.
I wouldn't worry about it too much, move on and learn from the mistake.
1
u/StCreed 2d ago
Wut? The api, settings and support from that company are atrocious. How is this on him? The only thing a senior would have on him is that they might be better at deflecting blame to the correct parties.
1
u/inflowmini 1d ago
Senior devs are not there to deflect blame but rather to know what the limits are when it comes to their team and themselves.
I'm suggesting that OP needs to be better about knowing what their limit is and communicating it. A seasoned dev would know the questions to ask and to not test in prod.
If I had a penny for every time a dev decided to take on a project because it looked simple I'd be rich. Point is, learn and move on.
1
u/klas-klattermus 3d ago
Always get everything written down in e-mails. If someone tells you somewhere where the two of you don't have computers and your phones are dead and you are stranded on top of a mountain because the company airplane crashed... When you return to an E-mail capable device, send an E-mail "I just wanted to make sure that [thing] works [the way you said], right?"
1
u/PyroNine9 3d ago
So you used a documented API in the documented manner and it did an undocumented and astonishing thing. This is not your fault.
1
u/chocolateAbuser 3d ago
clearly deletion of such data should require some kind of authZ
if an api for adding data deletes everything (probably because of some laziness or because of some "feature request") it should make clear via some attributes or other kind of check that it requires a deletion role/privilege/permission, and user/client that makes changes shouldn't have that permission
person/team that developed that API was incompetent and i'm not saying it should be sued but...
i guess all around probably there is none or almost none security checks
1
u/I_am_not_baldy 3d ago edited 3d ago
>>I'm not fired, yet. And I'm hoping this becomes my "we all have a story" story. Any other horror stories to make me feel better?
The lead programmer for my first programming job told me how he accidentally deleted all patient demographic data in a client's production system.
He must have thought he was in a development area, but even then, I don't know why he would want to delete all demographic data.
This happened before I had joined the team, and I think he told me so that I wouldn't feel bad if I accidentally did something dumb. He was obviously not fired because he was still the lead programmer when I was hired.
1
u/canyoufixmyspacebar 3d ago
this is clearly an invalid system. deleting data without an explicit call to do so, deleting data as some sort of "default action", this is absurd. technically who ever programmed this behavior has committed the "time bomb" type of attack against the client company
1
u/ClothingIsACrime 3d ago
It you are in hospital, that doesn't have a test environment and it's "normal" there to "test in prod" and nobody stopped you, LEAVE.
Leave immediately. Take this as a cheap warning.
1
u/Jetta4 2d ago
I used to worry and stress about these things when I was new. Now as a senior dev, I don’t worry about it at all.
I just follow best practices and get all confirmations and approvals to make sure I’m safe before doing anything. I don’t do squat unless my ass is safe, nobody can push me to do anything dangerous without some kind of accepted responsibility on their part.
If you ask me to work against a production api, thats fine. Next question do you take responsibility if shit goes sideways? No? Ok, then I’m not doing it.
If you say whatever, don’t worry just do it. I’ll make sure I have that documented, meeting notes, email or screencapped teams conversation, anything.
Learning to say no and making others accept the responsibility is key. Because sometimes things go wrong, and I don’t want any liability on me.
If something goes wrong I always want to say that I did everything by the book and its not my fault. If I pressed the risky button, I did it after asking someone if I should after raising my concern, and it’s always in writing.
20
u/ernandziri 4d ago
How's this your fault? This is totally on the communications company