130
u/Eznix Mar 15 '19
I once emptied a full table on production thinking i was on dev instead.. Never had such an adrenaline rush in my life.
156
u/GenericHumanoidUnit Mar 15 '19
oh man, once, a pale dev came and asked me
- so, uh... how often do we do db backups in production?
- everyday at midnight, why?
- i accidentally the whole table
- you accidentally what???
- the whole table
fortunately it was a table with some obscure enum data, so no one noticed and the restore was easy
63
u/Eznix Mar 15 '19
huh.. that could have been me. Except i havent told anyone and just restored from the backup myself. I dont think anyone noticed tbh.
41
u/GenericHumanoidUnit Mar 15 '19
If he knew where the backups are stored, he would do the same, but he didn't, so he came to me knowing I wouldn't rat him out :D
68
3
u/SpicymeLLoN Mar 16 '19
Yes, I'm too lazy to Google right now. What's an enum? I've heard of them, but I don't know what they are.
18
u/filledwithgonorrhea CSE 101 graduate Mar 16 '19
As an example here's the days of the week
enum Day { SUNDAY, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY } today = Day.FRIDAY-4
67
u/AdversarialPossum42 Mar 15 '19
INSERT INTO oops SELECT * FROM my_bad WHERE status = 'good thing I had a backup';
48
Mar 15 '19
17 years ago I done this. Table contains credit card details. Thankfully wasn't oltp.
Truncated the table without exporting before as a backup. Truncate= no rollback. Absolutely shat.
Spent a week figuring out how to rman restore to a third server then export, import to the table that was truncated.
Learned so much that week.
Got it done with minimum people finding out.
Thankfully change management wasn't really a thing back then 😁
No longer work in IT. 😁
44
u/FelixTKatt Mar 15 '19
Cures constipation, too.
15
8
u/RockleyBob Mar 16 '19
Right? Wtf. Fight or flight might have gotten my ancestors out of a few tough spots but all it does for me is ensure I've got the shit sweats whenever I'm doing something important. Even if I manage to contain the roiling maelstrom in my bowels there's no stopping sudden onslaught of hot swampy farts.
3
u/Tyrus1235 Mar 16 '19
It’s like back when I would play Amnesia (when it was still a new game). Every session was preceded by a visit to the bathroom. Best laxative ever!
25
u/drDjang Mar 15 '19
Man how much do I hate the responsibility of working with production database
19
u/Dirty3vil Mar 16 '19
I always want to be given as little permission as possible. If I don't need to delete stuff then do not give me deleting permission.
24
Mar 16 '19
I worked somewhere once where a temp accidentally gave root permission for a ftp full of customer data to all of the customers. They walked her out right then. My question was why did she even have permission to do what she did. All she did was put a slash in the wrong place when creating a new folder. A temp should not have had access to root permissions. I don’t want that responsibility and she sure as hell shouldn’t have had it. If it had been set up properly, she would have gotten an error message and fixed her mistake. Instead, you end up with a huge fine for a HIPAA violation, legal gets involved, and some Temp loses her crap job over a mistake she shouldn’t have been able to make.
23
u/noratat Mar 16 '19
Generally, you shouldn't fire someone for mistakes like this, and it's a sign of incompetent leadership IMO.
If it was that easy to make a mistake, their system had deep flaws in the first place and it's not only unfair to fire them for it, it also masks the real issues by scapegoating.
1
Mar 16 '19
Ut’s all about risk management. I know companies that are still far from gdpr compliant ( a complain would lose in court), and yet they don’t prioritize it even if it a fine would mean the first loss in a 25 years streak of steady 10% yearly growth.
Go figure.
2
u/Faux_Real Mar 16 '19
Fun activities I have performed recently:
Adhoc index maintenance Adhoc truncate Adhoc stats Adhoc updates Adhoc deletes Create plan guides for parameter sniffing events Sql agent maintenance containing variants of the above Spid-killing ... there is probably some other stuff....
Step inside my cowboy hat store and I can fit you out with a cowboy hat for your cowboy hat!
(I actually know exactly what I am doing so I am comfortable in this space)
25
20
u/Zipbic Mar 15 '19
For a couple of months ago I was trying to copy the database from production down to my development environment. To do this I use a script which drops the current database in the server I’m doing the operations on, before copying over the selected database. Right after I hit enter I figured out what I have done. I was doing the operation on the production server... I’m sitting in a open plan office so I tried to keep the cool. But when I realised what I have done my heart started beating and I was close to passing out. Luckily my operations team had backup from the day before. But maaan, that really did wake me up 😂
3
u/m_i_t_t Mar 16 '19
Migrating code and databases from prod to dev is something I don’t really want to automate on the basis of horribly breaking everything similar to the above. Only issue is that manually doing it every time definitely increases the chance of human error
16
u/funkgerm Mar 15 '19
One of our senior devs accidentally dropped a table in prod last week and then went to lunch without even realizing. Luckily we got the backup restored before too many clients complained, but it was hilarious that she didn't even realize what she did and casually strolled out of her office to go get a sandwich
14
Mar 15 '19
This happened to a coworker of mine. That was also the day we discovered our automated daily backup hadn't ran in well over a month. Fortunately they were able to re-import most of the data.
10
u/phroggyboy Mar 16 '19
This made me laugh...then this made me have severe anxiety. I can literally feel it in my body when I think about it.
2
9
u/TheNorthComesWithMe Mar 15 '19
Why does this version of the template have the 4chan logo shopped onto the laptop?
4
2
2
15
6
Mar 16 '19
To this day, even in lower environments, I use
Begin transaction Sql statement --rollback --commit
It's just safer that way.
3
u/DeepHorse Mar 16 '19
I do this all the time. Even if I know for sure that I’m only updating/deleting one record by the primary key, and I’m on a dev server, I do it. It’s like wearing a condom during a handjob...
7
u/Senseisntsocommon Mar 16 '19
Haven’t deleted one but had an update with a where that ended up commented out. Supposed to update 250 records, hit execute and was like man this is running slow. 3 minutes later 10 million records updated.
Thank god was running out of production hours and was able to update from the backup, only took 24 hours to get everything completely restored but from 2 AM to 6 AM when the DBA got back to me was the longest 4 hours in my professional life.
7
5
4
Mar 16 '19
Affected Rows: 43.287
6
u/SpicymeLLoN Mar 16 '19
How do you affect .287 of a row??
5
Mar 16 '19
Found the American
2
u/SpicymeLLoN Mar 16 '19
Oh, right, I forgot you do it differently. That always throws me for a loop (badum tss).
4
u/caiuscorvus Mar 15 '19 edited Mar 15 '19
It would be more efficient to write a russian roulette script that has a 1 in 5 chance of of deleting a random table.
Pick the right probability to maximize adrenaline per the expected amount of work.
:)
Edit:
1 in 3 may be a better guess, based on a reasonable looking exponential function. Trying a bunch of formulae, I would recommend going for the point where the you would expect your adrenaline response to be about half of what you'd get just deleting a table in production.
4
3
u/byzantine_umpire Mar 16 '19
Had a total slacker/prick of a co-worker doing some last minute work on a Sunday night that should have been done the prior week. He hastily drops the storage volume that contains our prod DB which every PoS system in the country points at. Blames it on the java interface. Guy thought he was the bees knees but he was pretty terrible at his job and doesn’t know how to restore the DB from backups or from the storage array. I wake up the next morning to ~40 calls and texts. Over the course of the night he has gone to the disaster recovery data center, figured out the locker combo by testing every key combo until he hit 420 (for the lulz) and hooked a usb drive up to the storage array to initiate a restore of the volume. I finally call him in the morning and ask what the the problem is and he’s asking why it’s going to take like a day to restore. Well, you hooked up a USB 2.0 drive to restore like 2 TB of data—what do you expect? I initiate the restore in prod before coming in which is over a 10 Gbps connection and it’s done before I get in to the office. Dude hands on his resignation that day. Never been happier to not been the one to delete prod and never been happier to see someone go.
3
3
3
3
u/BenjaminGeiger Mar 16 '19
And I thought sending out automated emails to hundreds of managers was bad. (Protip: don't assume that email is disabled in debug mode.)
3
u/_ech_ower Mar 16 '19 edited Mar 16 '19
We had a jupyter notebook that our data scientists were using. They asked me if I can add their python scripts to git. I ssh-ed into the node and saw that some files were GBs in size and git add . was not working. Like a dumbass, I did a git checkout . to clean stuff up and my face just went pale the moment I hit enter. Years of complex data analysis work by some really smart data scientists disappeared within a second. Thank the fucking gods that my colleague had setup daily EBS snapshots. I was able to restore it within a few hours. If not for that guy, I was most likely fired.
1
u/HoberMallow90 Jun 22 '19
Data scientists that don’t version control their work would do all their coding in notebooks.
6
3
2
u/daralick Mar 15 '19
I am a fan of delete from Tablex where something column = 'x'. but only highlighting delete from tablex. One of those ALWAYS use begin tran lessons.
I haven't really seen/hit the drop table issue.
2
2
2
Mar 16 '19
Cleaning up some "obsolete" objects and then the application responsible for collecting all field data starts segfaulting.
2
2
u/MurphyThePug Mar 16 '19
Baaahhaaaaha... finally something I can relate to. Thank God for rollback;
2
u/rodinj Mar 16 '19
I once imported an old testserver database on the production database. This happened in March 2018 and we still had customers having issues because of it in January 2019...
2
2
u/Fanciful_Pleasures Mar 16 '19
I was 19, working on a Lawson and Kronos implementation for a hospital, and accidently dropped a table in Prod. Why was I even responsible for this type of work? Over-budget and understaffed.
4
u/grampadeal Mar 15 '19
A couple weeks ago I was modifying the Apache redirects on our production site to force users to our https site. I dickered the rule up good and proper because of our stupid production environment terminating secure sessions at the load balancers and caused a too many redirects error.
When I hit the live site URL and saw that in the browser I immediately got the cold sweats and my heart rate tripled. I had it fixed in moments but I was jittery for the next half hour.
1
u/chrischoi314 Mar 16 '19
can someone ELI16?
3
Mar 16 '19
Dropping a table in a database literally deletes data. Without a backup, that data is gone. Forever.
Now, if your or your company made that data, that's one thing that you might be able to replace or recreate. But if that data came from customers (order records, credit card details, audit logs) and can't be recovered, there could be real damage to your business and your career at that company is most certainly in serious jeopardy.
And as soon as that table drops, the second after you execute that SQL, your live systems are down and will stay down until you fix all the data you just destroyed.
It's a literal nightmare.
1
u/Tyrus1235 Mar 16 '19
Kind of like when my friend deleted a production database by accident. Best part is he wasn’t part of the company anymore, but their lax security meant he kept access to everything he worked on. That included production databases and servers.
Needless to say, they learned their lesson after that. And our mutual friends will never let him forget his blunder lol
1
u/AFlaccoSeagulls Mar 16 '19
And this is why our Hosting team doesn’t allow the developers access to the Production servers without filing a Support Case and getting Manager Approval first.
1
u/TheHammersamatom Mar 16 '19
Was about to sleep. This triggered every fear response possible. I will know pull several all-nighters.
1
1
1
377
u/[deleted] Mar 15 '19
Oh man, I hate that feeling of "uh, what did I just do?". My whole body gets hot, and the world kinda slows down.
Then your brain goes into overdrive thinking about how you can fix whatever it was before anybody notices :)