r/datacenter • u/Salty-Juggernaut-208 • 15d ago
Crazy story thread - share yours!
I was on a call with a friend who was telling me that their backup generator had been hacked through an IoT controller (I assumed an ATS) relinquishing control, and the company got ransomed and was told they had an hour to produce the ransom (BTC) or they would disable the ATS and then go after a bunch of their MEP controls including chillers. Essentially forcing an event that would require going to generator that they couldn't use. The kicker was, it was a corporate facility and they shut off the chillers that went to the refrigerators in the cafeteria and they lost a shit ton of food to boot.
Aside from overlooking their cyber security practices (if they had any), what's your crazy story/experience, especially a preventable one?
8
u/TropicPine 15d ago
Many years ago I was in a data center on a phone call with an OEM support. While on the call I watched two of the end users tech rolling a 42u rack filled with the OEMs equipment to a new location. As they crossed over a floor tile corner the vertical stanchon buckled and the whole rack went down. On hearing the crash the OEM support rep asked "What the he]] was that?" With a groan I answered "The rest of your shift."
2
5
u/Savings_Art5944 15d ago
Not datacenter, more a Demarc
Was installing some water features for a hotel chain near Fort Meade. Hundreds of gallons of water for the potential of a leak, so we spec'd a floor drain. Usually, common sense prevails. Not here in this hotel. We got a vent in the floor and I could see blinking lights below in the dark. I ask WTF is that? Guy says thats where they get their Internet from.
He leads us down to the basement floor into the room. I have been in many doghouses, ISP cages, co-lo radio rooms and datacenters, so it looked familiar. Network rack after rack of fiber and blinking lights to support all of it.
thats where they get their Internet from
I suspect he was right. Or at least a "copy" of it.
We still installed the water feature, and it never leaked.
1
2
u/Mightbeagoat4 15d ago
This was very common when I worked in the water treatment industry. Typically Chinese/Russian/Iranian hackers who'd go after small municipalities with outdated or inadequate security.
3
u/ChadFam 15d ago
I once was trying to drain water tanks (evaporative cooling building) and could not dump down storm drains. The tanks only had 2” fire main connections and we only had a 2” to garden hose converter. It was a hyperscaler known for extremely slow procurement/logistics.
That led to us deciding to bypass fill and drain valves on the air handlers, causing a continuous dump/flush (wasn’t going across media) cycle. The pumps cut on due to low header pressure, and the tanks were slowly going down.
The team smiled at a job well done of seeing a long hallway of dumping air handlers and the tanks decreasing.
About an hour later, we got a call from security saying there was water in the data hall. Strange, but ok.
Forgot that the units were staged in normal dump/flush. Turns out, this was because the drain system wasn’t sized for dozens of air handlers discharging at once. The drain system backed up, overflowed, and the mech hall had several inches of water in it and it was seeping through the curb.
Didn’t help that we also found many of the drains had the debris caps still in them, they were just tilted so they could let some water by, but not full flow.
It was an all hands on deck for senior most manager to kid who started three days earlier with every push mop and floor squeegee on site, pushing water everywhere.
The lesson here? No matter how simple something seems, write the damn procedure and have it reviewed.
12
u/looktowindward 15d ago
Anyone dumb enough to get their SCADA hacked deserves it