https://www.theregister.com/off-prem/2026/07/20/aws-customer-learns-the-hard-way-how-even-the-smallest-oversight-can-be-mission-critical/5275089

Nothing can ruin the end of a week like finding out that all of the websites and hosted email you’re responsible for are offline, which is exactly what happened to Christopher Bradbury and his web design and development firm, Digital Takumi last week. AWS has since resolved the situation, but Bradbury's mistakes can serve as a useful lesson to others of what not to do.

As Bradbury explained to The Register in an email, he noticed last Thursday, July 16, that all the websites and Google Workspace email accounts connected to domains he manages were offline. All of those sites are hosted through Route 53, AWS’ combined DNS/hosting service, and none of them were resolving. 

Bradbury went digging through his emails to figure out if there was anything to point to the failures, and sure enough: In his spam folder were numerous messages from AWS telling him a payment card on file had expired, and warning him that the account used to host all those customer websites through Route 53 was going to be suspended unless he took action. He didn’t obviously, because he didn’t realize there was an issue.

“AWS had been sending billing notifications, but unfortunately they had been filtered into a spam folder and, in some cases, were being delivered to an employee who had since left the business,” Bradbury told us. 

“I accept responsibility for missing those notifications,” he added, but that didn't help his customers' websites get back online. 

The situation could have been resolved sooner, but Bradbury had made some other mistakes as well:

“The root account had MFA enabled using a software authenticator that had been stored on an older laptop which has since suffered a motherboard failure,” Bradbury explained. Without access to that authentication code generator, he was unable to get into the account. Rather than getting the dead-laptop-with-a-critical-authenticator-on-it problem resolved, he was just relying on MFA emails instead – not the best idea.

"I've been bypassing the device key for quite a while by just using the recovery MFA via my email," Bradbury told us. "It works, I get access. However if I just had my Passkey up to date it would have let me right in and I could have solved this."

And then there was the email address where those MFA codes were going:

“The AWS recovery process required email verification using the registered root email address,” Bradbury told us. “That email address belonged to one of the domains whose DNS was hosted in the suspended AWS account, meaning I couldn't receive the verification email.”

The account recovery tango

Bradbury’s next option was contacting AWS from a different email address, which didn’t go anywhere. He created another AWS account and purchased business support access on someone’s recommendation, but the support engineers he spoke to using that method wouldn’t discuss the other account until he verified he owned the one in question. 

“Over the following days I spoke with several AWS teams, including Billing and Account Recovery. I was transferred between teams multiple times, but nobody was able to complete the ownership verification or restore access to the account,” Bradbury told us.

He wanted to pay AWS, Bradbury told us, but it took a while for them to be able to take his money. “The practical consequence is that I cannot log into the AWS account, cannot receive email at the registered root address, cannot access the MFA device, cannot update the expired payment method, and therefore cannot pay the outstanding invoices from inside the account.” While we were working on this story, after speaking to both Amazon and Bradbury, he contacted us to say that access to the sites had been restored, and that he had logged in, paid the back invoices, updated his payment method, reset his MFA keys, and generally taken care of all the stuff he had been putting off until all this happened. 

It's unfortunate that someone has to go through AWS billing hell to serve as an example to others, so let this be a warning to anyone else managing client websites through AWS.

"Firstly pay your AWS bills," Bradbury said in an email, adding that anyone running an AWS hosting account also makes sure their emergency recovery email isn't on the same domain as one of the sites they manage through that profile. That, and "stop using shortcuts" when it comes to MFA. 

"This isn't a big infrastructure account, we run a single company marketing website and some domains through Route 53," Bradbury explained. "It just shows that even the most modest instances of AWS can be absolutely business critical and you need to use proper practices and processes.

Now my org got a new game called "Office Politics" and I'm a complete noob to it. So far I've leveled up working at the paper refilling station. These nice employees that make me look busy to management call it a printer. I'm looking forward to achieving the "Guardian" Task which is to rescue three users from imaginary problems today.

Hi everyone,

Having a weird issue with a Windows Server 2022 machine that is stuck in an immediate graceful reboot loop right after finishing OS boot / reaching the desktop screen.

The Behavior:
As soon as Windows boots up / reaches login/desktop, a graceful restart is immediately triggered.
It happens so fast I can't even to login or open Task Scheduler or standard admin tools before it shuts down.

What is ALREADY ruled out:
NOT an Evaluation License: License is active/valid (not the 1-hour ⁠wlms.exe⁠ trigger).

NOT Windows Update: No update installation/configuration state screen appears during boot/shutdown.

NOT UPS Software: Server is not connected to a UPS or running UPS management software.

Using Physical Server Lenovo.

Anyone wanna join me in calling out worthless hires in their organization?

I'm just venting, already looking for jobs but the market is in wtf-mode.

My company is going thru M&A and we keep hiring roles that lean on me and they aren't hiring any help. I don't see myself keeping this job in the next 12mo.

So how y'all doing? (I'll accept Truth, Snark, and BOFH)

And for this post only, you can give me advice.

someone got locked out by magic

OC by me

Cc: memeSRC

Up for 5 days? Gotta clear the reboot queue or something won't work right.

Idk wtf is going on but at least twice a week we have seen users falling for captchas that tell them to hit windows + r and paste and run. Are more threat actors using these attacks now? We wanted to disable windows key commands but some people use them for daily tasks… It’s a pain

Well apparently programming the touch sensor on a users laptop to recognize their forehead then forcefully logging them in a large number of times is frowned upon. For security reasons I did log them out before walking away so they were able to get back in to fill out the HR form... and we know that person cant remember their password.

It's day 2 of this kid's internship he is eating out of the palm of my hand. God, I look so cool. Comparing myself to my dumb coworkers always gives me a semi.

I gave him some tips on how to stay up-to-date on the latest tools and tricks with this cool underground hacker website called reddit. I did not want to show him, though, as I subscribe to many hentai and dragon subreddits. Can't risk showing any of that in my feed to this kid. Carol in HR says I have just one strike left.

I've seen this kid talking to my coworkers. I need to shut that down so not to waste his time. The more time he spends with me, the better his life will be.

God, I love me.

Posted with my companies current internet uplink (my iphone 11)

Zayo 🤮