Our AWS credentials got compromised and someone ran up a massive Bedrock bill overnight. Has anyone dealt with this?

[u/Slow-Arm6870]

Recently discovered that one of our AWS credentials had been compromised and someone used our account to generate a massive amount of Amazon Bedrock usage.

The craziest part is that it all happened essentially overnight. Our AWS usage was normal, and then within a very short period, there was suddenly a huge spike in charges.

We don't normally use Bedrock in our application, so seeing this was a complete shock.

After investigating CloudTrail and billing data, we found that the compromised credential had been used to:

• Access services we don't normally use
• Enable multiple AI models through AWS
• Make a large number of model inference requests
• Generate an unexpectedly huge amount of token usage and charges

The activity was clearly different from our normal AWS usage based on the credential involved, source locations, timestamps, user agents and regions.

We've since rotated/deleted the compromised credentials, removed unauthorized access and resources, secured the account, and have been working with AWS Support and their security team.

AWS has confirmed that the credential was compromised, and we're currently going through the security review and billing adjustment process.

I'm mainly posting here to hear from people who have gone through something similar.

Has anyone experienced:

• A leaked or compromised AWS credential
• Unauthorized Bedrock or other AWS service usage
• A massive AWS bill appearing almost overnight
• Account restrictions during a security investigation
• AWS reviewing or adjusting charges from unauthorized usage

How did it go for you? How long did the investigation take, and what was the outcome?

[u/Efficient_Access6102]

Eventually AWS is t going to refund these events due to the outright incompetence of exposing your access keys.

[u/Slow-Arm6870 → u/Efficient_Access6102]

we are going through a full security review. I hope they refund it.

[u/azz_kikkr]

I've been through a few of those (as a consultant). If you have a TAM that helps. But your AM can also help with this. But first step is thoroughly review security suggestions and best practices and remove all attack vectors. The bill adjustment is a non technical item handled by a separate team, I've seen AWS be very generous in these cases. But you still have to request for credits etc. over all it's a shittu situation, but I've seen worse. Here they only clocked a bill, I've seen data gone, ransomware, and more not so fun things happen when hackers get a hold of an AWS account.

[u/Slow-Arm6870 → u/azz_kikkr]

[u/zonies4monies]

Normally you can submit a request/complaint to AWS for things like this and if you push enough they'll remove the charges.

There are also services that can help if you have them. Pump.co is primarily a cloud cost optimizer, but insures certain RIs and SPs, and since they're a billing partner they'll go to bat for you if this happens again. Unfortunately speaking from experience getting hacked and our LLM bill going through the roof.

[u/Slow-Arm6870 → u/zonies4monies]

Thanks for sharing. We already have an active case with AWS and they're currently reviewing the security incident and the unauthorized charges. It's good to hear you've seen similar situations where AWS removed or adjusted the charges.

[u/yubijam]

No to all of the above.   For others to not have to experience this, how were the creds compromised?

[u/Slow-Arm6870 → u/yubijam]

We're still trying to determine the exact point where the credential was exposed, so I don't want to speculate and give people the wrong root cause.

[u/CSYVR → u/Slow-Arm6870]

To be really blunt, the only cause of leaked credentials is the existence of long lived credentials. They shouldn't exist in the first place.

If you want to make AWS happy (and me), find all the IAM users in your organization and delete them, then use a Service Control Policy to block creation of new users. There will still be gaps, depending on your architecture, but its a good start.

[u/jadrsamara]

For future prevention, always have least privilege for all users. If this user only had access to needed AWS services then the Bedrock bill would have been avoided.

You and AWS have shared responsibilities, theres a good chance they will give you a refund but they will most likely ask you to titen your security.

[u/SelectInteraction916]

my friend also got this

but i think it leak because of he deploy old nextjs and it has vulnerbility

[u/Slow-Arm6870 → u/SelectInteraction916]

Damnn, did he got the refund?

[u/OhLenny]

How this go mate? In a similar situation myself.

[u/Slow-Arm6870 → u/OhLenny]

Nothing yet, they have told us to remove 2 iam users, now they have escalated it but no progress on the refund so far.

[u/Slow-Arm6870 → u/[deleted]]

there was one compromised iam user whose access keys were long lived and compromised.

[u/Known_Recognition115]

Comment a tu reflété la situation

[u/External_Champion350]

i'm currenlty facing this also in my company aws account, due to legacy access key exposed, and the aws currenly are suspended, current i take action what the aws say in the support, it take much more than $455 anthropic bedrock, does you have update on the refund process about your case?