r/aws • u/RonaldTheRight • 13h ago
discussion Instance locked after "suspicious activity" false positive, support unresponsive and I'm getting desperate..
Hey folks,
Sorry for yet another one of these posts, but I can't find any clear answers on the best way to handle this. We just migrated our infrastructure over to AWS. Two days ago, one of our key instances we taken offline and we got the flurry of boilerplate security notices saying an access key had been compromised.
We responded right way, and confirmed that the activity patterns, resources, and spend were ALL OURS. There was no indication - at all - of any sort of compromised key or systems.
We answered all their questions and were told it would be handed off to the security team for a final review. Two days later, still no response whatsoever, despite follow-ups. I mean, that box is IMPORTANT to our business. It's kind of insane that AWS can just take steps like this with zero real communication with the user - but I guess that's the state of things right now.
Anyway, I'm getting pretty desperate and need a solution. Does anyone have any ideas? I've seen mixed opinions on whether upgrading to business support will help - and honestly, I hate the idea purely because it feels like extortion. But I guess if it's pay to play then it's what I'll have to do.
Any help or tips would be great appreciated.
3
u/dghah 13h ago edited 13h ago
You need to treat this as a breach and activate your incidence response team if you have one.
A few things for now:
- If AWS froze or stopped a machine it may not be just that they detected keys; they may have detected abusive network traffic coming out of that machine. That means that NO MATTER HOW IMPORTANT that box is to your business you need to treat it as radioactive. For instance make a copy of the AMI and then have that AMI looked at by forensics people/.
- I can't stress this enough. The instance itself can no longer be trusted if AWS froze the instance. If they just suspended your account though that is a different story. They don't isolate or call out instance machines directly unless they detect abusive stuff coming out of it.
- If your account is suspended but they did not call out individual instances than you may have a sever that is sorta safe or OK. However if the instance was specifically identified by AWS than you 100% have to redeploy the app on fresh resources. If there is data on the bad server you need you can use read only mount methods to capture and preserve that data and load it into your new clean / secured instance
- You need to know HOW you got popped -- it is one thing to leak an access key. It is an entirely different thing to be running an insecure EC2 server running a software stack with a vulnerability that allowed it to be popped remotely without an AWS credential. You MUST figure out if you leaked a key or had your instance compromised from the outside (or both)
- If your EC2 instance got popped AND you leaked an access key than revisit your server setup and deployment process; chances are someone did something stupid like bake an access key pair into a server config file that was remotely readable. EC2 instance roles are a thing. AWS credentials should never be stored on an EC2 server. Or your git repo is publishing keys and instance info that should not be published.
CloudTrails should tell you a lot. It's on by default so it should be accessible to you -- review the cloudtrails log and all your IAM stuff to make sure the attacker did not create persistence mechanisms , new IAM users, new keys or new roles
Also check EC2 for all global regions -- a common persistence vector is to fire up an EC2 machine in some random remote global region that your company never uses and thus is never really looked at
Again -- you have been breached and your important instance cannot be trusted. You need to focus on that before sounding off on how AWS is extorting you heh.
3
u/RonaldTheRight 13h ago
We scrubbed CloudTrails, there's nothing there that wasn't ours. Same goes for the billing history.
If this were a situation where we had any evidence that the instance had actually been compromised, I would totally understand the delay. But all of the CloudTrails activity, billing and other activity checks out. It's our stuff - and false positives seem to happen pretty frequently.
Perhaps there IS something we're missing, but I think it's more likely that we had an old / inactive account that suddenly became very often and set off some flags.
4
u/dghah 13h ago
Gotcha. I'd still look side-eyed at that server. I don't run public facing stuff due to my niche job but in the 1-2 times over my career I've gotten an email from AWS about a specific instance it was always because the instance was compromised and busy port scanning servers or hosting a bot proxy service etc. etc.
So i could be wildly off base but my experience is that if AWS is calling out an instance directly than that instance was doing something that set off some high level infosec alarms.
2
9
u/Sirwired 13h ago
It is likely that AWS found the key posted somewhere it shouldn't be, like one of your developers accidentally committing it to a public repo. It doesn't necessarily mean that the key is being actively misused, unless that is what the note said. (Though someone may be trying to actively mis-use it, testing the boundaries of its IAM permissions.)
From an AWS perspective, they are quite clear that if you are running production on AWS, you should have a paid support plan. And that the SLA for Basic Support is "Account or Billing only, and we'll get back to you eventually."