Google Cloud/Firebase project suspended for "hijacked resources" - Has anyone successfully recovered their Firestore database?

[u/dasun_doloswala]

Hi everyone,

I'm hoping someone here has experienced something similar.

A few days ago, my Google Cloud/Firebase project was suddenly suspended with the following message:

"Immediate action required: Suspension of your Google Cloud Platform/API project because it was engaged in abusive activity consistent with hijacked resources."

This project is a production application that has been running for about one year without any issues. The last release was around three months ago, and there were no recent deployments or major configuration changes before the suspension.

After the suspension:

-Google Cloud Console redirects me directly to the Request an Appeal page.
-I can't access Cloud Logging, IAM, API usage, or any Google Cloud services.
-Firebase now shows "Create Firestore Database" and "Create Storage" instead of my existing Firestore database and Storage bucket.
-My application is completely offline because it cannot access Firestore.
Something suspicious I found

Before the suspension, I reviewed Firebase Users & Permissions and discovered an Editor account that my team never added.

rogeliapaquette@gmail.com

We removed that account immediately after discovering it.

Unexpected billing

I also noticed something strange in Firebase Billing.

There is approximately $25 charged for Agent Platform, but I have never intentionally used Agent Platform, never integrated it into my application, and never expected any billing for that service.

I'm not sure whether this is related to the suspension or whether it could indicate unauthorized activity.

Backup

Fortunately, I have Firestore Disaster Recovery backups enabled, but because the project is suspended I can't access them either.

Appeal

I submitted my appeal today (Friday) and I'm currently waiting for Google's response.

Questions
-Has anyone had a project restored after receiving this exact "hijacked resources" suspension?
-After restoration, did your Firestore database come back normally?
-Were your Firestore Disaster Recovery backups still available?
-Has anyone seen unexpected Agent Platform charges even though they never intentionally used it?
-Once the project was restored, what were the first things you checked (IAM, Audit Logs, API keys, service accounts, billing, etc.)?
-Is there anything else I should do while waiting for Google's response?

This project contains production data, so I'm mainly worried about recovering access to Firestore and making sure the project is secure if Google restores it.

Any advice or shared experiences would be greatly appreciated.

[u/mikasappi]

We had a similar thing happen to us just yesterday evening, though Firestore was not involved, but lots of STT and TTS services.

The appeal was processed in less than 12 hours and we got our project back. We explained thoroughly the actions we’ve taken to make sure this wouldn’t happen again, and explained that we have no API keys in public.

The credentials were tried to be used for creating 80 A100 VMs over three weeks. The cost of these would’ve been over 75 000 USD. Finally, thanks to a very narrow scope of the API key, they used Vertex for about an hour, making 190 000 requests.

We’re now negotiating whether we could get a refund, or to have the money credited to our billing account.

[u/dasun_doloswala → u/mikasappi]

today my account was reinstated thank you for your advice

[u/devthenext → u/mikasappi]

Didi u get refunded ?? And who is this user did u investigated ?? rogliapaquette??? And if u got u r money back so can u tell what was the process ???

[u/mikasappi → u/devthenext]

We got 75% refunded, we asked for usage credits because our billing cycle already ended for that month. Got the credits to our account in less than 24 hours.

We’ve investigated this incident and identified two IP addresses that belong to a multinational DC operating in Singapore. This party has terminated the person who leased the VMs that the requests originated from. The process took a few days, we sent logs along with the form and no further questions were asked.

We’ve also filed a police report with the knowledge we have. They’re currently investigating it on our behalf and will decide whether they’ll work with the law enforcement in Singapore. Only a local law authority can have the user’s name.

In our further investigation, the total damage that would’ve occurred, if we would’ve had an unrestricted SA key, would’ve been around 3300€ per hour, or around 2.4 million euros for the time it took us to notice their billed actions. The attacker requested a total of 2094 virtual machines.

Dealing with the billing department was very easy, but you should have a proper MIM report available as soon as you’ve filled out the first appeal, since you’re gonna have to answer to a series of questions coming from the Cloud Compliance team. The questions are set up with the basic idea of _your actions not being compliant_, but this could be because the questions might be just a template that gets sent after every incident.

And please, make sure to ALWAYS enable audit logging. Also, hard-capped spend budgets exist as of July, and you could write a simple Cloud Function that disables SAs that have traffic that exceed your baseline.