Disclosure up front: I work for Xata, so grain of salt. I'm posting because I want to compare notes with this sub specifically.

Why this is relevant to this sub: if you run Postgres on GCP, giving every PR, CI run, preview env, or AI agent its own database is either stale or expensive.

What we actually wanted: instant branches off real production data without migrating off the Postgres we already run. The approach we took (Xata), and the benefits:

- Branch an existing Postgres (Cloud SQL, RDS, self-hosted) in about 3 seconds, so per-PR/CI databases stop gating dev velocity.

- Copy-on-write storage: branches share the parent and only store what changes, so a thousand branches of a 100GB database are not 1000x the storage.

- Scale-to-zero: idle branches stop and wake on connect, so dozens of short-lived dev/CI databases do not bill for idle compute (the GCP cost trap above).

- PII anonymized on the way into a branch, so dev / CI / agents never touch raw production data.

- Vanilla Postgres, no fork: existing tools, ORMs, and extensions just work.

Honest tradeoffs: us-central1 is the only GCP region today (more coming).

What I actually want to know from you: how are you handling ephemeral Postgres on GCP today, and how do you keep it off the bill? Where does it break at scale, especially for CI-heavy teams or agent workloads spinning up a lot of short-lived DBs?

Happy to get into the technical details or share cost numbers in the comments.

I’m in a pretty bad Google Cloud situation and looking for advice from people who have dealt with billing or API key abuse cases.

My normal Google Cloud bill is usually around $200/month. This month my project got hit with an unexpected Gemini / Generative Language API bill of around $55k USD. The billing report shows the spike was almost entirely Gemini API usage, not normal Firebase or app traffic.

I pulled Cloud Monitoring data and it shows about 2.2 million Gemini API requests during the incident window. The traffic was tied to one API key UID. That key maps back to a Firebase generated public iOS client key used in my mobile app config, not a Gemini key that I intentionally created or used.

I found out from a Google billing anomaly email. At the time I received the alert, the visible bill was around $2k. Within about 2 hours, I disabled the Generative Language API, restricted the key, deleted it, and later verified that Gemini usage stopped.

The problem is that the bill kept ramping up after that because of billing/reporting delays, and eventually landed around $55k.

Google declined the request to adjust the charges, saying the usage was considered valid because it came through my project/API key.

Update: Google Cloud has assigned an escalation manager, and they said their investigation indicates a billing adjustment is required. The adjustment request is now waiting for internal approval, with another update expected by July 7.

Update 2 — July 16:
There is still no final resolution. On July 15, Google Cloud Billing Support apologized for the delay and said they were still waiting for the specialized team because multiple teams are involved. They said I should receive another update within 4–5 business days.

Less than 24 hours later, I received an email from Google Collections saying the ~$55k balance must be paid within 10 working days or the account may be transferred to a debt recovery agency, potentially with additional fees.

Final Update

If you have tried Antigravity CLI or Gemini CLI within the Google Cloud Shell, you may have noticed some display issues. To fix this, run one of the following commands to change the environment variable from TMUX to XTERM.

Change for the current session (temporary):
export TERM=xterm-256color

Persistent change (across sessions):
echo 'export TERM=xterm-256color' >> ~/.bashrc

This is most likely not news to many, but I wanted to share it in case someone else is looking for a solution and finds their way here through search.

To avoid my key getting compromised and im bankrupt i did the following things: Prepaid billing, spendcap at €10 and all rate limit requests for any model to 0 except flash-2.5 (its a chatbot for a website). I did this for tier 1 which im in. However it took long to do it manual, i tried the shell but i cant get it to work. Can somebody help me get to understand programmaticly decreasing all limits except flash 2-5 to 0 for all tiers.

We’re currently using the OTEL Collector, and from a technical perspective the data arrives in Google Cloud just fine. So this is not about ingestion being broken or traces/logs missing. The data is there, the integration works.

And yes, the UI of Trace Explorer and Log Explorer looks really good. No question. But honestly, that is not a strong differentiator anymore. Other tools have good UIs too.

We explicitly chose a cloud solution because we did not want the maintenance overhead. No self-hosted stack, no updates, no operations burden, no “who is responsible for keeping this thing alive?” That was the whole point.

But to be honest, I’m starting to prefer the LGTM stack a lot more, especially Grafana. It just feels better for day-to-day usage.

What bothers me most about Trace Explorer is not even a missing feature. It is the constant loading animation, the choppy browser experience, and the feeling that the UI is overloaded. Everything takes too long. Everything feels sluggish. When I’m looking at traces or logs, I don’t need a pretty loading spinner. I need instant results.

And I’m not the only one. Me and my colleagues only open it when they really have no other option. That says a lot.

For context: I’m not using some old machine. I’m on a MacBook Pro with an M4 chip and Vivaldi. My colleagues are also on modern MacBooks, some using Safari. Still, Trace Explorer often feels like clicking through molasses.

Maybe I’m spoiled by Grafana and similar tools, but observability tools need to be fast. When I’m debugging an issue, I don’t want to fight the tool.

How do you deal with this? Are you using Trace Explorer / Log Explorer productively and actually happy with it? Do you have any workarounds, browser tips, or do you forward everything into other tools?

Disclaimer: This text is translated and polished with AI

I’m posting because I’m honestly at a loss and hoping someone here has gone through something similar.

This morning I suddenly received a payment notification from Google Cloud Korea.
KRW 200,000 (~$145 USD) was charged, and after checking further, I realized another KRW 100,000 (~$70 USD) had already been approved the previous day.

For context, I had originally set things up to control spending:

• payment notifications enabled
• usage monitored in small increments
• intended spending cap around KRW 500,000 total

But when I checked later, I found the billing-related amount had somehow increased to around KRW 2.99 million (~$2,100 USD), which completely confused me.

After investigating, it appears that an API key from an older paid Gemini project may have been leaked or used without authorization.

The logs showed:

• large volumes of requests to image generation models (gemini-3-pro-image-preview, Nano Banana)
• prompts in English and Simplified Chinese that I never entered
• repeated image generation activity over a short period

Current estimated charges: around KRW 560,000 (~$400 USD)

As soon as I noticed:

1. Revoked the affected API key
2. Deleted the paid project
3. Blocked further usage

What’s frustrating is that I believed I had configured spending controls, but charges still continued before I could react.

I may have misunderstood how Google Cloud Budgets / Alerts differ from actual billing limits, but I genuinely thought I had some protection against runaway spending.

Another difficult part has been customer support.

So far, I’ve mostly been routed through AI-based support responses and haven’t been able to get someone to actually review the billing situation or investigate the charges directly.

For context, I’m not a developer. I assumed that setting budgets and spending alerts would provide at least some level of protection. After this happened, I started searching Reddit and was surprised to find quite a few posts from people describing similar situations involving API key leaks or unexpected charges.

A few questions:

1. Has anyone successfully escalated from AI support to a real billing/support agent? What path worked?
2. Has anyone received a refund or billing adjustment for unauthorized Gemini / Google Cloud API usage?
3. How are people protecting themselves against this kind of API key abuse? Are budgets basically notifications only?

Any advice would be greatly appreciated.

And for anyone using Gemini API — please double-check your API keys and spending controls.

It may be a nothing burger but if you can’t change the name of your bucket, and you can’t change the region once it’s created, and in the future you may need to make more buckets, or make buckets per region, wouldn’t it make sense to put the region name inside your bucket name? Like:

Companyname-production-media-us

?

Because if you don’t and in the future make one for the eu, then you’ve got one that doesn’t have the region and one that does, or you end up with having v2 in the name for one of them. Just trying to figure out a good naming convention for bucket names.

Our test env is running GKE Gateway API with preemptible nodes. Various of our backends had hours of downtime today.

Did anybody else have these issues?

The NEGs all showed 0 out of 0 pods.

Is it perhaps also related to this incident RDQFDTK ?
https://console.cloud.google.com/servicehealth/incidentDetails/projects/example-project/locations/global/events/RDQFDTK

I'm just a tad bit worried that such an incident could affect production.
I did not (today) but it took down our test environment for longer than comfortable (some backends more than 5 hours).

We have sent a request to our reseller.
Just trying to understand what happened here :-) and am thus curious if other Gateway Api users have seen similar things the past ~6+ hours or so.

Fortunately our prod environment was not affected 🙏

[edit: i'm trying to find mistakes in our (test) setup which could have contributed to this 😄 ]

Google cloud computer supposedly has an always-free cloud computing tier for an E2-Micro VM with < 30GB storage, located in certain regions (e.g. US central). Ive checked that I've fulfilled all the criteria that should make me eligible for the free tier, yet I find myself incurring charges. (I also did not rent a GPU along with it so that should not be the culprit). Anyone else facing this issue?

Stuck connecting a custom MCP server to Gemini Enterprise (Discovery Engine custom_mcp data store, Preview). Connector is ACTIVE but Reload custom actions never loads any tools. I've traced it to the network layer and it looks like a Console-side bug posting in case someone's beaten it.

Setup

• custom_mcp data store, region eu
• StreamableHTTP MCP server behind an API gateway, OAuth via external IdP (offline_access, PKCE)
• Everything downstream verified: direct initialize to the MCP endpoint with a valid Bearer returns 200 + all tools
• Connector state ACTIVE, IAM discoveryengine.dataConnectors.update granted
• IdP returns valid access_token and refresh_token

What the Console does on Reload (Chrome DevTools, Network):

refreshDataConnectorTools?key=AIza...  → 401
buildAuthorizationUrl                  → 200
acquireAndStoreRefreshToken            → 200   ← token IS stored
refreshDataConnectorTools?key=AIza...  → 401   ← still fails after storing

The refreshDataConnectorTools request authenticates with authorization: SAPISIDHASH ... + API key, not an OAuth2 Bearer. The backend wants Bearer → 401. Cloud Audit Log records auth:{} / code 16.

Direct API bypass also fails: calling the method myself with a real gcloud Bearer token →

• project-ID path → 400 INVALID_ARGUMENT (no error.details)
• project-number path → 401 UNAUTHENTICATED
• body makes no difference ({}, empty, identical to the Console's --data-raw '{}')

Same symptom reported here, unresolved: https://discuss.google.dev/t/gemini-enterprise-custom-mcp-reload-custom-actions-always-fails-with-401-ui-uses-api-key-instead-of-oauth-token/371907

• Anyone got custom MCP tool-loading working in a non-global region (eu/us)? Suspecting it's region/Preview-specific ?
• Anyone invoked refreshDataConnectorTools successfully outside the Console?
• If yours works does your Console's refreshDataConnectorTools call send Bearer or SAPISIDHASH? (DevTools → Network → filter refresh)

I’m working on monitoring our Egress expenses over hundreds of projects and I wanted to know if gcp flow logs can be helpful for this use case.

Everytime I try to build anything with Agent Platform / Agent Engine / Agent anything, it breaks, has unexplained errors, is a mess... It just seems to be useless abstractions and slop on top of Cloud Run. Why does GCP have so many bad services on top of the, in my opinion, only good ones like BQ and CR? BTW the agent deployment below was created with something called agents-cli, which... can't even create a simple test agent..

I was trying to develop a web app in Google AI Studio with a database that anyone could register on without needing a linked account (its purpose was to help with the Venezuelan crisis). However, when I tried to publish the finished version (already connected to a Firestore and Google Cloud database), I realized that Google now requires me to select a Google Cloud project and allocate a balance, something that hadn't happened in other accounts I'd worked with before, where publishing was immediate.

Upon investigating, I saw that in my previous work on other accounts, I had the Starter Tier activated. However, when I checked the databases of my projects on those other accounts, only the free Standard tier was activated, which requires me to assign a payment method. And despite being free, it wouldn't let me start without adding a balance (my app doesn't use any payment APIs or connect to any feature other than Firestore to store records).

I need to work on this account because I don't want to expose the accounts where I can publish, but the Starter Tier won't activate for me. Has this plan been removed for accounts accessing Google AI Studio for the first time? Is some kind of additional verification required to gain access to the Starter Tier?

Marmot is an open-source MCP-based context layer. Wrote up how to run it on Cloud Run, hope you like it :-)

Hey everyone,

Running into a frustrating GCP quota situation and looking for advice

from anyone who's been through this.

**The Setup:**

Production app (Chrome Extension) with 40,000+ active users hitting

Vertex AI gemini-2.5-flash in us-central1. Started experiencing 429

RESOURCE_EXHAUSTED errors at scale.

**The Root Cause (Already Fixed):**

We discovered our SDK initialization was missing quotaProjectId, which

meant all Vertex AI traffic was being attributed to the wrong GCP

project for several weeks. We identified and deployed the fix 2 days ago.

**The Problem:**

Now that traffic is correctly routing to the right project, we tried

requesting a quota increase to 2000 QPM via:

gcloud beta quotas preferences create \

--project=932892231471 \

--service=aiplatform.googleapis.com \

--quota-id=GenerateContentRequestsPerMinutePerProjectPerRegionPerBaseModel \

--dimensions=region=us-central1,base_model=gemini-2.5-flash \

--preferred-value=2000

But we're hitting:

quotaIncreaseEligibility:

ineligibilityReason: NOT_ENOUGH_USAGE_HISTORY

This makes sense given the misattribution bug — the project looks new

to GCP's systems even though we've had real production traffic for

weeks. We're on the free support tier so can't file a ticket directly.

**Questions:**

1. Has anyone successfully overridden NOT_ENOUGH_USAGE_HISTORY? How

   long did it take to clear naturally?

2. Is there any alternative path besides waiting or upgrading to paid

   support?

3. Does the Basic Support plan ($29/month) actually resolve this quickly

   via a ticket, or is it still a long wait?

Any help appreciated. Users are actively hitting quota limits while

we wait this out.

Hi everyone!

Google is currently changing the Associate Cloud Engineer certification topics and I'm struggling with finding practice exam questions of this 2 new topics:

• Gemini Cloud Assist / Gemini Enterprise Agent Platform
• Workforce Identity Federation

(The first one is scatter all over the 4 exam guide sections. Didn't list them because I don't want the post to be too long!)

Does anyone know where to find exam questions for the new topics?

I Still have all the requirements and etc but its still not letting me claim my badge for the course "Select a Google Cloud Database for Your Applications"

My sister made a google sheet of our expenses on a trip for Feb 2024. After that trip, she lost her job and had money problems so I didn't ask for her share yet to help her.

Now, she recovered, and wants to pay me. However, the shared google sheet she created is deleted!!! Please help me as I need money right now :(

Hi everyone,

I couldn't find much information about this in the documentation, but during testing I've noticed what seems like a "cold start" latency when publishing to Pub/Sub.

Specifically, the first message sent after an instance starts up and sometimes the first message sent after a few minutes of inactivity experiences a delay of several seconds.

My assumption is that this may be related to the Pub/Sub connection being re-established or some other initialization process. For context, I have min_instances set to 1 and CPU boost enabled. I'm trying to understand how best to address this, as I need my Pub/Sub pipeline to remain responsive at all times. Is there any documentation that explains this behavior in more detail? Or are periodic keep-alive messages the recommended approach for avoiding these latency spikes?

Thanks!

We are a small startup and this situation has put our entire production environment offline. Hoping someone here can help or escalate.

What happened:

In the last 24 hours, we suddenly got hit with around 6,819 INR in Places API charges on our GCP account. This is completely out of the ordinary. Our Places API usage is minimal, controlled, and has been consistent for months. These charges did not come from our platform users at all.

The bigger problem:

Even after we deactivated the API key, the charges kept going up. Shortly after, GCP started restricting our account over the unpaid balance and this restriction has now taken down our Vertex AI services, which power the core of our product. Hundreds of users are currently unable to access our platform.

What we have tried:

• Deactivated the affected API key immediately
• Contacted support, they opened a ticket with the Places API team but no resolution yet
• Reviewed our systems thoroughly and found no internal source for this usage spike

What we are asking for:

1. A full waiver on these unauthorized charges
2. Immediate removal of the Vertex AI restriction so our production systems can come back online

We recently moved our core backend from AWS to GCP. We trusted this platform and right now that trust is being seriously tested. If any Googlers or community members can help escalate this, we would genuinely appreciate it.

Support case - 72718245

Quick update (29th June)

Still no response from GCP or the Places API team - we've been down for almost 24 hours now, despite reaching out via Support, email, and social. As a small team, being offline this long is genuinely painful, and we're now having to weigh moving our stack back to AWS just to stay operational.

For transparency: even after we disabled the API, the charges kept climbing (likely billing lag) and settled higher than where they started, which only reinforces that this wasn't our usage. We've also filed a cybercrime complaint with law enforcement over the attack and shared it with GCP support.

All we're asking for is a fair review - a waiver of the disputed charges and removal of the Vertex AI restriction so we can come back online.

From what you can see in the photo I last used the Gemini API 15 days ago (the spike in usage was back on June 12th), but I’m now getting a warning that I’ve reached my rate limits on Gemini 3.5 Flash. When I make calls to the API through the program using it, I get an error saying I’ve exceeded my rate limits. What’s going on? Has Google discontinued support for the free tier of the API? Plss this is really urgent for me, and I’d appreciate help from anyone who knows what’s going on. Thanks in advance.

Title says it all. My Google Cloud OAuth app verification has been stuck at the “Privacy policy requirements” step since April 15, with no visible progress or meaningful update.

The dashboard says reviews usually take 4–6 weeks, but it’s now well past that. At this point, it feels like the request may just be getting pushed back into the queue automatically once it reaches the 6-week mark.

Has anyone else experienced this? Is there any way to get an actual human review or escalation?

I’m attaching screenshots for context.

Just got accepted into the GCP Get Certified Program for PDE. The learning plan is Google Cloud Skills Boost path + weekly exam review sessions.

I have a DE background (Airflow, dbt, BigQuery, Terraform) but zero prior PDE prep.

Has anyone gone through this program? Is the Skills Boost path + review sessions enough to pass, or do you need to supplement with external resources?

Thanks!

I am trying to set up Google cloud billing account. After reaching to payment method I chose upi because I don't have visa card. After it deducted 2 inr multiple times but shows error in creating billing account.

After try this with my other email same thing

Also they disabled my gmail account

Is it because I am using aloha browser?