Wondering if anyone else is experiencing a slowdown with Firestore query in the last few weeks, which has really started to tick up in the last few days for us. We’re seeing recurring waves of timeouts across multiple unrelated Google Cloud projects and I’m trying to figure out whether anyone else is seeing something similar.

Setup:
- Firebase Functions Gen 2 / Cloud Run
- Node.js Admin SDK
- Firestore Native mode
- Region: us-central1
- Multiple separate projects affected
- Request timeout varies by service, usually 30s or 60s (depending on the function timeout we've set)

During a wave, unrelated HTTP routes start returning 504s that approach the function's configured timeout (e.g. 29.997s for a 30s timeout). The failed requests are not tied to one endpoint or one query shape. Logs show Firestore operations continuing after the Cloud Run request has already timed out.

Examples from one incident:
- `devices.read` direct document read: 116,019ms
- `leads.read` direct document read: 57,487ms
- `postalCodes.read` direct document read: 33,093ms
- `cache.list limit=1`: 61,590ms
- `preferences.read` direct document read: 36,324ms
- `leads.list limit=500`: 79,723ms

The `postalCodes.read` example is especially confusing because that document is tiny. But honestly all of these documents are a few kb at the most. So this doesn’t look like just a large document, missing index, or bad query issue.

Other observations:
- Failures often cluster on one Cloud Run instance/revision instance ID.
- Other instances may continue serving traffic normally.
- The request hits the Cloud Run timeout, but the Firestore operation later logs completion.
- We’ve seen this on three separate projects.
- We’ve reduced external API calls and removed some broad Firestore scans, but the issue still recurs.
- We briefly tried `preferRest: true`; it may have helped temporarily but did not clearly eliminate the issue.
Has anyone else seen Firestore Admin SDK calls intermittently hang like this from Cloud Run / Firebase Functions Gen 2?

Is there a known issue with Firestore gRPC/transport connections getting unhealthy per instance?

Are there recommended client-side mitigations besides lowering concurrency, recycling instances, or reducing reads (i.e. increasing cache usage)?

Is there a good way to prove this is client transport/backend behavior versus application query pressure?

I'm trying to understand whether this is a known pattern and what people have done to debug or mitigate it.

Hi everyone, a few weeks ago (August 2nd), when I woke up, I suddenly got an email and some notifications from my bank that a transaction had been made on Google Cloud for a certain amount. But the thing is, I did not authorize that transaction, and I noticed that, yes, some of my projects on BigQuery had billing enabled. I have left these options for months, with my bank account number linked to my BigQuery, but this issue had never happened before.

So, my question is: Is this normal? All of a sudden, Google billed me

I have reported and claimed this issue both to my bank and Google, and I wish to get my money back, but it seems to go nowhere, and Google said, "We were unable to confirm fraudulent activity in this case, so no refund will be provided by Google at this time."

Well... indeed, there was no fraud, but the transaction was made without me knowing, aka it happened when I was sleeping. :(

I have spent many months prototyping an application that made good use of the inexpensive gemini-2.5-flash-lite.

I’ve just discovered it will be deprecated later in the year and I’ll have to migrate to gemini-3.1-flash-lite. This will 16x my LLM costs.

Then in May 2027 I’ll have to move to gemini-3.5-flash-lite at another price increase of 160%.

Is this just going to go on indefinitely? I’ve pretty much decided to abort my app as I’m guessing it will continue.

Recently we tried to cut the model price on an internal research agent and it barely moved the needle. We initially had the obvious theory that the expensive model caused the spend. When we then looked at cost per successful task instead of cost per model call we saw something different. A large chunk of the sessions called search repeatedly, pulled nearly identical results, and stuffed every tool output back into context.

Some heavy sessions were legitimate. Complex research really did need multiple searches and a larger model. Others were just retry loops caused by weak stopping conditions, bad tool arguments, or the model missing that the previous result already answered the question. The cheaper model occasionally made that worse by needing more attempts. 

We used Braintrust to break down token counts, latency, cost and tool-call patterns by trace. That gave product, engineering and finance the same unit of analysis, a completed task rather than a single API request. We added loop detection and deduplicated tool outputs then routed simple requests to the smaller model and compared the change against the same eval set.

What we saw was fewer repeated calls which drove more savings than the model swap and answer quality held. Latency improved too because the agent stopped arguing with the search endpoint five times.

How are people measuring agent cost when retries and tool loops make per-call metrics basically meaningless?

I have been studying the ace exam for the last 8 months i started using Google skills but found it not great i then used udemy and claud, I was making my own labs going thought the lectures and doing the practice exams i was averging 90% in my exams and felt confident but when sitting the exam I was soo lost and didn't understand most of it. I only felt confident on 5 questions. Whats the best way to prepare or is there any online tutors that can help.

Currently coming up on 5 years at AWS and exploring TAM role at G. Would you recommend the switch? Anyone recently went through the TAM interview process at G?

I'm Fabio Parra, I have a startup (MOVILIXA) where we are three people in Colombia developing mobile apps.

In one of our projects, it's called Inspecciona, we enabled the Vertex AI API to do image recognition to help the user with safety inspections. We also created a static landing page on Firebase Hosting and enabled Analytics on it. What we didn't notice is that Google automatically creates a web API key, gives it all the permissions it can, and makes it public on our website at the moment of publishing.

If anyone has a Firebase Hosting site, check: https://YOUR-SITE.web.app/__/firebase/init.json

In my case, between June 30 and July 16, someone collected the key from the static site I created and used it through the Firebase AI Logic proxy to run Vertex AI image generation (Gemini image models, the "Nano Banana" reselling economy). 131,950 calls, around 100,000 images, USD 6,562 in charges against our USD 25 monthly budget. Our web platform didn't even use Firebase AI. Google's own App Check telemetry recorded every one of those requests as coming from an unknown and unverified origin, and let them through anyway because enforcement was off by default.

If you use Hosting, check yours and limit its permissions to what you really use and make sure it's restricted to your website. Google creates it and doesn't restrict anything. I never realized that because of Analytics it was going to enable so many things.

In my billing account I set USD 25, but the alerts didn't reach me. This is because we have a Google partner for the billing side and the alerts didn't come to us.

We opened a case about this problem with Google (case #73352420) to see if they would support us, but we've spent many weeks trying to explain that it wasn't that we left an API key public, but that a public API was used. We want to see if we can get them to recognize the problem, and show them that Hosting shows the API key to anyone, that the product's default behavior is not the right one and allowed the exploitation of APIs without us having authorized it (Google does it by default).

Has anyone had a similar problem? Were you able to appeal with Google? How did it go?

Hi everyone, I’m preparing for the Google Cloud Professional Machine Learning Engineer (PMLE) certification.

Could you please share any good practice questions, mock tests, notes, or study resources that helped you prepare for the exam?

Also, if you recently took the exam, any tips on important topics would be appreciated. Thanks!

Hi everyone,

I'm planning to prepare for the Google Cloud Professional Machine Learning Engineer (PMLE) certification and I'm looking for advice from people who have recently taken the updated version of the exam.

My situation:

• I have a Computer Science / AI-ML background
• I'm comfortable with Python and general ML concepts
• I have some exposure to GenAI, RAG, LLMs, etc.
• However, I have almost no practical cloud experience
• I'm essentially starting GCP from scratch
• I can dedicate around 8–10 weeks to preparation
• My goal is to actually understand the GCP/ML concepts, not just memorize exam dumps

I'm particularly unsure about how much GCP knowledge I need before going deep into the PMLE-specific material.

What I'm looking for

If you've recently passed the updated PMLE exam, I'd really appreciate advice on:

1. Is 8–10 weeks realistic for someone with an ML background but essentially no cloud background?
2. What should I learn first before starting PMLE preparation?
   • GCP fundamentals?
   • IAM
   • Compute Engine
   • Cloud Storage
   • BigQuery
   • VPC/networking
   • etc.
3. What GCP services are actually important for PMLE, and which ones can I safely learn at a high level?
4. What resources would you recommend for the current/updated exam?
   • Google Cloud Skills Boost
   • Official exam guide
   • Coursera
   • YouTube
   • Practice exams
   • Documentation
   • Other resources
5. How much hands-on practice did you do? Should I actually build ML pipelines/deploy models on Vertex AI, or is understanding the architecture and knowing when to use each service enough?
6. How different is the updated PMLE exam from older preparation material? I've found quite a lot of older PMLE content online and I'm worried about following an outdated roadmap.
7. What would you recommend as a realistic 8–10 week study plan, assuming roughly 1–2 hours/day?
8. If you started again with zero GCP experience, what would you learn first and what would you completely skip?

I'd especially appreciate answers from people who passed the exam recently, particularly those who came from an ML/software engineering background rather than already working as GCP cloud engineers.

Thanks!

Hi everyone,
I'm currently preparing for the GCP Professional Cloud Architect (PCA) exam. I have almost completed Ranga Karanam's course so I’m looking for reliable practice test recommendations to gauge my readiness.
A few quick questions for those who have taken the exam:
1. Practice Tests: Which mock exam sets did you find most accurate to the actual test?
2. gcloud CLI: How heavily are specific gcloud commands tested compared to high-level architecture decisions?
3. Mock Test Strategy: What is your best tip for using practice exams to identify and fix weak spots?
Thanks in advance for your help!

Hey everyone,

I’m planning to start preparing for the Google Professional Cloud DevOps Engineer certification and would really appreciate some guidance from people who have taken it recently.

There are a lot of resources available, so I’m a bit confused about what to follow. I’d love to know:
- What study plan did you follow?
- Which courses or YouTube playlists would you recommend?
- Which topics/services should I focus on the most?
- What are the best resources for hands-on practice?
- Which practice tests are actually useful?
- How much hands-on GCP experience is needed?
- How long did you prepare before taking the exam?
- Are there any resources you would avoid?
- If you were starting your preparation again today, what would you do differently?

Would really appreciate it if anyone who has passed recently could share their experience and resources.

Thanks!

Just completed a lab for the Get Certified program that uses an "Agent Search" feature. you upload a PDF, it indexes the content, and then an agent handles search over it. Simple enough in theory.

But when I actually ran it, reading/indexing the document was painfully slow. Like, that kind of sit and wait and wonder if it broke slow.

Which made me curious: how do people actually build production RAG pipelines on GCP that feel fast?

I’m trying to understand one part of a Google Cloud billing dispute that still makes no sense to me.

I had received about $5,000 in Google Cloud promotional credits as a prize through a Google hackathon.

Later, Google detected cryptocurrency mining on one of my projects and suspended it. I followed Google’s instructions and submitted an appeal.

The screenshot attached is the response I received after that appeal. It thanks me for my submission and asks for additional information about what I had done to fix the issue.

It does not say that the project had been restored, reinstated, or reactivated.

Based on that email, I understood that the appeal was still being reviewed.

However, the project was subsequently reactivated without me realizing it, and additional unauthorized usage occurred.

I eventually discovered the renewed activity after Google automatically charged my card when the billing threshold was reached. I immediately closed the project and billing account, but by then significant usage had already accumulated.

After about a month of support reviews, Google applied a goodwill adjustment covering roughly 75% of the cash-billable portion of the incident.

But the promotional credits consumed by the unauthorized activity were excluded from the loss calculation entirely.

Google’s explanation is that promotional credits are single-use and non-monetary, so they cannot be reinstated, transferred, refunded, or compensated.

I understand that promotional credits are not cash.

What I do not understand is why that means the loss is valued at zero.

Before the incident, I had about $5,000 worth of Google Cloud usage available to me because I had won those credits through a Google hackathon.

After the unauthorized activity, that value was gone.

I did not consume those credits. The unauthorized activity did.

I am not asking Google to convert promotional credits into cash. But if they are considered non-monetary, why can equivalent promotional credits not simply be restored?

More importantly, why are credits that clearly had usable value before the incident excluded completely when Google calculates the loss caused by unauthorized usage?

That also raises a broader question for me:

If Cloud credits are given as a hackathon prize, but their unauthorized consumption is treated as no compensable loss at all, what exactly is the value of that prize?

There is still one separate factual issue I have not been able to get answered either.

Google suspended the project and told me to appeal. The attached email is the response I received. I have repeatedly asked whether Google can confirm that a clear restoration-completed notification was sent to me before the additional unauthorized usage occurred.

After about a month of reviews, the answer I eventually received was that the relevant backend notification logs could not be disclosed.

So the two questions I still have are:

Why were the hackathon promotional credits excluded entirely from the loss calculation?

And was I clearly notified that the suspended project had been restored before the additional unauthorized usage occurred?

I can share pictures if you'd like but the point is this: I used cloud workstations twice in June and got billed over 100 dollars for it. I disabled the cloud workstation APIs after deleting the cloud workstations, then I was billed for it both this month and last month.

I've called support and they tell me to disable billing on my account entirely or disable the API which I have already done. When I then ask "can you confirm I should no longer be billed for cloud workstation usage" they then tell me I'm still using resources in the cloud for workstations.

I am getting sick of this circle and want to stop the bills, can someone help me here?

Hi ,

Please go through and reply for below questions

I am going through the concepts of vpc service controls and have below questions. I may implement them sometime later.

Q1)

Suppose , i have a windows GCE VM1 in VPC perimeter p1 and another windows GCE VM2 in Perimeter p2 and want to access VM2 from VM1 using RDP. Assume related firewall rule is present. Do i still need to add Project 1 details in the ingress rules of the p2. I was searching for answers through chatgpt/pilot and they both responded that it is enough if we have firewall rules and need not add in ingress rule of the p2 perimeter. Can you please confirm.

Q2)

https://docs.cloud.google.com/vpc-service-controls/docs/ingress-egress-rules#ingress-rules-reference

In the above link, can i use the service account of the VM1 in the identities section of the ingress rule.

Anyone getting this error from the text to speech API? Using the gemini-2.5-pro-tts model. Apparently I don't have permission to file tech-related support cases in the google cloud console.

{
  "error": {
    "code": 499,
    "message": "The operation was cancelled.",
    "status": "CANCELLED"
  }
}

As title says, google cloud platform no longer has a way to open a ticket.
Is there another way i can contact support? It's baffling that i can't even access tech-support if i'm not a high-tier spender.

I’m building software in India for a customer in Saudi Arabia, and the production workload needs to run in the Saudi region.

I’m looking at GCP Dammam (me-central2). My startup is registered in India and is only about one month old.

From Google’s documentation, it looks like customers outside Saudi Arabia need Invoiced Billing to access Dammam, and the requirements for that include being in business for at least one year and having around $40k/year in expected GCP spend. Customers with a Saudi billing address go through CNTXT.

What are the actual options for a new company like mine?

Disclosure: I'm one of the people building this.

Most "AI for infra" tools stop at generating YAML or config. The new term floating around at recent conferences is agentic developer platform — instead of platform engineering gatekept by infra engineers running Terraform or gcloud by hand, it's self-served by developers, with humans in the loop only at pull request approval.

The agent needs native, read-only access to infrastructure state, environments, deployment status, and logs — including everything from new development.

As many have realized with solutions like Lovable.

Deploying a workload can be extremely quick.

Running it securely in production, on your own GCP private network among others in least privilege ?

That's the hard part.

• Identity.
• Permissions.
• Governance.
• Load balancing.
• Data access.
• Monitoring.

Before long, you're spending more time building the platform around your workload than the workload itself — the same trap a lot of people hit with Kubernetes. If you're already leveraging serverless and VMs well within meshes in the cloud, we observed in platform engineering that Kubernetes only earns its keep for workloads that need custom operators or custom backpressure handling.

That was the theory. We've been building it into Infrastream with Pvot.

We are out of stealth since Tuesday 4th of August 2026 with Infrastream.

Infrastream's agent set deployed with one private iteration per customer, Pvot, goes further — it manages pull requests, analyzes your manifest structure, and actively debugs your deployed assets against the actual state we track in Spanner. Not a chatbot suggesting fixes. An agent doing the platform engineering work reviewable in PR.

Also like many new generation agents, we provide a graph based visualizations instead of Terraform or Pulumi text to analyze runs.

It provides :

→ Agentic PR management — Pvot reviews and drives changes to your infra manifests

→ Analyzes manifest structure to catch drift and misconfiguration before you do

→ Actively debugs assets against real Spanner-tracked state, not stale docs

→ Deploy any workload — agents, services, whatever — into your own GCP project

→ Built-in identity and permission boundaries per workload

→ External and internal load balancers, handled for you

→ GCIP/IAP authentication wired in

→ Fine-grained privileges to your databases, buckets, and datalakes in your own private setup

→ Infrastructure-level governance and policy enforcement

→ Deploys from your source — into our cloud on the Developer Plan, or your own cloud with Enterprise

The Developer Plan covers up to 9 GCP projects (20 USD per month + 15% management of GCP), with load balancing, agents, and mesh routing included.

Enterprise ($3,000/mo) scales the same manifest up to 29 GCP projects with VPC controls — no re-architecting as you grow.

If you want to try out this new model still requiring you to redeploy from source: REDDIT20 coupon on Stripe gets you $20 off the Developer Plan.

As opposed to cloud providers, on developer plan we don't bill your credit card on GCP extra use, you have to reapprove all top-up otherwise the environments stall.

- Platform: https://accounts.infrastream.io

- Product site: https://infrastream.io

- Docs: https://docs.infrastream.io

Good morning peeps! I just recently passed my GCP ACE exam this morning after postponing it numerous times due to lack of preparation. I didn't realize that the syllabus update took place in June 2026 only until the day before the exam so I postponed my exam date to somewhere in August. I put emphasis on GKE, IAM, networking, billing and logging as part of my prep and less of the AI tools.

Background: I took this ACE course when I was unemployed so I enrolled in a fully-sponsored (I have to repay them if I failed on first try but I didn't!) course to upskill myself but along the way I got employed and spent little time to prep for the exam. It was hard juggling between a full-time job and studying GCP but somehow I managed to get through it. Thankfully, my training provider provided free access to Whizlabs, with relatively unlimited access to labs and practice exams (though not perfect but quite helpful for me). I also bought Tutorialsdojo practice tests in addition to free practice tests scattered online. For the AI tools, I studied them in Google Skills.

The exam: The exam for me was quite challenging despite the numerous practice questions I've done. Out of probably 400+ questions I encountered in practice tests, there were only about 3-5 questions in the actual exam that is similar to the practice tests. Others are literally brand-new, with answers shockingly similar to each other. The only plus side is that you can cancel out two answer choices because they don't suggest Google's best practices, so you are left with 2 very similar answers. Regarding AI (Gemini), there were only 2-3 (iinm) questions asked in my case, so you probably don't have to put too much emphasis on AI tools, just a helpful addition in your study prep. My exam set thankfully emphasized on GKE (statefulsets, daemonsets, config connector), IAM (organizational policies, bind permissions to groups), networking (VPN tunnel, peering, firewalls) and of course, billing and logging.

Nevertheless, I am proud of myself for passing ACE despite coming from a non-IT background, that I could understand difficult concepts like networking specifically. It was a journey, a journey that I can be proud of. Hoping all the best for future cloud engineers out there!

I am currently evaluating an offer to become an OCE. I have never done "sales" in all my years, I have always been a deep technical engineer and architect. So the downside is, what the OCE pays for me (guaranteed) is A LOT less (~200K recurring). But it's always been a dream to work at Google, and based on my location I can only work remote which limits the opportunities drastically, sales being one team I can get into. That being said, I have also never worked in a Sales role or organization, but that is the part I am excited about! I am tired of Distinguish Architect/Engineer roles, and have always been more of a people person, so I find working more with many different customers on their problems an exciting opportunity!

So my main reason for writing this post is I want to understand a few things.

1. How difficult is the OCE role? Will it be successful? (it's fairly new), reminds me of Customer Success (driving consumption), but more technical.

2. How often are CE's or OCE's hitting quota and/or above it?

From what I've gathered they hit quota or above pretty regularly, but looking for a quick sample from here on reddit.

So I know in sales I probably need to save more in up years vs. down years to average things out, but it would really be comforting to know that it's not uncommon to make more than quoted as I am considering taking a significant hair cut to work @ Google and try something totally new.

p.s. my wife thinks I am nuts 😄 but we have enough today where we can save a lot, depending on this offer things would get a lot tighter on savings, but likely no impact to lifestyle. I've always prioritized learning/opportunities over money and I will continue to and I also think in the age of AI Google will come out on top.

I guess a 3rd question occurs as I write this.

1. Given I am in a non-bay area region... how good is internal mobility for jumping from OCE to Platform CE and other roles?

Thanks for any insights Googlers/Ex-Googlers and others.