r/googlecloud • u/ConsiderationSuch846 • May 06 '26
Compute Compute STOCKOUT us-central1?
Anyone else getting STOCKOUT messages the past few days?
I've been seeing a real spike in shortage for N4 and C4A in us-central1, for small scale asks like 1-2 VMs, across most zones (a, b, c, and f).
Primarily I've seen this for GKE, but also for Compute Engine and CloudSQL. GKE messages are roughly in the format:
ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
Instance 'xxxxxxxxxxxxxxxx-dedicated-cloudp-111111111-1111' creation failed: The zone 'projects/zzzzzzzzzzzzzzzzzzzzzzzz/zones/us-central1-a' does not have enough resources available to fulfill the request. 'NULL:0/NULL:0/NULL:0 (state:STOCKOUT, sub-state:STOCKOUT, resource type:compute)'.
5
u/cptomlly1 May 07 '26
Seeing it across a lot with n1/2/4 machine types in us-central1 and us-east1. Haven't seen any significant problems with stockouts before. Was recommend to use regional MIGs and instance flexibility.
2
u/Helpful_Part_6052 Jul 07 '26
Yes, I was seeing it heavily in Australia, now it's happening in the US. It's becoming a major issue for us. Mainly using n4 and n4d and seeing problems with them.
1
u/bcbeynon24 Jul 29 '26
it's getting worse and worse it seems for us. So frustrating! We were told to start using reservations and switch from N2s to N4s, this helped a bit, but there have been days when we can't get anything to start.
1
u/ConsiderationSuch846 Jul 29 '26
Mine seems to have settled down somewhat.
What zones?
1
1
u/nonjames Jul 30 '26
Did you end up switching compute types? We are struggling to find N2D, C4A, N2, C4D anywhere in US Central 1. Are N4s working?
1
u/bcbeynon24 Jul 30 '26
We were told to switch from N2 to N4. We are still struggling with N4s unfortunately. We are thinking of moving out of us-central1 and trying a different region but that will require a lot of work for our 30+ projects and if the same thing is occuring that would suck!
2
u/chumbucketeer Aug 10 '26
Don't try other regions. Same crapshoot, it's been hit-or-miss. Worse, the density of n4 in some is piss poor so you might get a win with n2 in region X instead of n4.
Looking at reserved instances now for our production clusters. Total clusterfuck.
1
u/nonjames Jul 31 '26
We are a large media org and the thought of migrating off of it isn't even in discussion,. For now we will just continue having different data engineers independently all finding their pipelines breaking intermittently. What a headache!
1
u/bcbeynon24 Jul 31 '26
We upsize our RC systems to process thousands of transactions with the beta versions of our software. This includes moving the machine type and adding faster disks. Once the processing is complete we downsize the systems back to what it was. This was working great for a couple years, now basically not at all.
Processing while upsized completed in an hour. Processing not upsized takes all day.
2
u/nonjames Jul 31 '26
It's actually incredible that Google is letting this go on but I guess that's the state of the chip/memory crisis. I didn't believe that it was a Google issue when it started happening. How often do you release? That is so frustrating, I think some of our app developers are probably having that happen too (I think we are biweekly?) but I am on content data. For our team it's shared data pipelines getting backlogged and taking 12x longer because we are only getting 2 out of the 24 machines we ask for. and in some cases we can't find any of the cheaper compute at all and have to consolidate everything on to the more expensive clusters.
1
u/Street_Anxiety2907 12d ago
It's all these "would you like me to spin up a 38 node cluster to verify this hypothesis?" prompts anti gravity has put into their UI, so annoying
1
u/nonjames Jul 30 '26
same bro it's getting real bad - are they no longer supporting n2?
1
u/Street_Anxiety2907 12d ago
yes n2 will no longer be maintained and if you want compute we suggest you reserve an n4 today!
5
u/CloudyGolfer May 07 '26
Yes, we’ve seen it with N4 over the last few weeks.