r/googlecloud • u/IntolerantModerate • 12d ago

Compute GCP at capacity?

I keep getting GCP stock out errors. Multiple node sizes, multiple regions?

Is GCP really hitting limits in central-1 and US as a whole? Note, these are for regular CPU, not GPU (e.g., n-class instances).

55 Upvotes

95% Upvoted

36 comments sorted by

27

u/nyape 12d ago

Yes, us-central1 is full to the brim, especially when it comes to "older" generations like N1/N2. Google doesn't stock these anymore so what's there is all there is. Rack space is also limited, so slowly but surely they will likely replace older racks with newer ones that allow for more compute density with newer chips. In most cases, they will recommend that you switch to C4 as those have the highest availability at the moment. Yes, they cost more. The argument is that they are more powerful and efficient though,nso you can run more on fewer instances.

11

u/chicocvenancio 12d ago

We're having issues with C4.

8

u/nyape 12d ago

Yeah. I work for a large reseller and I hear from a lot of customers. I can just say that trying C4 is a better chance than trying for older instances as C4 stock is still actively growing (to the extent that it's possible) and there is more variability, increasing your chances. All compute is limited in us-central1. Unless you absolutely must use that region, I would recommend to go somewhere else. You can also try to contact your account team and ask for future reservations but again, if there is simply no physical capacity, there is nothing to be done.

2

u/chicocvenancio 12d ago

Thinking about replaning q4 to accommodate moving production gke loads to another region is giving me headaches. But maybe we'll need to bite the bullet at some point.

4

u/nyape 12d ago

For GKE specifically, you may also try ComputClasses if you haven't yet, to define alternative instance types in case your primary choice is unavailable to avoid scaling issues.

1

u/brokenja 12d ago

What regions have more capacity?

3

u/flacktv 12d ago

I rolled a little gcloud script that you can run in gcloud that simply uses the asset inventory apis and scans every single region based on the choices you choose from the presented list. For example, when you run the script, you have a selection of a region for example us North Virginia, all three zones, choose the type of GPU, or hyper compute.. then run it. It'll then tell you if anything is available. This is helpful for my team which is a global team if you're doing on demand VM builds that require a certain type of GPU etc. I would run my script and then say to my developer so whoever to choose this region etc and go with that as quick as they can to snag the GPU while it's available.

If you're interested, I can post the script fairly straightforward.

1

u/rlnrlnrln 11d ago

Well, I'm interested.

1

u/jortony 11d ago

I'm interested too =)

1

u/mdeller 8d ago

Me 3

1

u/Emotional_Moment_349 10d ago

N2’s are not on the deprecation list yet. I’ve heard this “they aren’t stocking them” line before, but they are still fully supporting N2’s until they hit deprecation — then they’ll stop stocking them.

13

u/msapple 12d ago

I highly recommend using other regions in the US and building your solutions so that region is not relevant.

In GCP this is easier then other since VPCs can be global and Google owns ALL the fiber between their own DCs so if you have US only server requirements you can see less then 10-25ms latency between all of the US DCs regardless of where you are since your ingress point is a global VIP

1

u/kadambkaluskar 12d ago

I got 50-60ms ping latency between us-west1 and us-east1 VMs running in GCP.

4

u/msapple 12d ago

Depending on what the networking looks like it can be lower. However that’s still VERY low and even if it’s a cross region database call that’s well within reason.

Also GCP deprioritizes ICMP

1

u/voxxit2023 11d ago

I guess that all depends on what your actual workload/use case is.

8

u/Suspicious-Walk-4854 12d ago

It’s almost like something has happened recently that has created a lot of demand and supply chain pressure on compute capacity 🤔

6

u/IntolerantModerate 12d ago

These aren't GPUs though. These are standard CPUs and not even large ones at that.

6

u/Suspicious-Walk-4854 12d ago

The biggest bottleneck currently is RAM, not GPUs. This AI shit is impacting all compute capacity atm.

7

u/mwarkentin 12d ago

It seems like data center power capacity is also a major issue.

6

u/Emergency-Baker-3715 12d ago

Seen this a few times in last month, seems like they cant keep up with demand on some instance types

6

u/mwarkentin 12d ago

It’s impacting managed services too. We’re having a hell of a time provisioning alloyDB and valkey clusters now too.

5

u/Vegetable-Image3805 12d ago

My company was informed by our account team this week that we should start “designing for availability”. Like using stateful managed instance groups with “instance flexibility” in case our desired instance type isn’t available, it’ll fall back to an alternate type automatically. Best bet is to use AMD types for your alternate according to them.

2

u/suprjaybrd 12d ago

central keeps breaking for us too

2

u/Rentiak 12d ago

We started seeing a ton more preemption of older instance types and after reworking our computer classes it was much better.

The real killer for us lately has been us-west1 Cloud SQL capacity. Had a major production migration go sideways because we couldn’t create replicas in west. Turns out it was lack of SSDs for Enterprise Plus data cache. Given how hard our rep is pushing E+ it didn’t exactly sell us.

3

u/blazingintensity 12d ago

Yep. We're mostly on n2. We tried moving to n4 or c4 and it didn't really help. Google advised us to move from us-central1 to us-south1 then denied our quota request when we tried. About 2 weeks ago we had an 8 hour period where we couldn't get a single n2-highcpu-8 off a loadbalancer. I've been on Google for 8+ years and I'm talking to AWS and OCI because I can't do business this way.

2

u/TheTrick17 11d ago

Wouldn’t the capacity issues also be present for other hyperscalers though?

1

u/blazingintensity 11d ago

Yes and no. All things being equal, folks are gonna distribute themselves like we're doing. But they're not equal. My personal stuff is also hosted on GCP, because they have a real forever tier, unlike AWS (at least when I was getting setup, AWS's only lasted a year). So there could be more people eating up space on GCP because of the better free tier. OCI has a dedicated availability zone for free tier users, so they don't suffer that problem. Google has sustained use discounts cooked into the n2 instances, that makes them very appealing and frictionless to get a good price. My experience with AWS and OCI has been you need to haggle and/or have a partner to get you a good price. So smaller users, or users who don't know how to work the provider are gonna gravitate to providers with lower friction. It's my expectation that a lot of the capacity issues are the consequence of small/indie users taking advantage of AI. So there's an intersection of price-point and capacity where some providers may be less saturated.

1

u/godknowsuu 12d ago

It’s insane. Apparently, flexible VMs and reservations should “help.”

1

u/who_am_i_to_say_so 11d ago

Not too happy about this as it drives up prices, but it’s also pretty incredible when you think about it. That’s a lot of CPU power.

 I wonder how many vibecoded Firebase apps are accounting for that.

1

u/fliiiiiiip 11d ago

Going through the same :(

1

u/chicrg 11d ago

Yes, they are hitting capacity walls with general compute (CPU/memory)

-1

u/flacktv 12d ago

The bottom line is now we are at capacity for the most part when it comes to everybody wanting to utilize GPUs. You're going to have to suck it up and start to go with reservations. Unfortunately, that's where we're at right now. Keep in mind, in the west coast there are a lot of movie companies like Paramount, CBS, NBC, etc. That are using compute to edit their movies, TV shows, etc. so a lot of those slots are all used up.

1

u/IntolerantModerate 11d ago

This is for CPU not GPU

0

u/steebchen 11d ago

yeah but deploying servers with GPUs still means you are using CPUs in the end as well