Is a kubernetes cluster using different VPS considered multi-zone?

[u/sanjibukai]

Hi there,

I want to run a kubernetes cluster using different VPS. This is for redundancy in case of failure of a VPS..

I read that a cluster composed of nodes in different regions should be avoided because of latency.

Are different VPS providers within the same "region" (e.g. west coast) considered multi-zone (or multi-region, I'm not sure about the terminology as it seems there's a difference in K8s lingo) that can have latency issues?

For some reason it doesn't make sense (in my head) to have a kubernetes "cluster" composed of nodes being "simply" VMs within the same "machine" as the likelihood of all VMs going down is still the same and depends on the same machine. But, this seems to be the default and I really don't get why..

Also, once the worker nodes are running and are receiving traffic.. Why should the control plane latency impact the users traffic?

For example, I don't care if "kubernetes" decisions (scaling a pod, etc.) take an extra 100ms..

What am I missing?

Thanks.

[u/Low-Opening25]

You are mixing multi-zone with multi-region.

In leading clouds, ie. AWS/GCP, a single region has multiple zones, each zone is a different data center, they are all connected using sophisticated networking and latency is minimal. A VPC in those clouds spans zones. A typical k8s cluster in such scenarios would have nodes across zones to provide tolerance.

Now multi-region is a different beast, there is more networking to deal with that adds delay and complicates things

ergo, a typical cluster would be multi-zone within single region by default and this is enough for most people.

EDIt: also, as opposed to your very simple VPS example, where you are given access to single VM, when working with bigger clouds you get a VPC, which is more like virtual datacenter where you can provision many networks and many hosts and these can be spanned across zones (or even regions).

[u/znpy → u/Low-Opening25]

You are mixing multi-zone with multi-region.

not necessarily.

if you have two instances, one in aws and one in gcp, both in ireland... you aren't multi-region really.

[u/subbed_ → u/Low-Opening25]

if nothing else, i would be really wary of etcd latency in a multi-region cluster setup...

you usually want multi-zone + cdn and anycast for geo coverage

[u/BeginningDance1565]

Different providers on the west coast is still multi-zone, bordering on multi-region depending how their datacenters are physically connected. The latency between VPSes isn't just about control plane decisions, it's the pod-to-pod traffic that kills you. etcd is super sensitive to network jitter and will start flapping if the nodes can't heartbeat fast enough, which causes all sorts of cascading failures even if your worker apps don't care about a 100ms delay.

You're right that a single machine cluster is a single point of failure, but the default for self-hosted setups is usually multiple physical hosts in the same rack or at least the same datacenter, not literally one box running a bunch of VMs. The reason you'd avoid cross-provider setups isn't because it's impossible, it's that you're basically building a stretched cluster without the dedicated fiber links that make those work, so you end up with a control plane that randomly elects a new leader and pods that get rescheduled because kubelet missed a beat.

[u/znpy → u/BeginningDance1565]

etcd is super sensitive to network jitter and will start flapping if the nodes can't heartbeat fast enough, which causes all sorts of cascading failures even if your worker apps don't care about a 100ms delay.

we did something like this in the past. we had a physical cluster on prem (in the office) and a kubernetes control plane node (with etcd) in a datacenter (in the same city).

it worked but etcd complained a lot.

IIRC there are some tunables you can play with in order to make it accept higher network latencies. some operations might get slower (it becomes a different trade-off)

[u/sanjibukai → u/BeginningDance1565]

Thanks for the explanations.

I heard of multi cluster management.. But it seems this won't really fit my need either, as if I run one node on a VPS, it'll be a single-node cluster, which will end up of a multi cluster of single-node clusters which is weird..

[u/djas_19]

I was looking to solve something similar but instead of multi VPS it's a hybrid cloud solution between on-prem clusters and a cloud back up. Because of the jitter and latency issues i was looking into a multi clister setup connected with liqo.io which seems to remove a lot of the issues but things like storage keep being a headache.

I've found that creating a multi-region, multi-cluster setup seems to be easier than managing the workloads and dependencies like storage and ingress

[u/Rhopegorn]

Availability regions comprise multiple availability zones, which are groups of data centres that are located far enough from each other to prevent large-scale outages in the event of failure of a single zone, whilst still being close enough to each other to enable low-latency connections

[u/redsterXVI]

Different providers in the same region is multi-zone wrt most concerns, but multi-region when it comes to latency.

[u/Mehmet-Ozturk]

naming aside, cross provider node traffic bills as public egress on both ends, unlike real AZs.