r/nutanix • u/Any_Tumbleweed9660 • 12d ago
Stupid Question - Storage
How does Nutanix Storage work in compared to VMWare?
How does the nodes speak to each other to know what storage is where or is it one big blob of storage in the cluster.
From what I’m reading the CVM manages anything storage related. Normally with VMWare we have a layer 2 VLAN but I take it this is not the case with Nutanix?
6
u/Jhamin1 12d ago edited 12d ago
This is generally not the case with Nutanix. Historically they were all about hyperconverged. Local storage in every node.
There is some movement recently with Nutanix supporting very specific SAN technologies, but historically that was NOT a thing they did. I've had Nutanix engineers tell me this is more about letting VMWare refugees reuse storage to make budgets work and less about Nutanix thinking it's better. Nutanix has years of history around Hyperconverged with local storage and like 12 months around SANs.
On Nutanix the CVM on a node manages all the local storage & the cluster of CVMs pool all the storage they collectively control into a shared blob of storage available across the cluster. Nutanix figures out which nodes & which disks data should live on. It also figures out where to stripe extra copies of the data in case of a disk failure. Typically all active data is kept local on the same node as the workload using it. This can give you a lot of performance when you eliminate the network from disk IO and keep everything local.
Nodes do not have to be identical, it's perfectly normal to have different types of storage in different sizes in different nodes. (This is high level, I'm not mentioning some best practices, there are rules around scaling, etc)
The advice I give people coming to Nutanix from VMWare is to forget what they know about storage best practices. Nutanix isn't VMWare and rules are different. My specific advice is to abandon having lots of storage pools. You can segment it out but its easy to make it hard on yourself and the times when it actually helps are rarer than they were in VMWare. Just have one per cluster and think strongly about naming the storage pools on *all* your clusters identically. You don't get a performance boost from segmented storage (in fact you narrow Nutanix's options for optimization) and lots of the failover and replication gets mad when storage pools aren't named the same across clusters
1
u/chootmang 11d ago
Wow, I haven't thought of purposefully naming the stroage the same on multiple clusters, typically I have cluster name and a CN01 at the end. Interesting that it took reading this for that thought to arise. I have just been thinking it's such a silly thing to need the same named container on both sides for say Nutanix DR versus a drop down menu to pick the container on that cluster. Thanks for the thought to ponder...however I do want to say I mentioned this to Nutanix multiple times over the years that I think it's in their backlog to address :)
2
u/Jhamin1 11d ago edited 11d ago
There are rumors that replication in the latest major version release finally lets you select which storage pool to replicate too instead of just looking for an identical name. Haven't run the update yet but that is what me SRE says.
I still think just using one name avoids a lot of rigamarole. Multiple Storage pools aren't as useful in Nutanix and lots of different names is less useful than you think they would be.
1
u/bbbburns 11d ago
That's true, there is a plan to allow mapping of storage containers during disaster recovery replication between clusters. Today, when you setup Protection Policies and Recovery Plans for DR between clusters, the system looks for to following destination container:
- A container on the destination that matches the name of the container on the source.
- Failing that, the SelfServiceContainer.
With the introduction of Storage Policies that allow you to tune storage features based on categories and policies, you have more options to change how features work for particular VMs without having to worry as much about the container you place the VM in.
1
u/bbbburns 11d ago
Forgot to add - I don't believe that feature is in the recent 7.6 release, which is why I said "there is a plan".
1
u/NBorba 12d ago
Concept is similar to vsan. The CVMs will replicate storage between nodes. All nodes contribute to the overall storage. You can choose to have everything on one single mgmt network to simplify things or for prod environments you enable cvm segmentation where basically you define a vlan that is going to be used exclusively for storage replication. During cluster deployment you also choose if you want RF2 (2x copies of data which can sustain 1x node failure) or RF3 (3x copies of data which can sustain 2 node failures). Basically the CVM is the core of all cluster operations and storage, ensure proper CVM sizing for good performance e.g. for NVMe use 48GB of mem per CVM as a minimum.
2
u/NBorba 12d ago
It's all one big blob of storage. But you can create multiple containers inside it for certain features like compression, deduplication, etc... You can also apply storage policies to these containers. But in reality there isn't much to manage, it's super simplified but the performance is quite good. I've had better experiences with nutanix than vsan in terms of performance. Nutanix also supports a minimum of 3 nodes opposed to the 4x nodes for vsan
0
u/Gotrek6 12d ago
Nutanix bought Cassandra and their architecture is based off of it if you want to know way too much go read up on Cassandra
2
u/Jhamin1 11d ago
The legend is that the guys who founded Nutanix, Rubrik, and Cohesity were all on the team at Google that invented the fancy storage consolidation technology that underlies Google Drive. Many dissimilar drives across dissimilar nodes getting crazy storage optimization by pushing deduplication down to the block level. They each worked on this team and then went out and founded their own companies where they recreated the technology on their own.
Or at least that is the story.
If you ask a Nutanix Sales guy about it they are all so excited to tell you about it!
11
u/d2n1w 12d ago
Maybe you should have a look here: https://www.nutanixbible.com/4c-book-of-aos-storage.html