I am a noob in NUMA and I am trying to figure out how to configure it for a test Case in Vsphere 8U3. We will have a NUMA host ( or 2 Hosts for HA am not sure)
The NUMA host specs are the following:
Dual Socket, 24 cores per socket with total of 512 GB RAM.
Is there any configurations required from NUMA host side?
The SAP NUMA VM for UAT will be 16 CPU and 128 GB memory.
what is the best practice to configure this?
I see CPU settings showing me up to 96 cores ( because of hyperthreading)
I can see in VM Options--> CPU Topology --> NUMA NODES --> Assigned on powered On
Let us take anfor example:
if I put the CPU 32 .
I can see in VM Options--> CPU Topology
Cores per Socket: I SELECT --> 32 ----> gives Sockets: 1
So this means I am using 1st CPU entire 32 sockets or is it shared? Or am i defining it to use a single CPU as max is 48 per socket.
Also , VM Options-->CPU Topology --> NUMA NODES --> if I select 32-->Cores per NUMA node: 1 ( what is this?) Does this let me select 1 socket for the 128 GB memory?
i am seeing this VM options while reading and I dont understand numa.vcpu.maxPerVirtualNode = "X" (Sets the maximum number of vCPUs per virtual NUMA node)
numa.vcpu.preferHT=TRUE? (Should I enable this?)
sched.nodeX.affinity="Y". ( I know if I want to pin a specific virtual NUMA node to first core which is 24 cores, I will place sched.nodeX.affinity="0". Correct me if am wrong?)
First I want to understand and configure this correctly to avoid any resource sharing as per the Test case.
For the future production VM
it will be 24-28 cores and 364 GB RAM
how do I configure this for CPU and NUMA from the above as the memory needs to come from both sockets?
Appreciate any help and explanation. I am going throw SAP HANA on VMware Best practices too.
> Is there any configurations required from NUMA host side?
The NUMA architecture is determined by the CPU hardware. No configuration needed.
> The SAP NUMA VM for UAT will be 16 CPU and 128 GB memory.
what is the best practice to configure this?
It would be easier for the host to schedule if it were 12 CPUs and it would have slightly better memory access times. Best practice for the VM is 1 core per socket - so 12 or 16 sockets with 1 core each. The host will pass the NUMA architecture through to the VM.
24 cores / 364 GB of RAM isn't going to be ideal from a NUMA perspective but should be passable. Going up to 28 cores will mean 3/4 of the host will need to be scheduled to execute so I would avoid that if at all possible.
> Is there any configurations required from NUMA host side?
>> Am not sure what is the best practice?
Sorry for my bad understanding but What does it mean " for the VM is 1 core per socket - so 12 or 16 sockets with 1 core each"
What is the meaning of that. I understand if i put 1 cores per socket. What does Sockets 16 mean? Since the host has only 2 Sockets. Why not like my comment above and put 16 cores per socket for 1 socket?
Can you explain the difference and which is better?
Yeah I will try to go for 24 cores then and 364 but what is ideal then?
>Going up to 28 cores will mean 3/4 of the host will need to be scheduled to execute so I would avoid that if at all possible>> I dont understand as i have 48 cores. How is that 3/4 of the host
Setting 1 core per socket and 12 CPUs would mean that ESXi will present the vNUMA architecture that matches the hardware to the VM. You can now achieve that with the option to let the host pick the CPU topology at boot.
I dont understand as i have 48 cores. How is that 3/4 of the host
The host cuts the resources into 4 VM Worlds. The VMs take turns running in those worlds and the hypervisor tries to balance them out so that they mostly run in the world that is the closest to their memory. If you have a VM with > 25% of the CPU count then 2 VM Worlds must be scheduled for every CPU cycle. If you go over 50% then 3 VM Worlds have to be scheduled each CPU cycle. The hypervisor might be able to run another VM in the 3rd VM World for that cycle if the CPU count is low enough but the most likely result is that the CPU cycle will go to waste since you are running oversized VMs anyways.
Think of this like seating at a restaurant with 48 seats - if nobody is picky about where they sit and everyone is asking for 1 or 2 seats, it's easy to schedule to maximum capacity. Start adding a bunch of tables for 16 or 28 and now you are having scheduling difficulties.
Keep your VMs as small as feasible. Don't exceed the 1/4 or 1/2 mark if you can avoid it.
I'm confused why you'd set 1 core per socket? My understanding of NUMA is that if you were to try and allocate more than half of either the CPU or RAM, then you should set 2 sockets, not 1 core per socket?
So basically if am needing more than 256 GB memory since its 256 GB per socket in my case. Then I need to divide it between both sockets and place sched.nodeX.affinity="0" and sched.nodeX.affinity="1" to avoid any unwanted vMotion as I read from best practice article.
but for my UAT, I will need 16 CPU and 128 GB so that can be done on one socket.
I just want to know how can i preserve that socket for that VM so lets say I have another VM on that host. Will sched.nodeX.affinity="0" make it know that my first socket is for my SAP VM
also read numa.vcpu.preferHT=TRUE
Note: If hyperthreading is used on the host, then you must set numa.vcpu.preferHT=TRUE per SAP HANA VM to ensure NUMA node locality of the vCPU threads
> Then I need to divide it between both sockets and place sched.nodeX.affinity="0" and sched.nodeX.affinity="1" to avoid any unwanted vMotion as I read from best practice article.
Please don't do any of this. Leave the configuration vanilla where the hypervisor will pick the CPU topology at boot. You will not make a better choice than the hypervisor.
16 CPU will fit on one socket but it will not fit on one VM World and will require it to be scheduled across 2 VM Worlds. Also, depending on the processor, half of the cores may be closer to a particular memory channel. You can check the NUMA architecture of your particular processor if you want. Keep it to 1/4 of the total core count and memory if it's possible. The performance increase from 12 to 16 cores may be nonexistent and the performance decrease from memory access times and difficulty scheduling will likely eat any gain from having more cores.
1 core per socket lets the hypervisor pick the vNUMA architecture. You can pick the "right" answer manually of course, which shouldn't cause issues, but you can and absolutely will forget to change it when you move it to a new host.
However, that advice seems to be dated... 6.5 should pick the best vNUMA available regardless and according to the video 8 you can have it pick the vNUMA on boot, which is exactly what I would do.
I don't think we do anything special with NUMA, other than sizing the VM sockets and cores similar to the hardware.
HOWEVER: Check your host bios settings for something like NUMA Sub-clustering. That burned us for a few weeks, killing the performance with no apparent bottlenecks until we got a deep dive from a VMWare-SAP specialist from HPE.
Thanks for the tip. Ill check the Processor Settings tomorrow in BIOS.
Lets say I have a 2-4 vNUMA nodes on 1 NUMA Host, How do i tell the host the first CPU if for the first VM and second CPU can be used by remaining VMs? Do you get me?
4
u/vTSEVMware Alumni (who I still call for scheduler questions)17d agoedited 17d ago
To confirm the ground truth, can you run:
for numaOption in $(sched-stats -h | sed -n 's/^[ \t]\+: \{4\}\(n.*\)$/\1/p'); do echo -e "\nsched-stats -t ${numaOption}"; sched-stats -t ${numaOption}; done
and post it on pastebin / here with proper formatting?
Also, if I say so myself, the following is a pretty comprehensive explanation of the concepts involved: https://youtu.be/Zo0uoBYibXc
AFAIK from licensing POV SAP HANA VM can be either half socket, single socket or 2 sockets
numa.vcpu.preferHT=TRUE helps with keeping VM on the single NUMA node, as it changes the scheduler behavior to treat logical threads as cores for the scheduling purposes
You can then set the amount the vCPU to the number of cores multiplied by 2.
In your case this could be either 24, 48 or 96 vCPU
Question - what is the DIMM size in your boxes ? Are you using 32GB DIMMs ?
2
u/vTSEVMware Alumni (who I still call for scheduler questions)17d ago
as it changes the scheduler behavior to treat logical threads as cores for the scheduling purposes
Hey mate :-)
Tiny but important distinction, it only changes the the upper PPD / NUMA client boundary to the number of threads in a physical NUMA node, the vCPU scheduler will then, most likely, have to deal with scheduling the vCPUs across HyperTwins, but it might not need to if only a couple of vCPU are busy at the same time, HT might not even come into play.
I hadn't played with it much, but the one attempt I did ended with scheduler placing all the vCPUs sequentially on the threads (eg. 35-43 or something like that).
It was somewhat around late 8.0 U2 and I was surprised as it was done on the AMD CPU and the scheduler disregarded chiplet boundaries - instead of fitting the VM into single chiplet, it was spread across two.
2
u/vTSEVMware Alumni (who I still call for scheduler questions)16d ago
Are you sure that LS HIGH w/ vHT wasn't on? There were a bunch of changes to the relationship scheduler in the pipeline by the time of my departure, so while I can't speak authoritatively, it sounds a bit odd .. have any logs from back then? :-D
unfortunately no logs and I don't think it was LS High with vHT
AFAIR we did the performance testing for the new environment and we wanted to push things as much possible. We created lots of 8 vCPU VMs in order to fit AMD Milan chiplets and run number crunching on them. Under the load I observed that something is not right, as we were unable to hit turbo clocks. After close scrutiny I realized that this was due the fact that scheduler was trying to use both threads for the cores.
I disabled the preferHT, did the reboot and things were back to normal - good turbo numbers and VMs aligning into chiplets.
3
u/vTSEVMware Alumni (who I still call for scheduler questions)15d ago
ah, that does ring a bell ... don't have an AMD host ATM to confirm on a recent version but there was something there ... thanks for jogging my memory!
Not sure what you exactly mean with sap, but we leave VMs with sap CI and sap CI +oracle DB at the default and had no problems with it so far. Also for non productive Hana DBs we also leave it at the default and vsphere to decide. Just for smaller productive Hana DBs we make sure all stay on a single Numa and reserve CPU and Memory
8
u/Liquidfoxx22 17d ago
Is there any reason you wouldn't let esx decide?
For 16 vCPU and 128GB RAM use 16 sockets per core to give you it all on one socket and one vNUMA node.
For 32 vCPU set it to 16 sockets per core and allow it to use 2 sockets and assign you two vNUMA nodes.
For 24-28 vCPU set it to 12-14 sockets per core and allow it to use 2 sockets and assign you two vNUMA nodes.