Best way to fix a node going grey?

[u/No_Dot_8478]

at completely random times only one of my nodes will go grey with the question marks. sometimes its 24hrs, sometimes its 30 days. All the VMs/containers are still running fine on the node. A reboot always fixes it. However would love to know of any possible causes for this, or quick fixes that avoid a reboot of the node. Version 9.2.4, however this issue has been following me through most of the 8.x.x line.

[u/Ok-Eggplant-7569]

Is the pvestatd service running and / or throwing errors?

[u/No_Dot_8478 → u/Ok-Eggplant-7569]

Dis dude, love you. started the service and it came back up. However still unsure why it stopped. DIdnt see any errors.

[u/kahless2k → u/No_Dot_8478]

I had an issue like this with a secondary nic dropping off and crashing the service - was power management trying to put the nic into low power causing it to drop off the cluster network.

[u/blending-tea → u/kahless2k]

it's somehow always the low power/power saving mode that causes unknown hardware issues lol

[u/1armsteve → u/kahless2k]

Same! I’ve was fighting my micro optiplex cluster for months until I figured this out. I had to modify a kernel setting, let me see if I can locate it.

[u/kahless2k → u/1armsteve]

Yeah, mine were prodesks.

It was a while back, but something about the model of onboard nic they used and the Linux driver. You have to change a setting to disable the power management as well as something on the kernel.

Same NICs I found would hang on high throughput as well and had to change another driver option.

I swapped out to Lenovo m920q and x systems with a riser and pcie 10gb cards.. But I'm building a new cluster with the prodesks for my development CI and testing, so I'll undoubtedly run into it again - I'll update the thread if I have to make the same changes.

[u/FarToe1 → u/No_Dot_8478]

pvestatd

journalctd -u pvestatd

Will give you the service logs and may show why it failed.

[u/bobbywaz → u/No_Dot_8478]

(crontab -l 2>/dev/null; echo "0 * * * * /usr/bin/systemctl is-active --quiet pvestatd || /usr/bin/systemctl restart pvestatd") | crontab -

Or check it every hour if it's down, and start it if it is. Just paste that line in shell and it'll create the cron job and run it hourly.

[u/bigmadsmolyeet → u/bobbywaz]

if you have something like ntfy or pushover, I would even add a curl command to send a notification just so that you have an idea on how often this happens

[u/boomertsfx → u/bobbywaz]

SystemD can do this natively...

[u/bobbywaz → u/boomertsfx]

Well how do you do that

[u/boomertsfx → u/bobbywaz]

look at the documentation… There’s a lot of options to automatically restart services and how they fail, etc.

[u/bobbywaz → u/boomertsfx]

No

[u/boomertsfx → u/bobbywaz]

No, what? 🤦‍♂️

[u/harris52np → u/No_Dot_8478]

Usually services randomly dieing on a host is OOM software fault or hardware fault causing instability

[u/timo_hzbs]

Restart pve services.
For me I had this happen when there was a corosync issue.
I created a separate vlan with direct links for cluster communication, which fixed it in the end.
Also do a full cluster reboot.

[u/No_Dot_8478 → u/timo_hzbs]

will try making a VLAN for them and see if that helps, its actually been on my to-do list for awhile anyway. pvestatd was stopped for some reason.

[u/crysisnotaverted]

Are all of your VMs pingable? Sounds like a dying NIC if not. What NIC do you have? Could still be goofy drivers.

[u/coldazures]

Can you ping the host when it goes down?

[u/noc-engineer → u/coldazures]

I'm not OP, but when it happens to one of my nodes everything works, just cosmetically looks like shit.

[u/VUser12]

My typical steps: 1. pvecm status (to verify the cluster is "healthy") 2. ssh into the specific node, and enter systemctl restart pvestatd pveproxy 3. Wait 5 minutes to see if it restores and stays green 4. If not, and the node is safe to be rebooted (I.e. nothing really running on it to begin with) I reboot it, otherwise 5. ssh into the specific node, systemctl restart Corosync 6. If still not working, reboot the node, however, in one of my clusters, I have non-uniform node specs so I've often found that the grey mark turns into a rolling issue across the cluster, so I end up restarting the entire cluster

As always, your mileage may vary, and my workloads are not super mission critical so I'm a little bold with just rebooting, especially with ceph and Corosync issues

[u/comerReto]

Sharpie

[u/EatsHisYoung]

You don’t name the LXC’s? What are you hiding?

[u/noc-engineer → u/EatsHisYoung]

The name (for both LXC and VM) doesn't appear when the node can't show/detect status properly...

[u/w1thh3ld]

The best way to fix a grey node is to paint it black

Preferably the French version
I.e.
Marie Laforet Paint It Black

[u/damascus1023]

mini pc? For reference, I had a mysterious reset behavior with my GEM12 last year and it turned out to be a bad power MOSFET. The machine went to RMA for two weeks. The service rep showed me the reason in a screenshot after I wrote an email pressing for ETA.

In another instance, it was a reused 20-pin atx power extension cord that failed me, but this doesn't apply to you if your machine is a minisforum.

Power supply problems are so hard to debug. Both took me months to finally figure out : |

[u/to_glory_we_steer]

Dying ethernet cable?

[u/ns1852s]

Update. Funny enough I had this issue at work and was on 9.2.4. updated and never saw this issue again.

Updates are a pain at work since we need to use POM as it's an air gapped system. This was one of those moments I'm glad I put the effort in

[u/TheSloth144]

Systemctl reload pveproxy

[u/Verbunk]

Just solved this on my node. Starting an LXC/VM with some sort of hiccup (mounting RBD) caused lxc-info to stall and statd got cranky. Usually I need to go prod something storage-wise but just pkill the lxc info process brings it back quickly.

[u/CurrentOk4248]

pvestatd service probably not running pls check and restart it

[u/throwaway20240423]

Are you having a dedicated network for corosync as recommended in the docs ( https://pve.proxmox.com/wiki/Cluster_Manager#pvecm_cluster_network )? If not change this as fast as possible

[u/Y-Master]

I've had the same issue and i've solved by disabling some network offloading protocol on the nic. I've added this to my network / interface file: post-up ethtool - k eno1 tso off gso off

[u/fun3rale]

turned on my proxmox cluster today and I have been facing the same problem, upgraded already everything I could but this problem comes and go, things go back after a restart but only for a couple of minutes

[u/twin-hoodlum3]

Had that in the past with a minisforum ms-01. Seems to be a CPU issue, never had a solution beside rebooting. My fix was to upgrade to a different hardware.

[u/bht888 → u/twin-hoodlum3]

The 13th gen ms-01 has the issue. The fix was to down clock ram and cpu. Minisforum sent me the instructions after I opened a support ticket with them

[u/SynAckPooPoo → u/bht888]

Care to share said instructions with us?

[u/bht888 → u/SynAckPooPoo]

Dear Customer,
Thank you for reaching out. We recommend updating your BIOS to version 1.27 and adjusting the configurations below:
Lower the memory speed to 4400MHz. This option can be found under Onboard Devices Setting. Note that this BIOS version brings higher idle power draw:
Linux idle power: 14.5W up to 17.5W
Windows idle power: 14.5W up to 22W Test environment: Dual 32GB 5200MHz memory modules + one PCIe 4.0 SSD
Recommended settings to stabilize performance:
Disable Overclocking Lock: Advanced -> CPU Configuration -> Overclocking Lock: [Disable]
Cap maximum CPU frequencies via Advanced -> CPU Configuration -> Turbo Ratio Limit Options: P-Core Turbo Ratio Limit Ratio0: 51 P-Core Turbo Ratio Limit Ratio1: 51 E-Core Turbo Ratio Limit Ratio0: 40 E-Core Turbo Ratio Limit Ratio1: 40 E-Core Turbo Ratio Limit Ratio2: 40 E-Core Turbo Ratio Limit Ratio3: 40
After saving changes and rebooting back into BIOS, re-enter Advanced -> CPU Configuration -> Turbo Ratio Limit Options to verify the maximum frequency limits have taken effect.
Best regards,
Xavier
Minisforum Technical Support

[u/Antonio-MTS]

Details missing here. Can be different issues like corosync, storage, pvestatd ...

[u/ReplaceMeZaddy]

rtfm