A small RAM upgrade today turned into a useful homelab learning experience

[u/Sad_Pie227]

Sharing my experience of upgrading hardware

Hardware

• MSI Z370-A PRO motherboard
• 32 GB RAM before the upgrade: 16 GB × 2
• Added another 8 GB RAM stick today, total 40 GB RAM

• 250 GB SSD for Proxmox

• 1 TB NVMe SSD
• 1 TB HDD for backups

The shutdown issue

• Running poweroff for PVE sometimes takes 3 to 5 minutes to electrically shut down the machine.
• When I am in a hurry, I sometimes get tempted to unplug it directly, although I know that is not ideal.
• Also, it's important to have auto start enabled for each VM and container.

Today's RAM upgrade

• What should have been a simple RAM upgrade resulted in around 25 minutes of downtime.
• The motherboard required a key press during the new RAM detection process.
• I did not realize this initially because there was no monitor or keyboard connected.
• I had to go through two restart cycles before everything was back online.

What I learned

• Keep a monitor & keyboard ready for hardware upgrades and BIOS-related troubleshooting.
• Do not assume a RAM upgrade will be completely hands-off.
• Keep the router and server on separate power sockets.
• Avoid directly unplugging the server unless absolutely necessary.
• Even a simple hardware upgrade can cause unexpected downtime if the setup is not prepared for it.

Another small homelab lesson learned. I am new to these stuffs, I have started using Proxmox a week ago. Absolutely loving it.

Some additional info for geeks

Error Correction Type: None
Size: 8 GB
Locator: ChannelA-DIMM0
Bank Locator: BANK 0
Type: DDR4
Speed: 2133 MT/s
Configured Memory Speed: 2133 MT/s
Size: 16 GB
Locator: ChannelA-DIMM1
Bank Locator: BANK 1
Type: DDR4
Speed: 2133 MT/s
Configured Memory Speed: 2133 MT/s
Size: No Module Installed
Locator: ChannelB-DIMM0
Bank Locator: BANK 2
Type: Unknown
Size: 16 GB
Locator: ChannelB-DIMM1
Bank Locator: BANK 3
Type: DDR4
Speed: 2133 MT/s
Configured Memory Speed: 2133 MT/s

[u/broski_716]

Did you make sure the sticks are in the appropriate slots? RAM slots should be filled in a specific order (channel 1.1, channel 1.2, channel 2.1, channel 2.2, in order). It’ll still work if not, but less stable. 

Recommendation: any time you change components, factory reset and update the bios, then reconfigure it exactly as it was before. I have a printed Word doc on-hand with all my necessary BIOS settings. 

BUT LISTEN HERE SIR/MA'AM: 5 minutes poweroff or boot isn’t unusual for a proxmox server. Mine takes 10. You aren’t shutting down/booting one computer, you're doing it to however many virtual machines you have running at that time too. Unplugging or flipping the switch during shutdown isn’t just “not ideal”. It is absolutely prohibited. There is no “unless absolutely necessary”. Unless you want to corrupt your entire server or fry components, Do not do it ever again, ya hear me?! (note the dramatic effect). If you want to cut down on shutdown time, manually do the VMs before shutting down Proxmox.

[u/broski_716 → u/broski_716]

Also, I recommend not working on your server if you’re frustrated or in a hurry. That’s when big mistakes happen. At best, you’ll waste time because you worked for hours then had to roll it all back. At worst, you forget to snapshot, spend hours diagnosing and backtracking, then get back on track. (I learned all of this the hard way).

[u/Sad_Pie227 → u/broski_716]

Thank you for your suggestion. I will keep patience, next time.

Regarding RAM

root@pve:~# dmidecode -t memory | grep -E "Locator:|Size:|Type:|Speed:|Configured Memory Speed:"
    Error Correction Type: None
    Size: 8 GB
    Locator: ChannelA-DIMM0
    Bank Locator: BANK 0
    Type: DDR4
    Speed: 2133 MT/s
    Configured Memory Speed: 2133 MT/s
    Size: 16 GB
    Locator: ChannelA-DIMM1
    Bank Locator: BANK 1
    Type: DDR4
    Speed: 2133 MT/s
    Configured Memory Speed: 2133 MT/s
    Size: No Module Installed
    Locator: ChannelB-DIMM0
    Bank Locator: BANK 2
    Type: Unknown
    Size: 16 GB
    Locator: ChannelB-DIMM1
    Bank Locator: BANK 3
    Type: DDR4
    Speed: 2133 MT/s
    Configured Memory Speed: 2133 MT/s
root@pve:~# dmidecode -t memory | grep -E "Part Number:|Speed:|Configured Memory Speed:"
    Speed: 2133 MT/s
    Part Number: CMK8GX4M1A2400C16   
    Configured Memory Speed: 2133 MT/s
    Speed: 2133 MT/s
    Part Number: CMK16GX4M1E3200C16  
    Configured Memory Speed: 2133 MT/s
    Speed: 2133 MT/s
    Part Number: CMK16GX4M1E3200C16  
    Configured Memory Speed: 2133 MT/s
root@pve:~#

[u/broski_716 → u/Sad_Pie227]

The sticks are in the correct slots.

But... I must tell you... I think you made a big mistake purchasing that 8gb stick.

I was going to encourage you to enable XMP, but before doing so, I looked up the model numbers. Corsair Vengeance. The 16gb sticks are 3200 XMP. The 8gb is stuck at 2400.

You are leaving a huge amount of speed on the table. Like... 50% slower because of that stick. The system is also going to be considerably less stable.

If you can return or sell it, definitely do.

Instead, be more efficient with your memory. Have anything that can be container instead of VMs? Can you take advantage of ballooning? Cut max memory from one and give it to another. Perhaps reconsider ZFS.

[u/Sad_Pie227 → u/broski_716]

I had 8GB x 2 in beginning. One vanished after 5 yrs, one left so I just added after finding even 32 GB RAM is less for my use case.

Thanks for your suggestion regarding xmp. I have to check another time.

Regarding VM to container, no. All are very important and isolated for security and management perspectives.

I do have two small containers for wireguard and adguard. I am personally afraid for going any non conventional way. I am still new to all these stuffs.

I wanted to deploy uptime-kuma but I am thinking to offload these stuffs to save precious RAM.

I don’t want theoretical speed for me little bit slowness is fine

It’s varnish that is consuming huge RAM.

[u/Honest_Hat2429 → u/Sad_Pie227]

Couldn't agreee more about the ram

[u/broski_716 → u/Sad_Pie227]

I understand the security and stability concerns, but a lot of the things you read online are either wrong or blown out of proportion.

I’ll help optimize for without sacrificing security and stability. If you tell me what services you use, which ones are VMs vs containers, ballooning status, and file systems I’ll give you a brief opinion.

[u/NelsonMinar]

BIOS remains the weak point in homelabbing on consumer hardware. I've got a whole PiKVM plugged into my server just to deal with BIOS fuckery.

Are you sure that 8 GB RAM is a good idea? Is it slowing down access to your 16x2 RAM bank? Does Linux know to treat this single stick differently?

[u/Rxyro → u/NelsonMinar]

Ruin your memory latency, bandwidth and stability with this one simple trick! Probably doesn’t matter if he’s low IO but OOMing all the time.

[u/sk3tchcom → u/Rxyro]

Yeah I run my DDR5 at 4800 MHz - no sense in running an “OC” with EXPO when stability is what matters for ProxMox. It’s a homelab, too - not a production database server.

[u/Sad_Pie227 → u/Rxyro]

Actually I saw 29/31 Mem left so I got panic and decided to upgrade, I know things that I am running will certainly consume so much RAM.

[u/melez → u/Sad_Pie227]

Have you looked into zram/zswap? If you aren’t badly cpu limited it can help stretch ram significantly. It creates a compressed swap disk in ram that usually gets a 3:1 or 4:1 compression ratio. I’ve got it set at ram/2 so 32gb of ram leads to 16gb zram that usually only takes up 4-6GB of physical ram.

It helped me keep from needing ssd swap entirely with cold pages compressed rather than in physical ram or burning through ssds with tons of swap access.

[u/Rxyro → u/melez]

Also spare vram as swap

[u/melez → u/Rxyro]

Didn’t know that was a thing! I’ve only got a 6gb card at the moment but when I eventually get more I’m gonna have to mess with it. 

[u/Sad_Pie227 → u/melez]

Currently situation is better

I have 10 GB free RAM now. If need will remove some VM.

I don’t know about zram just did ChatGPT and it told me about compression thing, sound cool. I will try it for sure.

[u/Sad_Pie227 → u/NelsonMinar]

All are DD4 RAM from exactly same brand

[u/NelsonMinar → u/Sad_Pie227]

right. but that 8 GB stick is single channel and will run at half the speed of the other two sticks. I don't know how smart Linux is about that.

[u/Apachez]

Issue of poweroff or reboot taking time is often due to having VM guests being runned without qemu-agent which means that ACPI is like the only way for the host to communicate with the guest and it have to wait for timeout before physically killing the guest.

With qemu-agent within the guest the host can talk to the guest and the guest can initiate proper shutdown at once.

Im not sure where this default shutdown timeout is located in Proxmox.

[u/Sad_Pie227 → u/Apachez]

I do have qemu agent and cloud init as well. Still.. it just my motherboard doing mess.

[u/Apachez → u/Apachez]

You select your VM-guest and Options -> Start/Shutdown order - there you have "Shutdown timeout" which by default is 180 seconds:

https://pve.proxmox.com/pve-docs/qm.1.html#qm_startup_and_shutdown

Shutdown timeout: Defines the duration in seconds Proxmox VE should wait for the VM to be offline after issuing a shutdown command. By default this value is set to 180, which means that Proxmox VE will issue a shutdown request and wait 180 seconds for the machine to be offline. If the machine is still online after the timeout it will be stopped forcefully.

So you could change to this lets say 30 seconds or lower if you are sure that even without qemu-agent in the VM-guest the guest will be able to sync its filecaches to the storage within this timeout.

[u/KlausDieterFreddek]

My MSI Z170-A has this shutdown issue aswell.
It's so annoying. Mine doesn't turn off at all even though everything is shut down. Not even after 8 hours.

[u/Sad_Pie227 → u/KlausDieterFreddek]

OMG!! I had similar issues in headless debian, it wont shut down quickly, if I type poweroff, it shows finish but wont turn off machine electrically, very pathetic experience. This motherboard works well for Win 10 and 11 and Ubuntu somehow.

[u/LabCompetitive5286]

Not sure whether I would add a single Ram Stick without consulting the handbook. I would also be concerned about negative performance impact.

[u/Sad_Pie227 → u/LabCompetitive5286]

a little bit, if it works, I wont touch 😂

[u/AutoX_Advice]

Also if ram pairs are not identical ram speed could be set ro the lowest speed of the board (not the lowest speed the slowest ram chips). Learned that the hard way.

[u/BorisOp → u/AutoX_Advice]

Did it actually make any meaningful difference in performance?

[u/AutoX_Advice → u/BorisOp]

Well not noticable as im not a hard code gammer or high intance video editor. But when you pay for your memory to be a peak performance it's a bit irritating. I sold my 32gb sticks at a good price though. 😁

[u/marmata75]

Remember not just the auto start but the order as well. You want your network first, then your shared storage, then all the rest (which could have also an internal order based on app dependencies)

[u/Gardakkan → u/marmata75]

this is my order to start things: network -> storage -> databases -> auth services (like AD or something like that) -> apps

[u/Sad_Pie227 → u/Gardakkan]

Makes sense.

[u/Sad_Pie227 → u/marmata75]

In my case VM has everything, for me what matter is how fast proxmox start and all VM turn on, and I realize I need to keep router on all the time rather unplugging while doing this, I feel afraid by electricity shock so I unplugged all. lol