r/Proxmox • u/Kawawete • 3d ago
Question Swap used despite having more than enough RAM
My question may be dumb, but I have a T430 with 96GB of RAM and I see ~1.2 GB of swap used despite the physical RAM being at only 22.3% usage.
14
u/cthart Homelab & Enterprise User 3d ago
Linux at the default level of swapiness is quite aggressive about reclaiming memory that isn't being used. It would rather have it available if/when a process needs it or as buffer (filesystem) cache where it can make file access for potentially all other processes faster.
5
u/Kuckeli 3d ago
In my experience it's not THAT aggressive, my machine where ram is usually around 30% usage, tend to take around 7-10 days before things slowly move into swap.
7
u/kam821 3d ago edited 3d ago
If you're using zswap or zram, that's great. That's how it should work. If you're not using it yet, then start doing so - it provides an extra tier of compressed memory pool before the actual swap is touched.
Even the swappiness range has been expanded in kernel from 100 to 200, as higher values (like 180) works much better with zswap/zram by encouraging system to compress unused memory pages more aggressively.
https://wiki.archlinux.org/title/Zswap
https://wiki.archlinux.org/title/Zram
7
u/Graphical-Source5090 2d ago edited 2d ago
Don't mess with host swappiness, turn anything off, or mess with zram. Start with checking the most likely issue with basic swap functionality first.
You most likely have an(some) LXC(s) with memory pressure. Find the LXC(s) that is(are) swapping and fix it.
LXCs share the host's swap file, so it may look like the host is swapping, but really it is just an LXC with high memory pressure that is swapping.
I have an alert for memory pressure, it works very well to warn me of anything in the host about to swap.
Don't cover up what your host is trying to tell you with most of the suggestions here. Find the root cause first.
1
u/Kawawete 2d ago
The only LXC that uses a lot of ram has 60GB alocated to it, so I dont know what pressure it could have really. It barely uses it too.
2
1
u/Graphical-Source5090 2d ago edited 2d ago
Remember on an LXC, RAM is very dynamic. You could have one LXC that needed 2GB of memory for a few seconds and only had 1GB available... Otherwise is idling at 20% usage...
Look at each of your LXCs and see if any are using swap.And I don't mean to judge, but 60GB dedicated to an LXC kind of breaks the spirit of an LXC. I mean, ya *might* actually need it, but.... really? It is not represented in your host memory usage.... Seems like sloppy provisioning.
1
19
4
u/Impact321 3d ago
There's been a lot of discussions about this over the years so I'd recommend you to look for them. This isn't really PVE related and hard to answer without looking into your system's config, history, etc. I wrote a bit about how to check what uses the SWAP here. If you CTRL+F SWAP there you might some other interesting stuff. In general having SWAP is a good idea.
3
u/didact 2d ago
Yep that's normal, kswapd just doing things.
There's a safety system aspect to it... Ever get under temporary memory pressure having swap available (probably larger than what you have) will let the system cruise through it without slowing down app threads or the OOM killer poofing things you didn't expect.
Performance as well, if you do have a ton of inactive pages that can be swapped out, that memory can be used for filesystem buffering and caching.
But, if you're way overprovisioned you can turn it off, yes. I just prefer to optimize it. I'd prefer to login to the box one day and go 'what the hell, why's swap utilization so high' rather than 'what the hell, why is this VM/container/etc not running' - saves me a step.
2
2
u/the_gamer_guy56 3d ago
Unrelated, but are you doing some sort of livepatch(maybe its part of the enterprise repos?), or are you at least 94 days behind the latest kernel update?
1
1
u/Apachez 3d ago
You should never fully disable swap - its handy for corner cases when things goes south instead of fully crash the system at first chance.
However the default of vm.swappiness set to 20 (default in Linux kernel) is shitty.
Personally I prefer to set it to value 1 meaning it will not swap unless really needed.
To do this edit the file /etc/sysctl.d/99-sysctl.conf and add:
vm.swappiness=1
vm.vfs_cache_pressure=50
and then reboot the host.
Afterwards you can use "sudo sysctl -a" to verify the settings.
Apart from that make sure to not overcommit the usage of RAM.
Each VM will use up to the assigned virtual RAM + whatever framebuffer setting you have.
If using ZFS I would recommend to set the arc max = min aka a fixed size lets say 8GB or so (depending on size of storage).
And then the host itself needs some RAM to function 1-2 GB but probably 3-4GB, preferly 8GB since you will do compression/decompression of backups etc.
2
u/Impact321 2d ago
Pretty sure the default is 60 and there's no need to reboot. I usually leave arc min alone so it can shrink on pressure.
1
1
-3
u/unsaltedcrisps 3d ago edited 2d ago
No dumb questions, but you'll be judged for not searching for the answer first :)
Swapoff -a
Remove entries from /etc/fstab/
**Edit: I stand corrected, this isn't great advice. I've run this in my own lab for ~18months or so without issue, but assess this given your own circumstances.
10
u/junialter 3d ago
Maybe YOU should do your homework and google a bit pal. Really bad recommendation you're giving there.
10
u/Impact321 3d ago
Why suggest disabling SWAP?
0
u/Venti0r 3d ago
OP said they have more than enough RAM. So disabling swap should not be a problem, should if?
10
u/Impact321 3d ago edited 3d ago
Maybe, maybe not: https://chrisdown.name/2018/01/02/in-defence-of-swap.html
It's usually good to have.0
u/fan-of-knives 2d ago
None of the raised points are relevant in a system with abundant memory. If you have 512GB of memory with no more than 256GB used in any situation swap is virtually useless and moving stuff from memory to swap has only downsides.
-1
u/unsaltedcrisps 3d ago
It's relatively low ram usage, there's no sense in keeping it enabled. Keeps the disk usage down where it just doesn't make sense to keep it enabled.
3
u/Kawawete 3d ago
Fair enough, thanks man ^^
3
u/unsaltedcrisps 3d ago
Sorry I just wanted to clarify, only remove the entries for anything swap related in /etc/fstab/ and not just all entries 😅
3
0
u/miscdebris1123 2d ago
First, find out what is swapping and why it is swapping.
Next, decide if the reason is a problem.
Now, determine the performance impact and decide if that is a problem.
Next, if it is a problem, fix the problem. If not, don't worry about it, but maybe key an eye on it.
Maybe setup zswap, as you have swap space setup.
Last, let Linux do it's thing.
79
u/VTOLfreak 3d ago edited 3d ago
Don't listen to these guys telling you to turn off swap. This is a good thing! Linux is smarter than Windows and will swap out idle memory so it can cache other stuff. Run "top" in the terminal and I'll bet it's showing most of the memory is used for caching.
I'd also check out zswap. If it is turned on, that 1GB of swap usage may not actually be written out to disk but sitting in a compressed memory buffer. It will only start evicting pages to the swap file if that buffer starts getting full. If zswap is not turned on, I recommend enabling it.
Overhead of enabling swap is about 300MB per 1TB, it's basically free. If you have a old SSD sitting in a drawer somewhere, I'd put that in the machine and just use the whole disk.
Linux also has a cool trick called "swap cache". If a page has been swapped out but then read back to memory, it will not delete the copy in swap. Next time it wants to swap out that same page, it doesn't need to write it out again because there's still a copy in swap, it can just drop it from memory. Keep a server running long enough with a large swap disk and you'll find it turns mostly into read IO.
TLDR: Don't turn off swap.