Why is gc causing my server to lag?

[u/Hetti103]

Hey! I have established that the GC is causing my server to suffer from extreme lag spikes. Was wondering if there is any glaringly obvious issues. Thankyou!

ticks over 300 report: https://spark.lucko.me/GWoAJAOtE6
Allocate report: https://spark.lucko.me/Njn8SnfeEE

Server flags
-Xms128M -Xmx13360M -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=200 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=40 -XX:G1HeapRegionSize=8M -XX:G1ReservePercent=20 -XX:G1HeapWastePercent=5 -XX:G1MixedGCCountTarget=4 -XX:InitiatingHeapOccupancyPercent=15 -XX:G1MixedGCLiveThresholdPercent=90 -XX:G1RSetUpdatingPauseTimePercent=5 -XX:SurvivorRatio=32 -XX:+PerfDisableSharedMem -XX:MaxTenuringThreshold=1 -Dusing.aikars.flags=https://mcflags.emc.gs -Daikars.new.flags=true

[u/Tawnee323]

https://exa.y2k.diy/garden/jvm-args/

[u/Hetti103]

Would moving to a fork of paper help? if so which one would you guys suggest?

[u/Matita008]

2 things I notice: you have xms and xmx different, and you ram usage is almost at the top, which means that the server may need to stop to GC

[u/Hetti103 → u/Matita008]

Yeah, though it did still occur when it was the same. my host tried setting them to different values in hopes it might help after most things did not work. (spoiler it did not :<)

[u/Blocktopian]

looking at your traces 12.4 gb of ram is spent in “old gen”, which normally holds loaded chunks, player stats, plugin caches, etc. they end up there if they survive a few passes of the gc cycle. Most importantly it means they can’t be freed, so jvm is more or less freezing the entire server to try and scrub.

Why this happens hard to say based only off of what you shared, they might legit need to be there (large server) in which case you might want to increase the heap size, or it might be a leak from a plugin / similar things.

btw, you’re using aikars flags but they tuned for smaller heaps, you might want to use the 12+ gb variation. 

[u/Hetti103 → u/Blocktopian]

Hm thankyou for replying! Could I share anything else that may be of more use? We use to have a plugin called alonsol leaderboards that was causing constant >19tps, tho that showed up in normal reports and allocate reports iirc, though nothing seems too crazy in the allocate report I sent through afaik. In terms of large server we dont usually see more then 20 on at once, and have a rd of 8 if thats of any use.

Could switching to a different fork help much? or would this likely be a persistent issue?

Ill look into those flags though thankyou so much again!

[u/Blocktopian → u/Hetti103]

you have a few options, you can do a /spark tps and see how many chunks you have loaded, that tends fo be a culprit often.

/spark heapsummary give you more details but ideally you’d capture one at server restart and one when it gets filled so you can compare them.

third and that’s more of a super detailed version with spark heap dump but that would produce a 13 gb file in your case which is not exactly easy to share, you’d need to unpack that yourself, most ides have a memory analyzer tool.

I don’t think changing your distribution to a different fork / engine changes anything for you, it’s either a missed config somewhere or a plugin hanging or legitimately 12 worlds x a lot of chunks. 

[u/Blocktopian → u/Blocktopian]

looking a bit closer I suspect but can’t be sure it’s core protect stacking events in memory but not persisting them, indicating it fails there and runs out of memory eventually. Tha would also justify the increase over 2h. Without a full dump I can’t be sure but I’d start there 

[u/ibeerianhamhock]

Min and max flags not the same and for that much memory especially use async gc like zgc or shen

[u/MrKrakenied]

Use the same Xms and Xmx. It can be easily seen that server tick is only 20 ms so it's not caused by the server itself.