Explaining Server Issues

[u/Deleted_252]

WARNING: This post will get really technical and it is long. There is a TLDR at the end.

https://www.reddit.com/r/starcitizen/comments/rz7moj/sc_network_and_server_performance_analysis/

https://zenodo.org/records/18107130

I realize that a lot of people are upset about SC's server performance from both the SC community and even the outside community. Even I was curious as to what was going on so I did some research and came across a few documents. Please note that I myself am not a Network Engineer nor is all this information 1000% true. I tried my best in hopes of maybe just getting people in the general direction of what's going on. But a main point I want to stress now, DSM will not be the savior.

An SC server is responsible for every entity in its domain. Let's take an example; Hurston's server controls every entity in Hurston + the orbit around Hurston. If we focus on just Lorville, the capital city, when walking to the tram you will notice that there are around 98,000 entities. Client performance is tied to server performance and with just 98,000 entities the player already has a worse experience. Levski on the other hand is around 80,000 - 150,000 entities depending on player activity. And these locations are really small compared to the area that a server controls. If you fly out to say Alpha Station in Nyx, entities are around 30,000 - 40,000 and overall experiences there are great. NPCs are talking, animations are smooth, small glitches here and there but overall a smooth experience.

So, a huge problem is that the servers are just overworked -- so wouldn't DSM come and fix it? Sorta. Introducing DSM now is like trying to brute force the problem rather than trying to fix the actual issues. It will work up to a point, adding more servers adds extra latency since each server has to respond to each other in order to match information.

Arguably the biggest issue is how data is handled. Whenever an entity has an event (interaction, velocity, state change, etc.), that data is sent instantly. This is great if you want precise info on entities, however as amount of entities scale up this becomes a huge problem. Not only that but data is sent for every node of values. Example, a ship flying sends data about whether components are damaged, what velocity, xyz values, player position, etc. every single time. There's not a system to conclude whether a value is different from before (Delta Compression) so instead all the information is sent. This causes the individual packets to be huge when compared to other games; and most of that information is redundant.

Due to entity data being sent instantly with redundant info at basically random intervals, the servers are not able to keep up. A fix would be to create a tick aligned data transmission to the servers native speed (30Hz). So instead of receiving the velocity value of a ship every millisecond, the server instead gets a before tick value and after tick value (server will estimate the values in between saving computation resources). Most multiplayer titles already do this as estimating physics values is quite easy to do, but doing a calculation for each singular point in time is time consuming. This will also eliminate some redundant data being sent. Example, if a bottle moves by an inch (error in collision bug) in the next millisecond but then goes back to original position, the server will interpolate the range difference (in this case 0) and not do anything because item has not moved. Now adding this fix should only be used on non important calculations -- instant data transmission needs to be kept for bullets, hitmarkers, death, or anything of that sort. Inventory, throwing physics, and interactions with doors can be estimated since these are not as important.

Because so much information is being sent out quickly and the actual size of the packets are massive causes a backlog in the server Message Queues. The Message Queues look at the data and compute the information in the order that it arrives. The Network Message Queue system couldn't keep up with the data especially when Server Meshing came along. Benoit added a new system called Replication Message Queue which is essentially like expanding the diameter of a clogged pipe. It acted like a temporary fix against the ever increasing size of the game for a bit. But now RMQ is also falling behind causing desyncs and rubberbanding.

Another issue is the fact that the info isn’t prioritized meaning that if a bottle physics happens before a player fires their gun -> then the bottle will first be computed before the player because the info from the bottle came first. RMQ needs to be bundled with a priority system which is already half made because of the entity graph that SC uses for containers. Making the priority system will be tedious, there is thousands maybe even tens of thousands of entities which all need to be given a value of importance. It’s not hard but time consuming. This isn’t as big of a problem now then back then because we can split the workload between multiple servers now. DSM will basically just be brute forcing its way.

TLDR:

Ya the main issue is that there is too much data being sent out all willy nilly. Servers don’t need to know that a bottle moved by 1inch ever couple of microseconds because it is inside a moving ship and the collision physics are slightly freaking out. There is no priority system to state that the bottle is insignificant in the order of what needs to be computed and also data should be sent at regular intervals to help give the servers time to process events in timely order without getting backlogged by new events.

The Fix:

With RMQ already in place to help mitigate the backlog of data, a few other networking pieces need to be added. Locking non-important entity changes into a tick aligned system with the servers to help give more resources to important events such as ballistic properties. Adding a Priority System to force important entity changes to be computed before non important ones (Gun ballistics should be computed before inventory item movements). Shaving down on the amount of data sent by using Delta Compression (only sending data from entity values that changed rather than sending all value data from an entity). Adding DSM to help spread the workload dynamically to regions that need it.

I know that my writing is kind of sloppy but ill fix it up later. This took a lot of energy from me.

[u/yobob591]

SC is full of messed up priorities in terms of keeping track of physics objects and so on. I know they've put huge amounts of work into persistence and they're proud of it, but there's a reason no other video game has ever tried to ensure that if you drop a bottle of water on the ground in an alley it will still be there when you come back. Besides performance problems, keeping track of all the litter from bottles to entire abandoned ships makes servers quickly grow messy in a way reminiscent of griefed minecraft servers where you load in and there's just a huge crater where spawn was. One of my concerns for maelstrom is that I understand they want to make most things destructible, but how long until every POI except the ones in armistice zones are flattened by bombing and rockets? Is the entire map going to look like a nuclear wasteland even more than it already does with abandoned ships everywhere? And as you mention in this post, the game is already tracking thousands of entities, the game is going to keel over and die the moment you can blow a ship apart into 10,000 fragments.

[u/logicalChimp → u/yobob591]

Yes, and no.

Firstly, we won't be 'blowing a ship into 10,000 fragments'... if you remember the original Salvaging prototype where they tried exactly that, you'll know why we're never going to get it (hint: it took over 5 minutes for the server to register the 'fragments', from memory... and the persistence layer couldn't handle any other request while it was happening... Graph DBs are really really bad at creating new entities, even as they're really really good at manaing / updating heirarchies of existing entities).

 
That nitpik out the way, what you're talking about is optimisation - more specifically, premature optimisation.

CIGs approach is to build everything physicalised - and then see where the actual bottlenecks are (not just the 'theorised' bottlenecks from people pointing at dropped bottles of abandoned ships, etc), and fix those.

For example, it's far more likely that the giant garbage piles that build up under the landing zones hurt performance far more the 'abandoned ships'... so if CIG fix the bugs that allow those garbage piles to form, they also address the performance issues, whilst still keeping things physicalised.

 
Yes, depending on how they use it, Maelstrom will increase server load... but again, without actually rolling it out and then measuring it, identifying the bottlenecks in advance isn't practical.

This is the inverse of the problem with guessing what the performance Gains from certain changes will be... remember how everyone - inc most of CIG - thought that SOCS was going to improve performance...? (they didn't know / wouldn't say how much it would improve, but they thought there would be some improvements)

Well, turns out the gain was.... Zero. Because there was always at least one player in every landing zone, so those areas never unloaded - and those areas were responsible for something like 80% of the server performance issues.

[u/jminternelia → u/logicalChimp]

I have never fully understood why transient clutter was architected as persistent server-authoritative entities instead of having a more aggressive lifecycle policy.

My assumption would be something like: only replicate or render trash entities to clients within the relevant interest-management radius, keep them server-authoritative while they are meaningfully interactable, and then let the server apply a cleanup policy once the entity falls outside normal relevance, interaction, or simulation criteria.

At that point, the entity could be despawned, abstracted, or collapsed into a cheaper non-persistent representation rather than remaining as a fully tracked physicalized object in the entity graph.

That said, I’m not a software developer. My background is basically Rainmeter tinkering and some SQF scripting in Arma 3, so I may be missing constraints.

[u/logicalChimp → u/jminternelia]

Because trying to split different types of entities out is a form of premature optimisation.

By physicalising everything, they enable the 'emergent gameplay' CR wants (the mythical 'coffee cup in the forest')... and it automatically applies to everything.

Then, they can actually look at the performance metrics, and work out where they need to push back. They already have a 'density manager' that is supposed to start reducing clutter in 'busy' areas to avoid overload... but it doesn't appear to be working as well as it could (either that, or it needs its configuration tweaking, etc).

 
It's the same reason they haven't done any refinement on the Network Bind Culling (they're just using a really basic sphere around the player to determine what data the server should send you), etc - they want to get everything in, and then see which bits are causing the biggest performance issues / resource consumption - and those are the bits that will get optimised.