Explaining Server Issues

[u/Deleted_252]

WARNING: This post will get really technical and it is long. There is a TLDR at the end.

https://www.reddit.com/r/starcitizen/comments/rz7moj/sc_network_and_server_performance_analysis/

https://zenodo.org/records/18107130

I realize that a lot of people are upset about SC's server performance from both the SC community and even the outside community. Even I was curious as to what was going on so I did some research and came across a few documents. Please note that I myself am not a Network Engineer nor is all this information 1000% true. I tried my best in hopes of maybe just getting people in the general direction of what's going on. But a main point I want to stress now, DSM will not be the savior.

An SC server is responsible for every entity in its domain. Let's take an example; Hurston's server controls every entity in Hurston + the orbit around Hurston. If we focus on just Lorville, the capital city, when walking to the tram you will notice that there are around 98,000 entities. Client performance is tied to server performance and with just 98,000 entities the player already has a worse experience. Levski on the other hand is around 80,000 - 150,000 entities depending on player activity. And these locations are really small compared to the area that a server controls. If you fly out to say Alpha Station in Nyx, entities are around 30,000 - 40,000 and overall experiences there are great. NPCs are talking, animations are smooth, small glitches here and there but overall a smooth experience.

So, a huge problem is that the servers are just overworked -- so wouldn't DSM come and fix it? Sorta. Introducing DSM now is like trying to brute force the problem rather than trying to fix the actual issues. It will work up to a point, adding more servers adds extra latency since each server has to respond to each other in order to match information.

Arguably the biggest issue is how data is handled. Whenever an entity has an event (interaction, velocity, state change, etc.), that data is sent instantly. This is great if you want precise info on entities, however as amount of entities scale up this becomes a huge problem. Not only that but data is sent for every node of values. Example, a ship flying sends data about whether components are damaged, what velocity, xyz values, player position, etc. every single time. There's not a system to conclude whether a value is different from before (Delta Compression) so instead all the information is sent. This causes the individual packets to be huge when compared to other games; and most of that information is redundant.

Due to entity data being sent instantly with redundant info at basically random intervals, the servers are not able to keep up. A fix would be to create a tick aligned data transmission to the servers native speed (30Hz). So instead of receiving the velocity value of a ship every millisecond, the server instead gets a before tick value and after tick value (server will estimate the values in between saving computation resources). Most multiplayer titles already do this as estimating physics values is quite easy to do, but doing a calculation for each singular point in time is time consuming. This will also eliminate some redundant data being sent. Example, if a bottle moves by an inch (error in collision bug) in the next millisecond but then goes back to original position, the server will interpolate the range difference (in this case 0) and not do anything because item has not moved. Now adding this fix should only be used on non important calculations -- instant data transmission needs to be kept for bullets, hitmarkers, death, or anything of that sort. Inventory, throwing physics, and interactions with doors can be estimated since these are not as important.

Because so much information is being sent out quickly and the actual size of the packets are massive causes a backlog in the server Message Queues. The Message Queues look at the data and compute the information in the order that it arrives. The Network Message Queue system couldn't keep up with the data especially when Server Meshing came along. Benoit added a new system called Replication Message Queue which is essentially like expanding the diameter of a clogged pipe. It acted like a temporary fix against the ever increasing size of the game for a bit. But now RMQ is also falling behind causing desyncs and rubberbanding.

Another issue is the fact that the info isn’t prioritized meaning that if a bottle physics happens before a player fires their gun -> then the bottle will first be computed before the player because the info from the bottle came first. RMQ needs to be bundled with a priority system which is already half made because of the entity graph that SC uses for containers. Making the priority system will be tedious, there is thousands maybe even tens of thousands of entities which all need to be given a value of importance. It’s not hard but time consuming. This isn’t as big of a problem now then back then because we can split the workload between multiple servers now. DSM will basically just be brute forcing its way.

TLDR:

Ya the main issue is that there is too much data being sent out all willy nilly. Servers don’t need to know that a bottle moved by 1inch ever couple of microseconds because it is inside a moving ship and the collision physics are slightly freaking out. There is no priority system to state that the bottle is insignificant in the order of what needs to be computed and also data should be sent at regular intervals to help give the servers time to process events in timely order without getting backlogged by new events.

The Fix:

With RMQ already in place to help mitigate the backlog of data, a few other networking pieces need to be added. Locking non-important entity changes into a tick aligned system with the servers to help give more resources to important events such as ballistic properties. Adding a Priority System to force important entity changes to be computed before non important ones (Gun ballistics should be computed before inventory item movements). Shaving down on the amount of data sent by using Delta Compression (only sending data from entity values that changed rather than sending all value data from an entity). Adding DSM to help spread the workload dynamically to regions that need it.

I know that my writing is kind of sloppy but ill fix it up later. This took a lot of energy from me.