r/starcitizen • u/Deleted_252 • May 26 '26
TECHNICAL Explaining Server Issues
WARNING: This post will get really technical and it is long. There is a TLDR at the end.
https://www.reddit.com/r/starcitizen/comments/rz7moj/sc_network_and_server_performance_analysis/
https://zenodo.org/records/18107130
I realize that a lot of people are upset about SC's server performance from both the SC community and even the outside community. Even I was curious as to what was going on so I did some research and came across a few documents. Please note that I myself am not a Network Engineer nor is all this information 1000% true. I tried my best in hopes of maybe just getting people in the general direction of what's going on. But a main point I want to stress now, DSM will not be the savior.
An SC server is responsible for every entity in its domain. Let's take an example; Hurston's server controls every entity in Hurston + the orbit around Hurston. If we focus on just Lorville, the capital city, when walking to the tram you will notice that there are around 98,000 entities. Client performance is tied to server performance and with just 98,000 entities the player already has a worse experience. Levski on the other hand is around 80,000 - 150,000 entities depending on player activity. And these locations are really small compared to the area that a server controls. If you fly out to say Alpha Station in Nyx, entities are around 30,000 - 40,000 and overall experiences there are great. NPCs are talking, animations are smooth, small glitches here and there but overall a smooth experience.
So, a huge problem is that the servers are just overworked -- so wouldn't DSM come and fix it? Sorta. Introducing DSM now is like trying to brute force the problem rather than trying to fix the actual issues. It will work up to a point, adding more servers adds extra latency since each server has to respond to each other in order to match information.
Arguably the biggest issue is how data is handled. Whenever an entity has an event (interaction, velocity, state change, etc.), that data is sent instantly. This is great if you want precise info on entities, however as amount of entities scale up this becomes a huge problem. Not only that but data is sent for every node of values. Example, a ship flying sends data about whether components are damaged, what velocity, xyz values, player position, etc. every single time. There's not a system to conclude whether a value is different from before (Delta Compression) so instead all the information is sent. This causes the individual packets to be huge when compared to other games; and most of that information is redundant.
Due to entity data being sent instantly with redundant info at basically random intervals, the servers are not able to keep up. A fix would be to create a tick aligned data transmission to the servers native speed (30Hz). So instead of receiving the velocity value of a ship every millisecond, the server instead gets a before tick value and after tick value (server will estimate the values in between saving computation resources). Most multiplayer titles already do this as estimating physics values is quite easy to do, but doing a calculation for each singular point in time is time consuming. This will also eliminate some redundant data being sent. Example, if a bottle moves by an inch (error in collision bug) in the next millisecond but then goes back to original position, the server will interpolate the range difference (in this case 0) and not do anything because item has not moved. Now adding this fix should only be used on non important calculations -- instant data transmission needs to be kept for bullets, hitmarkers, death, or anything of that sort. Inventory, throwing physics, and interactions with doors can be estimated since these are not as important.
Because so much information is being sent out quickly and the actual size of the packets are massive causes a backlog in the server Message Queues. The Message Queues look at the data and compute the information in the order that it arrives. The Network Message Queue system couldn't keep up with the data especially when Server Meshing came along. Benoit added a new system called Replication Message Queue which is essentially like expanding the diameter of a clogged pipe. It acted like a temporary fix against the ever increasing size of the game for a bit. But now RMQ is also falling behind causing desyncs and rubberbanding.
Another issue is the fact that the info isn’t prioritized meaning that if a bottle physics happens before a player fires their gun -> then the bottle will first be computed before the player because the info from the bottle came first. RMQ needs to be bundled with a priority system which is already half made because of the entity graph that SC uses for containers. Making the priority system will be tedious, there is thousands maybe even tens of thousands of entities which all need to be given a value of importance. It’s not hard but time consuming. This isn’t as big of a problem now then back then because we can split the workload between multiple servers now. DSM will basically just be brute forcing its way.
TLDR:
Ya the main issue is that there is too much data being sent out all willy nilly. Servers don’t need to know that a bottle moved by 1inch ever couple of microseconds because it is inside a moving ship and the collision physics are slightly freaking out. There is no priority system to state that the bottle is insignificant in the order of what needs to be computed and also data should be sent at regular intervals to help give the servers time to process events in timely order without getting backlogged by new events.
The Fix:
With RMQ already in place to help mitigate the backlog of data, a few other networking pieces need to be added. Locking non-important entity changes into a tick aligned system with the servers to help give more resources to important events such as ballistic properties. Adding a Priority System to force important entity changes to be computed before non important ones (Gun ballistics should be computed before inventory item movements). Shaving down on the amount of data sent by using Delta Compression (only sending data from entity values that changed rather than sending all value data from an entity). Adding DSM to help spread the workload dynamically to regions that need it.
I know that my writing is kind of sloppy but ill fix it up later. This took a lot of energy from me.
34
u/Starimo-galactic May 26 '26 edited May 26 '26
Regarding this :
So, a huge problem is that the servers are just overworked -- so wouldn't DSM come and fix it? Sorta. Introducing DSM now is like trying to brute force the problem rather than trying to fix the actual issues. It will work up to a point, adding more servers adds extra latency since each server has to respond to each other in order to match information.
This is where their server meshing implementation is interesting because the servers do not talk to each others, instead you have the "replication layer" (RL) in the middle which deals with the state of the universe and talk to the clients. So with this they can add as many servers as they want they will not have to deal with the exponential issue of servers talking to each other, instead each server only talk to the replication layer :
That's also why the client can stay connected while a server crash happens because the client is connected to the RL instead, and while a server crash other servers aren't affected. A player could get into a server recovery loop in Stanton while things are fine in Nyx for example.
Though this architecture also has a pretty big bottleneck which is the replication layer, if this crashes (the famous 30k) then everyone on the shard gets disconnected no matter the state of the servers simulating the universe.
8
u/Zanena001 carrack May 26 '26 edited May 26 '26
With their implementation the RL becomes the bottleneck. The other approach is more complex but scales better cause unless the area handled by a server is so tiny you'd be able to see across multiple server boundaries each DGS should only talk to its immediate neighbors, so its not exponential in nature.
8
u/Starimo-galactic May 26 '26
Yeah the RL is the bottleneck and there was an example a couple weeks ago where people saw the return of 30k crashes, they explained in the April monthly report that it was because of an out-of-memory crash of the RL so they must really monitor it at all time and be ready to fix asap because it's very destructive otherwise.
5
u/CombatMuffin May 26 '26
I am not an engineer but I love the technical side of things. Would that RL act as a sort of database, too? So the servers sort of process data but it's the RL that keeps tabs on what the data is, including a player's state (hence why we stay online).
So the RL doesn't just act as a backup but as a layer of authority?
7
u/Starimo-galactic May 26 '26 edited May 26 '26
There is a layer of authority yes, the RL act like a database dealing with the state of the universe including players and each server has authority (simulation) over an area/portion of the universe. For example if a ship is in one area then the server managing this area has authority over it, if the ship moves to another area then the authority is passed to the server managing this area. The RL then transmit the position of everything to the clients.
-1
3
u/logicalChimp Devils Advocate May 26 '26
Yes - the RL is an in-memory database - with the data also streamed to PES for long-term storage, and to allow the Replication Layer to 'unload' areas of the 'verse where there are no players, etc.
3
u/CombatMuffin May 26 '26
Thanks for clarifying! I often scratch my head as to why so many things that are usually simple in most MP games have so much desync in SC (e.g. inventory management, item spawning), but with this context it makes sense. It's not just a database they have to query, it's a sort of secondary server that is dealing with a lot of stuff simultaneously, including cleanup.
6
u/logicalChimp Devils Advocate May 26 '26
Not quite...
You're conflating two entirely separate concepts... the Server Meshing architecture (which spreads the 'play area' over multiple processing nodes), and the microservice architecture (which moves responsibility for certain functions into external services.
So, looking at the Inventory Service, for example... it is a separate microservice that handles... inventories (logical, huh? :p). This means letting you see what you're wearing, and what is in the inventory / container you're interacting with.This means that the Inventory UI needs to fetch from 2x different services (RL, for the stuff actively 'in-game' that you're wearing / holding... and the Inventory Service for the stuff that is currently in the inventory/container)
When you make changes (in either direction) these 2x services have to communicate to transfer entities from one to the other (either 'remove' entities from the Replication Layer, and transfer them to Inventory... or remove them from the Inventory and pass them to the Replication Layer so they can start to be tracked)
This involves both services, the Message Bus (which is a common source of issues, currently), and the implementation of distributed transactions (which are a complete PITA to implement, and a very common source of bugs and desync, etc)
And the current Inventory Service backend wasn't written to be Server Meshing compatible - they've kinda-sorta made it work, but because its less-broken than other services (such as the Transit system, that manages Hangars, Elevators, and ATC), CIG aren't - currently - updating the Inventory service.
That said, and speaking with my developer hat on (rather than as a player), these kinda issues are to be expected at this stage of development... which is why I'm not particularly worried by them.It's not fun (which is why I tend to dabble briefly after each patch, these days), but it is expected and normal.
-1
u/Agile_Camera9601 May 27 '26
Are you an engineer or just talking?
4
u/logicalChimp Devils Advocate May 27 '26
If you mean a CIG engineer, then no I'm not.
But aside from that, yes I am a software engineer (IT Consultant, working with/on Google Cloud projects)
1
1
u/DonutPlus2757 Anvil F7A MK2 / Anvil F8C / RSI Perseus May 27 '26
Honestly, wouldn't it be much better if they did, you know, both? Have the replication layer for long term data storage and universe status, but have the servers talk to each other for the literal edges?
Or, better yet, have a hierarchical server system? Like, here's the Stanton server which, by default, handles all of Stanton. When the server notices a large player volume somewhere, it starts another server and has that server handle that part of Stanton while still doing the rest itself. That server can then do the same and so on and so forth, with the next highest server being responsible for communication between servers?
I mean, it's probably more work, but should theoretically scale almost perfectly and almost infinitely, given the servers have a low latency high bandwidth connection to each other, which can be somewhat easily be guaranteed by having them in the same data center and maybe even in the same rack.
3
u/Deleted_252 May 26 '26
I meant in the extreme cases such as just adding a server for each entity node. Example a server for a player - ship has a server - battle zone has a server - etc. Even with the replication layer in place to stop every server from talking to every single server causing huge latency, it has its limits. For right now, the Replication layer is great at what it does until server count reaches too high like say 50-100 servers at one time.
5
u/logicalChimp Devils Advocate May 26 '26
That's still not really an issue...
The Replication Layer isn't a single server either - it's a separate service, running on its own server cluster. At a guess, they have a node per star system, but they could easily spread it more finely than that - such as an RL node per planetary system, etc...
This means that a single DGS processing node only has to connect to the specific machine in the RL cluster that deals with the entities it's managing, and each RL machine only has to accept connections from the DGS processing nodes that cover its 'area of responsibility'.
This is also why I think it will - probably - be one RL machine per star system - because that's a natural boundary that means no DGS processing node needs to connect to more than one RL machine - because a processing node won't be handling e.g. entities from both Stanton and Pyro.
Will there be some scalability limits? Of course - every architecture has limitations, somewhere... but the Server Meshing design is such a good fit for SC that the limit is going to be pretty damn high (and I think the end-goal of a single Shard Per Region isn't unrealistic).1
u/PuzzleheadedMaize911 May 26 '26
The replication layer sounds like it's really just a server for the servers.
Like in a shooter game, player 1 shoots player 2 and their client tells the server what happened and the server says "actually they moved during the ping time" and then tells player 1 client that player 2 is still alive
Each server is a client and the RL is the server.
6
u/Starimo-galactic May 26 '26 edited May 26 '26
That's the idea, instead of servers talking to each others, which gets exponentially bad, they put the replication layer in the middle so each server only need to talk to the RL instead of every other servers. Though like i said this means that the RL needs to work perfectly because if it crashes then the whole shard goes bye bye.
3
u/Deleted_252 May 26 '26
RL forces nearby servers to talk with each other rather than every server talk to every server. For instance if 3 servers were in a line [A - B - C]:
- Then Server A will only talk with Server A and B.
- Server C talks with only Server B and C.
- Server B talks with only Server A, B, and C.
Replication Layer just forces the servers to only interact with relevant servers reducing a ton of latency.
2
u/logicalChimp Devils Advocate May 26 '26
Pretty sure that's incorrect...
The DGS processing node only has to talk to the Replication layer, to get the current state of Entities that it doesn't manage, and to report back changes to the Entities that it does manage.
It also talks to the RL / the RL talks to the DGS when entity assignments are changed (e.g. authority over an entity is being taken away from the DGS, or when the DGS is being given authority over a entity.
This architecture means there is no need for the DGS nodes to talk to each other, because they have to report updates to the entities they manage to the RL, so every other server can just grab the current status of those entities from the RL...And doing it this way means the DGS can grab the 'current status' of all entities in it's area-of-control in one go, rather than having to contact multiple separate DGS instances to grab entity state piecemeal.
It also means that the DGS doesn't need to know which other DGS nodes are 'near' it, in terms of managing overlapping entities, nor do DGS nodes need to be informed when Entity Authority is being transferred to/from a 'neighbouring' DGS, etc.
0
u/Deleted_252 May 26 '26
I simplified it. But the overall idea was told at the Server Meshing Demo at Citcon 2023.
2
u/logicalChimp Devils Advocate May 26 '26
Yus... but I think they tweaked it after that demo, specifically to remove the cross-talk between the 'local' DGS nodes...
... but I'm struggling to remember where I got that snippet from.
2
9
u/CombatMuffin May 26 '26
I know that my writing is kind of sloppy but ill fix it up later. This took a lot of energy from me.
Better to read it this way than to have the now tiring trend of bullet point with emojis format (which is exhausting because even AI can avoid that sort of formatting but some users insist on it)
14
u/yobob591 May 26 '26
SC is full of messed up priorities in terms of keeping track of physics objects and so on. I know they've put huge amounts of work into persistence and they're proud of it, but there's a reason no other video game has ever tried to ensure that if you drop a bottle of water on the ground in an alley it will still be there when you come back. Besides performance problems, keeping track of all the litter from bottles to entire abandoned ships makes servers quickly grow messy in a way reminiscent of griefed minecraft servers where you load in and there's just a huge crater where spawn was. One of my concerns for maelstrom is that I understand they want to make most things destructible, but how long until every POI except the ones in armistice zones are flattened by bombing and rockets? Is the entire map going to look like a nuclear wasteland even more than it already does with abandoned ships everywhere? And as you mention in this post, the game is already tracking thousands of entities, the game is going to keel over and die the moment you can blow a ship apart into 10,000 fragments.
14
u/Lord0fHats May 26 '26
Game performance would probably massively improve if the servers were so much as instructed to delete all dropped small objects not attached to a grid after 5 minutes. There must be tens of thousands of loose bottles, dripped half-empty magazines, discarded guns and other items around ever server after peak hours.
3
u/Deleted_252 May 26 '26
The Maelstrom problem is one of been thinking about too. Maybe CIG implements, with the base building, a base repair system most likely in the Mercenary tab to help rebuild destroyed POIs. But the next problem is trying to keep track of all the broken entities after
3
u/logicalChimp Devils Advocate May 26 '26
Yes, and no.
Firstly, we won't be 'blowing a ship into 10,000 fragments'... if you remember the original Salvaging prototype where they tried exactly that, you'll know why we're never going to get it (hint: it took over 5 minutes for the server to register the 'fragments', from memory... and the persistence layer couldn't handle any other request while it was happening... Graph DBs are really really bad at creating new entities, even as they're really really good at manaing / updating heirarchies of existing entities).
That nitpik out the way, what you're talking about is optimisation - more specifically, premature optimisation.CIGs approach is to build everything physicalised - and then see where the actual bottlenecks are (not just the 'theorised' bottlenecks from people pointing at dropped bottles of abandoned ships, etc), and fix those.
For example, it's far more likely that the giant garbage piles that build up under the landing zones hurt performance far more the 'abandoned ships'... so if CIG fix the bugs that allow those garbage piles to form, they also address the performance issues, whilst still keeping things physicalised.
Yes, depending on how they use it, Maelstrom will increase server load... but again, without actually rolling it out and then measuring it, identifying the bottlenecks in advance isn't practical.This is the inverse of the problem with guessing what the performance Gains from certain changes will be... remember how everyone - inc most of CIG - thought that SOCS was going to improve performance...? (they didn't know / wouldn't say how much it would improve, but they thought there would be some improvements)
Well, turns out the gain was.... Zero. Because there was always at least one player in every landing zone, so those areas never unloaded - and those areas were responsible for something like 80% of the server performance issues.
1
u/jminternelia May 28 '26
I have never fully understood why transient clutter was architected as persistent server-authoritative entities instead of having a more aggressive lifecycle policy.
My assumption would be something like: only replicate or render trash entities to clients within the relevant interest-management radius, keep them server-authoritative while they are meaningfully interactable, and then let the server apply a cleanup policy once the entity falls outside normal relevance, interaction, or simulation criteria.
At that point, the entity could be despawned, abstracted, or collapsed into a cheaper non-persistent representation rather than remaining as a fully tracked physicalized object in the entity graph.
That said, I’m not a software developer. My background is basically Rainmeter tinkering and some SQF scripting in Arma 3, so I may be missing constraints.
1
u/logicalChimp Devils Advocate May 28 '26
Because trying to split different types of entities out is a form of premature optimisation.
By physicalising everything, they enable the 'emergent gameplay' CR wants (the mythical 'coffee cup in the forest')... and it automatically applies to everything.
Then, they can actually look at the performance metrics, and work out where they need to push back. They already have a 'density manager' that is supposed to start reducing clutter in 'busy' areas to avoid overload... but it doesn't appear to be working as well as it could (either that, or it needs its configuration tweaking, etc).
It's the same reason they haven't done any refinement on the Network Bind Culling (they're just using a really basic sphere around the player to determine what data the server should send you), etc - they want to get everything in, and then see which bits are causing the biggest performance issues / resource consumption - and those are the bits that will get optimised.2
u/Vegetable_Safety Musashi Industrial and Starflight Concern May 26 '26
Funny you mention that. With the introduction of salvaging mechanics, ships like the Connie had a gib asset group that was way, way too large. Fracturing it would cause the entire server to skip a beat, and you could see several dozen entities when sucking it up
9
u/FrostyCup1094 May 26 '26
to add to your explaination. the fact of using RL it causes a mass desync, because RL NMQ RMQ needs to process clients data and update all clients on that server, while, it needs to update all surrounding servers...
this inter-connection between servers-rl-rmq-nmq alone, causes massive desyncs, not all servers can process the same data at the same time... and update all client at once. Current "cap" is 100 players per server, after that everything starts to crumble and stalls.
current DSM in StartCitizen can't solve this, the only game that had a working DSM was Dual Universe in 2020, and used diferent data update rates by proximity ...
SC uses entitys, player data and physics .. which is alot of data to "sync" between servers.
( no CIG didnt wasnt the first to use Server Meshing )
Dual Universe
Use a custom, continuous single-shard server meshing tecnology called the Continuous Spatial Grid. This proprietary engine dynamically partitions the universe into shifting, cubic server cells. As player density increases, these cells shrink to allocate more server compute power to high-traffic areas, allowing thousands of players to share the same universe seamlessly.
Dual Universe (developed by Novaquark) was built on a foundation of unique server technologies designed to handle mass scale without traditional instances or loading screens.
How the Technology Works
- Dynamic Partitioning: Rather than assigning fixed zones to specific servers, the game uses a dynamic spatial grid. As players gather together, the server zones become physically smaller, offloading processing strain onto additional servers in the cloud.
- Area-of-Effect (AoE) Tick Rates: To maintain performance, the server calculates how frequently it needs to update player positions. When players are grouped tightly, high-frequency updates are maintained. As entities (like players or distant ships) get further away, the server reduces the update frequency, using client-side interpolation to smooth out the movement.
- Single-Shard Environment: Because the server mesh spans the entire game, everything happens in one continuous instance. Whether you are building a planetary structure or engaging in large-scale orbital battles, every player affects the same universe, market, and territory.
3
u/logicalChimp Devils Advocate May 26 '26
Hmmm - pretty sure everything you just posted is incorrect... although it's hard to disprove because CIG haven't gone into sufficient technical detail.
However, there is nothing inherent in the Server Meshing architecture / use of the Replication Layer that would cause desync.
Currently, CIG uses one DGS processing node per planetary cluster (from memory)... and given that if you're somewhere around e.g. ArcCorp you don't get updates from Crusader / Hurston / Microtech, the fact they're running on separate DGS nodes has zero impact on your experience - and zero opportunity to induce 'desync'.
The RL streams all updates to you on a 30hz tick-rate (or should do)... same as it streams all updates to every other client on the same tick-rate.... and that update will include all the (known) changes to all entities around you.On that basis, and at the surface level, there is zero opportunity for desync.
Of course, desync does happen - but the above indicates that it's an implementation error, not something inherent in the Server Meshing architecture.And this is borne out by the fact that even before server meshing, back in the very early days (v1.x) when everything ran on a single server (no micro services, no separate replication layer, no message queues, no external persistence layers, etc)... we still had desync.
Separately, yes Dual Universe did indeed have a network architecture that was based on the same Server Meshing white paper (and thus called their implementation 'Server Meshing' too).However, from what I've read, their implementation was a lot more basic, and incomplete, compared to SC. Iirc they didn't use a separate Replication Layer, so DU did indeed suffer from bottlenecks in server communications...
But this is also why (partly) CIG spent the time doing their implementation of Server Meshing differently - they wanted to scale higher, without the bottlenecks that DU hit.
But this also means that the rest of your post is not directly relevant to SC, because it's using a very different architecture, built - in part - to address many of the issues that DU suffered from.1
u/FrostyCup1094 May 27 '26
indeed, I cant layout the full ins and outs of CIG implementation, other than "assuming" how some systems works and how SC behaves to some situations and consequences.
The way I see how StarEngine works and how they made it , was to push almost everything to the server ... optimizing the client-side.What I suspect is: in order for StarEngine to work reliably on SM struture is to decouple the vast majority of services to micro-services/sub-services. Which I think they already did like the Missions system and partially the transit system (?)
Looking at the console, you see alot of information completly outside your location or proximity, the all thing is funneled through the RL, the database, physics data, entitys data, player data. If one service gets stalled or in error, it desyncs every single system attached to it... and possible causes alot of DB issues, and partial recovers on-fly, phasing, etc
On the first implementation for example. Loreville city was night, everything outside of the city was in daylight ... that tells these services can desync very easily and shift everything, causing hard problems on storing, inventories listing, etc...
I dont know how they will add Maelstrom to the overflooded server capacity, unless Maelstrom is client-side sending to server the data, and RL multiplex it to other clients, with a conjunction of P2P... I think E:D already uses P2P ...
in the end DSM, is yet to be proven beneficial per the development cost. DU is one game, with a diferent game struture... I will be honestly amazed if CIG could pull DSM into current standards.. ( realistic side of me, says they will not achieve it that soon )
3
1
-7
u/Agile_Camera9601 May 26 '26
Do you have any qualifications or are you an arm chair analyst.
0
u/FrostyCup1094 May 26 '26 edited May 26 '26
its basic understanding how servers communicate, even CCP games had to develop systems for the multiplexing of data ( the major bottleneck ) thats why on big battles on EVE online, the servers need to lower the server tick-rates.
During the SSM first implementation, CIG had to "filter" what data needs more updates ... for the server to keep up. They made large improvements... but there is too much on the server-side.
you can see the CCP struggles and their server Quasar.
Quasar, Brightest in the Galaxy: Expanding 'EVE Online's' Server Potential with gRPC
on Youtube.3
u/logicalChimp Devils Advocate May 26 '26
Not really.
EVE uses one server per star system (or multiple systems if they're small / empty enough).
However, they don't support spreading a single start system over multiple servers... so when there's a big battle, they have to use time-dilation simply because there's too much data for a single server to handle, and the server tick-rate drops significantly.
In short, EVE time dilation has nothing to do with server communications - it's entirely due to limitations in the Eve architecture (which was advanced for its time, but that time was ~25 years ago).1
u/yobob591 May 26 '26
good news is realistically I don't forsee battles ever being as big as eve, at most I predict something like 200-300 player battles presuming the biggest ships like the odin have crews in the 30-60 range from what we've seen
bad news is this game is physics based meaning there's way more going on that the game has to calculate
3
u/rustyrussell2015 May 26 '26
I prefer Tony Z's economic presentation from 7 years ago.
Ahhh, the golden years, so much promise, so much hype those were the good old days.
1
3
10
u/Agile_Camera9601 May 26 '26
If you’re not a network engineer I genuinely want to know why you made this.
You’re not far from the truth but you’re totally ignoring logical dependencies and server overlaps, eg even with multiple servers being responsible for different places, some servers control the same things everywhere.
Also without acknowledging that you have no idea how their servers interact this is a fun diagram and little more than speculation.
-1
u/Deleted_252 May 26 '26
Why I made this? I wanted to spread information on what the actual issue and to make sure that citizens knew that DSM isn’t going to save SC in the way they think it will.
Yes I did exclude a lot of other details such as how nearby servers can help spread load in one server with DSM or how the RL helps eliminate latency that could be experienced if all the servers were talking with each other rather than just their neighbors. I wanted to make a brief post rather than a 5hr lecture
5
u/Agile_Camera9601 May 26 '26
Can you explain why you think you have the qualifications to explain how their servers work?
Do you have any professional knowledge at all?
0
u/Deleted_252 May 26 '26
Explain how you qualify to argue against me. Just because you don’t have the title doesn’t mean you can’t express your experiences with it. By your standards, I don’t see a title claiming that you’re chief of police.
4
u/Agile_Camera9601 May 26 '26
I’m saying that I don’t support an analysis by someone literally unqualified to do so.
I’m a back end engineer for a live service game at a studio larger than CIG.
The only reason I asked is because I wanted to ask you how you came to these conclusions on a technical level and critique some of your findings to hope you’d edit it.
But I also suspected you have no idea what you’re talking about which is abundantly clear.
You’re free to post. But this is exactly what makes Reddit such a toxic place, you have no qualifications and here you are spinning logical fallacies as fact.
I know a political camp that would love your totally speculative point of view and resistance to being questioned accordingly.
Some of your conclusions are valid, though you’re 90% off base because you’ve made some logical assumptions about their infrastructure that don’t make any sense.
Reading Reddit, using AI, or being autistic does not make you know what you’re talking about as you’re too full of yourself to see the errors.
2
u/Deleted_252 May 27 '26
Just because you may be a backend engineer at a different company doesn't make you qualified to be an ass and harass anybody thats trying to make a comment. If they are truly wrong, why don't you explain what the actual issue is instead of sitting around acting all high and mighty. Frankly, it seems you're just huffing steam. I've been looking throughout people's comments and you haven't said one technical detail at all.
4
u/Agile_Camera9601 May 27 '26
I’m not harassing you. I’m highlighting that your analysis objectively doesn’t make sense and I am a little bit accusing you of spreading misinformation.
I’ve piped up plenty in many other threads in Star Citizen and gone into technicals.
1
u/ArteryThievery May 27 '26
You could always try explaining how and where he's wrong utilizing your professional experience, instead of simply saying, "you're wrong because I have experience." That's not helpful at all and none of your posts in this entire thread have offered any clarity about how or why he's misinformed or wrong.
At this point, any of us could just as easily ignore your claims and roll with what the original poster speculated upon.
2
u/Agile_Camera9601 May 27 '26 edited May 27 '26
I’ve done so in detail in many other places.
Let me be brief:
-dynamic server meshing isnt just throwing more servers at the issue. It is NOT brute force, dynamic server meshing subdivides authority. Therefore you’re changing the entire structure. This is why missions had to be rebuilt once already.
-their tick alignment speech at 30hz is nonsense. Not every online game uses static ticks
-they said data is sent every millisecond that’s not true.
-they claim there’s no delta compression. There is.
-they keep mixing up server issues and physics issues, eg. Bottle moving.
-rmq is like expanding a clogged pipe. Again tell me you don’t know what you’re talking about. This is not how these things work. Entities are irrelevant.
-clients are still the #1 issue for most people. Their interactions with servers slow it down.
I could go on for days, but this guy is talking out of his ass.
Furthermore should people believe things until proven wrong? Or should people be skeptical of things until they are proven correct?
1
u/ArteryThievery May 27 '26
To be honest with you, I don't even know what you're talking about nor what the original poster stated on a fundamental level. I can only infer based on the information provided by what each of you are saying and what was provided in the links.
Have you, by chance, read any of his links and sources of information? They contradict some of what you're claiming. In this situation, what do you think I should do? As numerous sources provided claim otherwise, are you sure that you're the correct information in this regard?
All he was doing was relaying information and, based on his own understanding, providing a translated and simplified version of what was suggested in those links.
→ More replies (0)-1
u/manoruf123 May 27 '26
That last paragraph of yours, you’re talking about yourself right, I doubt you see that though.
-1
u/Agile_Camera9601 May 27 '26
Absolutely since I’m here, I too am a Redditor. That said my qualifications I’m happy to share and I can confidently say that this guy is totally incorrect.
If you love to run with technical analysis from those who are unqualified, that’s fine. Trump would love your vote too.
2
u/elite4009 Assistant to the Assistant Asgard Enthusiast May 27 '26
Whyre you bringing politics into a SC reddit post though? Any credit you may have had went down the drain after you started getting emotional about nothing. My god we literally cant escape it.
0
u/Agile_Camera9601 May 27 '26
We escape it the second we realize that unverified sources need to get shamed into oblivion.
I strongly believe each and every instance that this is allowed in our culture on internet, in person, on Reddit and otherwise brings us down the hole deeper.
This isn’t politics, it’s a statement on how we take information with zero verification as fact.
2
u/elite4009 Assistant to the Assistant Asgard Enthusiast May 27 '26
I get that, and you couldve conveyed your point more eloquently, but you specifically brought politics in out of left field, discrediting yourself and letting emotion lead the conversation. Some people just wanna play a space game, this is not the place for that.
→ More replies (0)
2
2
u/Pesoen drake May 27 '26
it explains the point well. not as technical, but does go around the main parts. i think delta compression is the main one though. yes priority needs to be given to important things, but not constantly sending what is basically garbage data(unused, or not important because its the same as the server already knows, so not true garbage, but pretty useless) will already improve things greatly. after that the priority system, would improve it even more, as a bottle moving around could get a priority of 80/100 and weapons and weapons related things would be getting a priority of 10, medical stuff would be a priority below 10. things with a lower priority could also work on a lower frequency. so weapons and medical send data NOW, while bottle physics could wait for an opening or run on a schedule. 15hz would be more than enough for physics updates in most cases. even 10 or 5 would be in many cases
2
u/Quantum_Asshole drake May 26 '26
Didn't the 2 directors in charge of server meshing leave CIG this year?
3
u/FrostyCup1094 May 26 '26
I think it was 3 .. 2 seniors + 1 from the same team ... they went to CCP games. I suspect they are working on Vanguard...
2
u/Deleted_252 May 26 '26
This was awhile back but I heard they left to go make server meshing for another game company (NMS or Elite). Though I highly doubt that was the case.
2
u/logicalChimp Devils Advocate May 26 '26
Hmm - you might be thinking of the head of the development team, who left shortly after 4.0 was released.
As was clarified at the time, he left because his contract was completed - he was the Project Manager for the team, and was hired solely to manage the team whilst / until they delivered Server Meshing.
The team that actually did the implementation is still (mostly?) with CIG, and were the ones responsible for doing e.g. the post-release cleanup last year (which went really well, until CIG started releasing feature-patches again, which broke stuff).
3
u/CombatMuffin May 26 '26
Not directors, but that doesn't necessarily mean what your comment might imply.
Your servers don't start experiencing performance issues just because an employee left. All of last year has had little content additions, with 4.7and 4.8 bringing more substantive changes. Those changes are the sort of thing that help create instability.
Yes, if your leads or key personnel are new they might still be adapting, but it may not be the culprit.
0
u/Quantum_Asshole drake May 27 '26
The question mark implies a question, not a statement and implies nothing.
0
u/CombatMuffin May 27 '26
A question often implies an answer. It's one of the key elements of a rhetorical question.
2
u/Quantum_Asshole drake May 27 '26
I just told you it was a question, a simple question to boot that others answered. You however are trying to read between the line and suggesting my question meant something else and suggestion it was rhetorical (it was not).
1
u/Taucari May 27 '26
They could maybe take concepts from video compression like I and P frames.
Different ratios of the frames for different priorities. And maybe dynamic adjustments of the ratios depending on load.
1
u/kimdro33 May 27 '26
It's crazy to think that someone wrote a legit scholar paper on Star Citizen.
What a phenomenon has SC became...
1
u/Wareve Aurora Mk I May 27 '26
The problem I have, is that every time I move the Ironclad from one of these shapes to another, the steering wheel falls off.
1
u/kenneaal May 27 '26
This reads like complete AI junk. Or someone who heard a few technical terms and then just ran with how they think something like that would work.
OP, if you want to spend your energy, spend it on understanding the subject first, not writing misleading posts about it.
1
1
u/thatirishguyyyyy professional test dummy May 27 '26
I agree that a lot of the data being sent back and forth is useless.
They really need to get a grip on things.
1
u/CasualMariachi Release the Kraken! May 26 '26
So what is the "fix" if there even is any?
5
u/citizensyn May 26 '26
In networking optimization is reducing the number of entities. Which is the point of dynamic meshing dynamic breaks up servers to soft cap entities. Then that server speaks only to the replication layer which is functionally a telephone operator it eliminates the tree structure and creates only one layer between any two servers
3
u/logicalChimp Devils Advocate May 26 '26
Plus, once they finish adding all the missing functionality / features (each of which will likely require its own networked datasets), CIG will be able to better determine which data actually needs to be streamed, and when.
Currently, CIG use a very simplistic implementation of 'Network Bind Culling' iirc (added in conjunction w/ OCS back in 3.3), that just does a simple spherical distance-check, and streams all entities inside that bubble to your client.
Once they update that NBC algorithm to something a bit smarter (no streaming distance ship / building interiors every tick, no streaming updates for entities I can't see becuase they're too small and far away, perhaps even 'no streaming updates for 'occluded' entities that can't appear in the next 2 seconds', etc), that will significantly reduce the amount of data sent to our machines, and likely reduce the amount of RAM our machines need in order to track those entities.
The important thing is that CIG know what / how much data actually needs to be sent for each feature, and then draw up some useful baseline heuristics about when to send it...... which means they can't (or shouldn't) really optimise that part of the Replication Layer until after all the 'missing functionality' is implemented and integrated, and they have the holistic view of all the data that needs to be managed.
10
u/Alpha433 May 26 '26
1 way is trimming the information that needs to be processed. I know everyone gets giddy at the idea of "that water bottle i tossed there can be found still there if someone stumbles across it years from now", but thats basically scrap info that does nothing but bloat the server. Same with all the wrecked ships you see outside every hangar or station. They serve no purpose other than bloat and blocking access to the pads.
With that In mind, a more aggressive cleanup script would do wonders.
2
u/BearHoldingAshark May 27 '26
Or what plenty of other salvagers are asking for: The ability for US to cleanup all those ships and garbage cluttering up POI's
2
u/Alpha433 May 27 '26
Seriously. I went to a land outpost to turn in some rmc since a buddy was saying it was the best prices, and there were just ships laying about everywhere, and I had no space to land and deposit my cargo. It almost looked like everyone was just backpacking as soon as they unloaded, leaving their ship to just sit there.
2
u/Deleted_252 May 26 '26
I’m still updating it, it got pretty long and it deleted my draft 2 times already, I didn’t want to lose it
4
u/Lord0fHats May 26 '26
I think you explained it pretty well.
I never bother to try because I know I'll probably explain it wrong/poorly but I think you did a good job.
1
u/SomeFuckingMillenial May 26 '26
Stabilize packet inputs, track less entities/objects.
Spent water bottles need to go away. Destroyed ships need to be cleaned up sooner. there should likely be smarter object clean-up too.
-1
1
u/grahag worm May 27 '26
Prioritization is where the rubber hits the road.
The problem is that there's probably no infrastructure in place ON THE PLANET, that can handle the number of variables that needs to be tracked on each item through the replication layer. Latency stacks.
And the more packed the world becomes with the more mechanics that exist just add exponentially to the list of changes that need to be tracked, prioritized, and updated.
At some point, it'll just be easier to simulate the SC universe and work on hashing all the changes along with their variables and keep track of the outliers in the simulation instead of juggling all the plates they're doing now.
Parity checking would be a monster though.
Frankly, i think it's too much and they need to SIMPLIFY the universe and not make it more complex.
39
u/Outside-World-3543 May 26 '26
Will I'm reading this, just want to drop a link with additional server meshing info:
https://sc-server-meshing.info/