Read more about Ashes of Creation ➜ https://ashesofcreation.mgn.tv
Hey! I’m Grolo of Unhinged ( https://unhinged.gg ), a system architect by trade. In this video, we’ll discuss why the borders between servers are an interesting place to be in Ashes of Creation and Star Citizen!
Join us on Discord at http://discord.gg/unhinged
source
Great to see you continuing this series! Thank you! I'm in Devops and this is right up my alley. I'll be checking out Unhinged
This whole series has helped me understand the concept of server meshing so much more, thank you for sharing your expertise!
Thanks for the follow-up! I love the way you explain things.
I almost think an easy way to think about star citizen's meshing is that the replication layer is like the entire server that knows about everything, every single player item and ship etc, but it then just offloads its calculations of physics, movement, items, bullet hits, collisions, etc. to many other servers. then, only the changes get fed back into the main server, which now only needs to handle sending the smaller finished state changes out to some of the other servers on a need-to-know basis.
Ashes servers are complete dog shit rn
well done we want MORE
What is the whiteboard tool you are using?
Another great video! One difference between AoC and SC is the replication layer. SC has a singular replication layer but AoC has replication layers on each server aka inter server replication. Does this mean that AoC will have a higher fault tolerance than Star Citizen? If so, what could CIG do to address this? Would ATLAS help?
ps I was desperately hoping someone like @spacetomato would make a video series about these things, specifically comparing server meshing in many games including a focus on AoC and Star Citizen but you are totally fulfilling that wish so thank you.
Beautifully delivered in this vid and previous. I have a question/topic for next video, the logistics to run these games, how many servers, the quality of servers, with current limitation from fidelity, the total price and ultimately, will SC be able to run on current marketing strategy or will they need to increase their revenue such as subscriptions
Hey ! Thanks for the video ! 2 things I find interesting that you didn’t touched regarding SC : the replicant seems to be a hard bottleneck and I wasn’t sure how scalable it is, but Benoit confirmed they are working with 1 per shard but one of the next step is to have more. Benoit also said that dynamic server meshing is more that just dynamic assignment of game zone, it wasn’t clear but it seems that multiple server may work on the same zone by sharing load and calculations like on server just do physics calculations for a zone and another take care of the rest. 😮
The real question is… what comes before sc or ashes of creation
Because of SC's object container streaming, they're able to make a server boundary for something as small as a box
Some stuff like a water bottle does get cleaned up quicker than a ship for example. the ship might last a week or whatever timeline. I did enjoy the video and helps explain some of this stuff for other people.
The only issue I can see from the long term persistence are players. Because you just know someone somewhere, is going to draw a big old dick on a moon using bottles of Cruz.
I don't get some of the the things that would work for SC: How the Replication Layer is supposed to scale with so many servers/players ? At some point it can't handle replicating to every server and every player all the entities needed… I don't understand the tech behind this.
Even for one player i don't really get how the RL computes what that player will need to get (since it is not just simple distance), i wonder if the entitygraph is stored in a sort of bounding volume hierarchy that is very easy to traverse? I really don't get how a RL could handle a whole region of the world players+NPCs.
Ashes of Creation told they did not go for RL strategy because they couldn't get too many players (thousands) with it, and it is a lot less complex than Star Citizen (especially with persistence, and upcoming Maelstrom)
Whenever i see stuff like this i keep thinking how people are going to cheat and hack this
I'm not sure that persistence in SC is necessarily a good thing. Rubbish and wrecks lying around everywhere is ugly and a navigation hazard. Recently I had trouble leaving a hangar because there were the carcasses of at least 3 ships right at the exit. I think a bit of judicious garbage collection would be better for the players and the servers.
"In 100 years some explorer will find that bottle" Yeah that is realistic when the game is released in the 2090s… 😂
That was a genuine joke, I believe this project can work.
I wonder how much difficulties latencies cause in the SC architecture simply based on the connection latency of clients. I imagine it can be difficult having a smooth representation when clients from different continents come together. When all the nodes in this system have enough redundancy it might work out just fine. But I expect this one of the most difficult parts for the PU to be completed, and one of the most important parts too.
I like the videos man, really interesting
You know what? Your explanations are simple and easy to follow from the perspective of a totally ignorant individual when it comes to technical subject matter such as server meshing. You earned another subscriber sir!
On AoC they mentioned that the replication layer, server, is a hazard. If that goes down. Everything goes down. Can Star Citizen solve that?
29:30 This is an interest point. Glad someone else is also seeing this issue and bringing it up. I think it is the main limitation of these implementations. They scale great, until they dont. When there are many game servers simulating small areas, then the overhead from exchanging game state between all of these is going to be huge. This might be one of the reasons why these implementations have not been taking off as much as they should (besides maybe because it allegedly needs a lot of configuration for each game object to be optimized, that is at least what I heard from those that used SpatialOS).
I can only think of one optimizations to push this limitation. The state replication from other game servers needs to limited to only near the boundaries of the other game server. They cant just keep sending data of everything in the other zone. Therefore, the game objects (e.g. the ships) are already loaded into memory on the other game server, but may only need to be synced up when they are very close to a boundary and about to transition from one server to another. This way, replication may be reduced. SpatialOS had something like this too for its cross-server replication. Of course, the question remains how far this can be pushed.
In general, limiting the data that is transmitted to other game servers (and clients) should be limited. CIG with its ships has an advantage here compared to most other games with open areas. Not all data – such what exactly is going on in the ship – may be of interest for the other game servers. So such data may be filter out beforehand. So only exterior data like the turret's aim direction and if it is shooting may be of interest. Only when the whole ship is about to transition to another game server does all the interior data need to be exchanged.
33:00 Hm, that is interesting. So each server would keep being responsible for its shot bullets, even across server boundaries. And when it hits a "ghost" entity in another game server, it will trigger a hit event. Which I assume will then be executed by the other game server (the one with the ship that was hit). Since only that server is allowed to determine the state for that ship. This would be efficient because bullets dont need to be spawned/replicated to other game servers, correct? Only downside I see to this is that they would have to always replicate far-away game objects from other game servers. At least the distance of how far projectiles can travel. So that those ships can be checked against the projectiles by game server from which the projectiles were initially fired from.
I know that CIG implemented their own Projectile Manager with fancy SIMD and SoA memory layout to be able to update and collision check many thousands of projectiles (which are indeed handled differently to normal entities like missiles). But it also seems as if there is some kind of system in place, that determines at the time of the projectile being shot if it is going to hit or not. Since it knows both the trajectory of the bullet and ship ahead of time. This may be one of the reasons MM with its slower ship speeds has been introduced as well. So shots are less likely to miss when they actually arrive where the ship was predicted to be.
Thinking about it right now, I do think this how it is going to work. So far, I myself thought that it would work differently. I expect that when a projectile is fired, a spawn event is replicated to the other game servers. So each server would spawn the projectile on its side and check it against only its authorized game objects. If it hits a ship, then a hit event is send out to the other game servers which will delete the projectile on their end so no more hits can occur. But to me, this seems to create a lot of load both in terms of networking usage and laod on the game servers. The question is if this is more or less load compared to having to replicate all the ships and other entities from other game servers 🤔Since there are going to be many projectiles spawned every second, your approach seems to be more efficient.
Great videos! Keep them coming 😃👍
Would love to hear your thoughts on my Unofficial Road to Dynamic Server Meshing wiki, but it seems we already align on the large majority of how these systems work.