DISCLAIMER: I will speak about what happened, progress and performance optimization over the last few days and since it's mostly been coding, it will get kind of technical. I could post some images but there really is nothing new on them that I could show you, so I don't see the point of doing it. This will most likely also be more like a blog post or something. We'll see.

Alliances & Progress

Over the last weeks we finished up on the alliances mechanics, meaning: Alliances are basically finished (feature-wise)! They need more testing, but the features are all in. However, there's a lot of other features that we want to ship alongside alliances with the next update, the most important ones being Steam Workshop support for ship and station models and out-of-sector-updates for player ships and stations. We want to ship them all at once because we know that if we ship them one by one they will break their own database structures (ie. corrupting your saves) or we'd have to add a converter each single time (which is actually not an option if we have to do it for lots of small updates) since we're doing a lot of work in that area right now. I also improved how player ships are saved and, most importantly, restored when there are database corruptions. Right now ships are always stored in the sector files, since technically speaking, they belong to a sector. But this lead to annoying ship losses in a few cases when the server crashes before a sector can be written to disk. With the new system, which will be shipped with the update, player ships and stations will be saved in the player's file. In addition to that, the server will now save multiple instances of the player's file, meaning even if a player file gets corrupted due to lack of HDD space, BSOD, a power outage or other reasons, there are 5 other fallback saves which you can access so at max only the progress of the last 5 minutes is lost. This also means that you can now transfer entire players, including their ships (!) to another server by copying that player's file. It also means that sector and player files no longer need to be written out every time a player changes sectors, which should improve performance of multiplayer servers dramatically. This also allowed me to fix an issue where the entire database would get written out in some cases to ensure a consistent state. This has been identified as the major cause for those huge lags that tended to occur on large servers (like the public test server) whenever a player logged in or whenever a new faction was created (ie. when a player changed sectors into a region of a new faction).

Steam Workshop and Out-Of-Sector Performance Improvements

Steam Workshop implementation should be quite straightforward. If its ease of implementation is similar to the other Steam API features that I've used in the past it should be realized quite quickly. Performance optimization for holding lots of sectors in memory at once, however, has proven quite the problem (but a solvable one, phew) over the last few days. When we're talking about keeping lots of sectors alive at once, The most important things are update performance and memory consumption of the server. Now, sectors without players are actually really lightweight when it comes to update performance, since the game is using a simplified update variant for these, however they can still easily use 100 - 150 MB of RAM per sector while in memory. And players tend to visit the populated sectors more often since they're a lot more interesting than empty space. So we have to improve on the memory consumption most importantly. So we profiled what was actually causing the high memory consumption. At first we thought "obvious, it's all the geometry of the objects". After measuring memory consumption before and after stripping down all the plans of all the objects in several highly populated sectors we realized that those only took up 2 - 5 MB of RAM, out of 130MB. So that clearly wasn't the cause for the high RAM consumption. Which was to be honest both scary and a relief. A relief because if it were the block structures, we'd have to strip them every time a player leaves a sector and re-add them every time a player reenters the sector, which would lead to a myriad of special cases where components and scripts would always have to check if a plan was actually loaded or not (have fun, modders, haha). But it was also scary since it had to be something that we hadn't thought of before, and that's never good. After further investigation it turned out to be the lua scripts. In highly populated sectors we saw that there were on average around 220 scripts active (basically nearly every single station or ship interaction is its own script). But it wasn't the scripts themselves, it was the fact that we were using a single lua VM for each unique script. This might sound ludicrous, but at the time it seemed like a good idea since these VMs are actually rather lightweight, and each script could be cleaned up easily when it's being detached or deleted. Well, they were lightweight until we started adding the Avorion Scripting API, which blows each VM's memory consumption from 5 - 20KB (depending on which built-in lua libs we offered) to at least 130KB. 130KB times 220 is 28.6 MB, so we had at least 28.6 MB per sector. And we're not yet counting any initialization or actual scripts, this is just the absolute minimum basis per script. Initialized scripts could take 500KB to 2.5MB. After profiling some more we saw that the scripts used up to 80 MB per sector - quite the amount. The big issue here is that when using a separate VM for each script, there will have to be a lot of redundant data available to each VM, since the VMs themselves don't know that there are other VMs.

The improved Scripting API

So what I've come up with now is simple: Use a single lua VM for each entity, and no longer for each script. We were forced to make a few script API changes in order to do this, but the good news is: We managed to do it in a way that will NOT kill current mods! I'd still strongly advise modders to adjust their scripts to make use of the improved memory consumption, though. The changes are as follows:

• Since every script defines functions that will be called from the main game, and these functions have to have the same names, I've added modules or namespaces (I'll decide on which term we'll use later). Each script can define a namespace/module in which its functions will live that it'll expose to the main program. By separating them into different modules or namespaces they will no longer have the same name in the VM and they won't get overridden when another script is loaded into the VM.
• When loading a script, Avorion will check if it's using a namespace, and if yes and if there are no duplicates it will be injected into the existing lua VM of the object.
• If no namespace is defined or the namespace exists already (basically whenever there's a potential for methods being overridden by the newly loaded script) the script will be run in a new VM. This is basically exactly the same as the old model that we've been using so far.

We've profiled this change and so far it's been looking very promising. Script memory consumption has been more than halved and it looks like we might even be able to reduce it to 20% of its previous amount, which is actually crazy good. We only adjusted the major scripts and we already got the memory consumption down from 130MB to ~40MB for four loaded sectors. We'll be on the lookout for more of these optimizations, but that's what we've got so far. Another great feature of the new system is that you can now easily access other scripts in the same entity - provided that they're loaded in the same VM (which they should be, by default): Just check and access the namespace of that script. With these changes we also paved the way for improved modding support where you won't have to merge files manually or with tools but should be able to just copy them into, let's say, a specific folder and they'll work.

Alright that's it for now!

If you made it here, congratulations and thanks for reading! Hopefully I managed to give you an insight on what we're doing right now.

OMG I DON'T CARE UPDATE WHEN!?

I won't give another date right now because this stuff is really (and I mean REALLY) hard to predict, but I want you all to know that we're working hard on new features and improving the game and we've been making some great progress over the last weeks. As I said in the last announcement: Software engineering is a complex thing and there can always be unforeseen issues (these issues I just talked to you about are basically the proof) so telling an exact date is hard if we want to ship with as little issues as possible. There's a reason why lots of companies just say "when it's done". But to give you at least some feedback: About 50% of the work of the update is done, and hopefully the rest will be more straightforward so let that be an indicator for you. Oh, and there will be more new screenshots of nice new features in the next update, promise :) Have fun and see you all soon!

[ 2017-05-08 17:51:14 CET ] [ Original post ]