Hello!  It’s Pontus here this time. You've already read about us recently making a trip to USA for GDC and PAX, and about the push for new content and polish we did before that.  As things go, that kind of quick development tends to mean lots of new stuff is added at the last minute, and sometimes maybe not tested in every possible situation.  Or there's a good chance there are some odd bugs and things that only appear after a while, or in very specific conditions.  Things that in the long term would get noticed, added to our bug list, and eventually fixed... but in a 4-week rush, things sometimes go less than perfectly and some features might end up being a bit buggy. In multiple ways, sometimes. Most of my sprint was spent on fixing biggest issues we found in our PAX build, and most of that didn't end being much to talk about.  But one of the bugs ended being a bit more interesting to figure out, and while I'd usually prefer talking about design, I though some insight on this side of game development could still have some value.  So, this is going to be more or less accurate story of our guards not behaving as intended, as I saw the issue through bug reports on our Jira board, and watching people play the game & testing things myself afterwards trying to figure out what exactly was going wrong.  And also a nice example of how the most obvious explanation often isn't quite true!

There's something wrong with our guards

So, the first thing I heard was one of the other team members saying our guards were sinking to the floor, in t-pose (which is basically a clean starting pose used for character animation, standing with your arms at 90 degree angle to your sides). Not what you'd want to see in the game, but my immediate guess was that there's just a missing animation file somewhere, which would be easy enough to open the animation tree, figure out what's missing, drop the file in, and it's fine.
The second problem, which was noticed during PAX, was that sometimes the guards just seemed to stare straight at the player, but not actually react in any way. Of course there's a distance check for how far away they can see you, and also the lights and shadows affect the player visibility, so it's not so easy to say for sure. But it seemed like they should have spotted the player. So we did what you do when demoing a game at events, not being able to open Unity and debugger... we watched people playing it and noticed some things. The guards seemed to work fine, until they went for a coffee break. When they did, they sometimes seemed to continue drinking coffee, either just spending the rest of their working day hanging around the vending machines sipping coffee, or sometimes taking their coffee with them, going to patrol but constantly stopping to sip a bit more from their cups. (Actually, that sounds pretty realistic! :D  Sadly it didn't look great, and wasn't the behavior we wanted from them. And they didn't bother to chase the player any more, which is less than optimal for a stealth game).
So, more info added to to Jira bug reports, and since we had just added the animations and related AI actions for the guards to do the coffee breaks (rather than just standing next to the vending machine for a bit to "take a break", something missing in the animation tree and probably in the AI action seemed a pretty likely explanation).  Now I just had to survive people running into same issues over and over again until the PAX weekend was over and I was back at my desk with Unity again.  Always great to see your work being broken again and again without being able to do anything about it!  :D

Debugging and fixing the bug

The first thing I did was open a scene with few guards and a vending machine in it, and just let the game run with error log open to see if I could observe the same issues, and if there were any useful errors. After half a day, I hadn't seen the "t-pose through floor" issue, but I definitely did notice the guards gave up their work day around the first coffee break.  Sadly no useful errors appeared, and even with our AI tools open everything seemed fine, the guards took a break, and returned to patrol (based on what AI was doing), yet on-screen they stayed drinking coffee instead. So, maybe we were right about animation issues. Time to open the Animator window.
(not the exact Animator setup I was facing, this screenshot is from earlier in sprint so it's missing some gestures we added later) The states seemed OK, we were triggering a "gesture" with a number code to tell which gesture to play. It would then play one of the states, check for when the animation has completed, and then tell our animation controller script it's done. There are some issues with that, which I'll get into a bit later, but at least that should work for the guards to complete the gesture and get back to patrol. So I needed some other explanation. ...Half a day more of staring at the guards walking around the level, with various logs and inspector windows open, trying to see what could be wrong.  And then I noticed it.  The AI takes breaks when it runs out of "motivation."  Motivation in this case is pretty much just a number we track, decreasing it a bit for every boring task they do, and then when it reaches 0, we tell the AI it wants to take a break, which will then restore the motivation.  But for some reason it wasn't being restored. The guards would go for the coffee break, but somehow the coffee wasn't refreshing them at all, so they kept drinking more, and more, and more! It turns out the gesture system was telling the animation system that it was done playing the gesture, but nothing was informing the rest of the AI that it's fine, you've completed your gesture action. So, with a few more lines of code the guards were now returning to patrol after few sips of coffee to refresh themselves. Kind of.  They were indeed returning to patrol, but would still sometimes keep staring at the player in front of them like you were invisible. Or happily drink their coffee as if they were thinking that the odd guy running around the office building in the middle of the night was someone else's problem during their coffee break. And that's where the aforementioned problem with the gesture animation setup comes to play. The characters would start the gesture, then go through the animations, and then tell the AI to complete the action and start doing something else.  But what if the player walk past the guard while he is in middle of the gesture?  We needed to add a way to break them away at any point if the AI tells there's something more important to do:
Slight pain to set up, but basically, every single animation state needs to have two exit transitions, one for when the animation completes normally, and other that can be triggered by the AI and set up to immediately interrupt everything and exit the gestures tree. This seemed to work:  I ran around the guards, interrupting them in different ways and at different times during their breaks, and they now reacted immediately, dropped their coffee cups, and started chasing me.  Bug fixed!

...or more debugging and fixing more bugs.

While doing some final testing to confirm the bug was now gone, I somehow got the guards even more confused.  Now they were correctly reacting to me and starting the chase, but then sometimes just ended up stopping in middle of everything, with their AI not able to figure out what to do.
That was odd, as based on the logs, the guard was happily chasing the player, waiting for the chance to fire his taser (probably with a grin on his face even though our characters in general don't have mouths!).  He lost the sight of the player, so he made an estimate where the player might be, but couldn't find anything.  And then it all goes wrong.  At that point,the guards should go into search mode, where they select a bunch of search points from the level around them, then jog to each of those locations looking for the player, and after doing this for a while decide the player must have disappeared and return to normal patrol.  But sometimes the guards would just completely give up instead - the AI not being able to come up with any sensible plan left the character frozen in place. So I spent another hour running around the level, trying to get the guard to see me, and then trying to escape him in different ways and in different places, looking for some kind of pattern at least.  Let's just say it's not easy - the guards are pretty good at catching you.  Eventually, I noticed that this only happened sometimes (but not always) when I was running up the stairs from our test scene lobby. To be fair, that's just about the only way I could actually escape from the guards in that level, so it took a while before I even started considering that the stairs might have something to do with it.  And of course, verifying that as the reason was even harder, since now it wasn't enough to get the guards to see me and then escape, but I also had to make that happen so I'd escape exactly at right time while running in the stairs, with enough distance between me and the guard.
Indeed, that ended up being the very specific situation which was triggering the AI confusion.  I probably would have never even ran into it if I hadn't spent that much time testing for the animation/AI/coffee break issue in the same level.  After some nosing around in our code, I found an old hack we had put in place earlier to prevent the guards from using stairs (we had some issues with bit too steep incline on some stairs earlier, but the limitation should have been removed already).  I found a line of code that was pretty much saying “if your target is more than 2 meters above or below you, you can't go there.”  And, annoyingly, it was doing it bit too late to really work nicely to start with, so the AI had already set the target and planned for it, and then that line was just saying that the plan can't be done. Not that the AI would have any other plan to work with, so all it could do was throw its hands up in the air (metaphorically, although I'd really like to actually add an animation for that in the game!) and give up.

Done!

So, after few days, and staring through tons of code, and carefully observing the Animator trees, and our AI inspector, and tracking values in the code, what we thought was one bug somewhere messing up the guards and making them not see the player (and maybe play wrong animations) it actually ended being a bunch of separate issues that all just happened to surface at the same time, plus one more which I only found because of all the testing I had to do to figure out the others. As for the "Sinking into floor in T-pose" bug I started with?  I was never able to replicate it.  Not a single time during all this, even though I even had a screenshot as evidence.  Maybe it was some even more odd glitch, or maybe it was fixed as a side-effect of something else I did, I have no idea.  But I haven't heard any complaints about that happening from the rest of the team either, so I decided to just mark that bug as "fixed" in Jira as well. :D Also, after getting annoyed about trying to get spotted and escape a very specific situation again and again, while also trying to monitor logs and other stuff, I ended making good use of our Lua apps system, and made one that turns the player invisible. That took about 10 minutes to do, so at least our apps system is showing some great potential here.  I don't see this one being an actual feature in the game for obvious reasons, but it is great tool for testing the levels and debugging AI!
That's it for now, maybe I'll get to talk about more design-y stuff next time! Pontus If you haven’t already - be sure to Wishlist on Steam - each wishlist makes a big difference to us, and we really appreciate your support!

[ 2018-06-13 11:31:02 CET ] [ Original post ]