Verification & Validation - What Gaming Badly Needs

André Cohen |
2019-05-15

Verification & Validation - What Gaming Badly Needs

Originating from the cartridges era, the QA department in gaming is alive and well today. Even in the current era of developers pushing game updates daily, ensuring that the game runs as intended is as critical as ever. However, the QA department has slowly acquired more and more responsibility, leaving many complex game systems without proper testing. For example, who checks that the analytics recording code generates the right aggregate data in the analytics dashboard? Who monitors basics questions like are the total number of unique players yesterday correct, how do you know what the right number is, and what timezone is the data from? For most game shops, the answer is the QA department.

The testing done in QA by definition happens at the end of a sprint or feature integration. It is assumed that the programmer and specifications were correct, that no new bugs were introduced, and that only the gaming experience needs to be checked. However, in modern games, a lot occurs outside of the player experience, and therefore many bugs go unnoticed until players realize find them and complain. This problem is why in modern games verification and validation should become standard procedure during development, adding a new layer of quality control that addresses the complexity of games.

Verification

In the simplest terms verification is making sure that a game feature was programmed correctly and that the code compiles and follows the basic specifications that were outlined. The specifications could be as simple as “every 5 game rounds show an interstitial ad before the game over screen.” While the QA department should check that this feature has been added to the game, the developer should verify that this feature is working before going home. his process is the current state verification in games.

Not as commonly seen in the game development world is unit testing as part of verification. Unit testing focuses on testing one or more small components of a piece of software to ensure that they work as intended and meet specifications. From the earlier example, a unit test could simulate the player having played 5 rounds of the game and checking that the interstitial ad code is called. Unit testing almost always involves automating the testing of small bits of code and checking that they produce some expected outcome. The beauty of unit testing is that it can be done continuously, ensuring that the game always meets the specifications. Unit testing often gets overlooked in games because it is believed that most of the testing for a game is visual or requires player interaction. But, as freemium games have become more complex, there is plenty of ground for unit testing that doesn’t require a player.

Unfortunately, some things are outside of the capability of unit testing that are still part of verification. Consistency checking, for example, deals with checking how multiple game rules work together. The interplay of multiple game rules is a common source of bugs in RPG games where game features or the appearance of characters in the game depend on complex rule systems (example, the boss only appears if the player has the sword and has HP greater than 50). Consistency checking would ensure that rules and the desired effects work together. The reason consistency checking is not common in gaming is that it needs the rules to be clearly written out, often outside of the code base. Another concept outside of unit testing but still within the broader scope of verification is static analysis. Opposed to most other forms of verification that require the game code to be executed, static analysis deals with walking through the code without actually running it. The process analyzes the code’s structure and checks that it adheres to programming best practices. Recall classic arcade games where it was possible to have the player score get so big that it went back to 0 -- that is a bug that static analysis would spot immediately.

Validation

If verification is focused on whether a game feature was programmed properly and followed specifications, validation is about checking that the outcome of the game feature is as intended. At Gondola, validation is as important as unit testing because recording player behavior can be unit tested, but only validation can ensure that the behavior recorded about a player actually happened exactly as encoded in the data. For this reason, validation often requires bringing subject matter experts into the process. In the case of player behavior, it might require talking to the team that manages player profiles to cross-check that the player profile matches the behavior. For example, the behavior data shows that the player mined for gold 5 times, the player profile should indicate that the player has 5 gold pieces. Or in the earlier interstitial ad example, validation should include making sure the right ad networks were used and that the ad request was captured on the ad network’s side. Again, the programmer alone might not have the access or the foresight to validate the game entirely, and this is a good thing.

Validation should be a group effort, and the best part is that it can be done on a live game. In the interstitial ad example, someone should validate that “after 5 game rounds” is the correct rule in the first place. This kind of validation can only happen if the feature is live in the game. To validate the game feature, you will need the product manager (PM) responsible for the insight that led to the addition of the game feature and the business intelligence person who understands the game analytics (and can pull out the correct data). Together they can validate that the game feature has the outcome that the PM envisioned.

Conclusion

While game testing and QA are still pivotal elements of all game development, the complexity of games has grown over the years to a point where verification and validation are a must. Often verification is pushed to the testing and QA department; this leads to overlooking common bugs until they surface months later. Validation is more common in games due to A/B testing which forces hypotheses to be validated. However, validation should be implemented in more than just the testing process; all game features and rules should be validated -- there is no other way of knowing if the feature is producing the right outcomes.