Scripted Automated Tests Vs. AI Driven Tests: Finding the Right Approach for Game Testing
Over time I saw a few companies sinking lots of money into trying to automate tests in games and getting no results. There are a lot of reasons I believe these companies failed, but the main one is what I want to discuss here: trying to use scripted automated tests to test a game. What works for regular software, many times don't work for games.
The Limitations of Scripted Automated Tests
My definition of a scripted automated test is:
I experienced first hand, a few times, companies trying this approach and failing in getting return over investment (ROI). And here I want to share my thoughts on why I think this approach is not the best one.
These experiences showed me that the cost of writing these tests is too high and the benefit they bring is not enough to cover the costs. Scripted automated tests can be valuable when you automate a deterministic area of the game that is easy to interact with, like:
Access a shop inside the game, buy a power up, apply the power up and verify that player stats improved according to the specification.
Starts the game on different devices and checks that the game is loading and displaying the start screen.
Go through the tutorials.
Create a new user profile, change preferences, save the changes and verify that the changes have been saved.
By automating these deterministic scenarios you increase safety in making changes and can cover important scenarios (like any profit related scenario). I believe they should always be part of a sanity check that confirms that a build is ready for testing. And that is where the value ends, when test scenarios get more complex, these tests do not delivery enough ROI.
The main problem: scripted tests are fragile and they require frequent maintenance. Even if you do continuous integration it will take too long until you identify the need to update the tests since you can't run the tests for every commit, because they are too slow.
I got convinced that for more complex scenarios, scripted automated tests should not be the focus: AI driven tests are easier to create and more robust.
Shifting to AI-Driven Tests for Complex Scenarios
Games already collect a lot of player data with telemetry in order to understand the players. This data is labeled and could easily be used for supervised learning. We can even expand the learning if the game as a replay functionality or multiplayer.
Example:
Imagine a multiplayer game with synchronous sessions. All the clients will need to constantly send their location in the world (coordinates) and update different player stats (points, health, ammo). The server then has to broadcast the status of the entire game to all players. All this data can feed the learning AI and generate test cases. Besides the data generated by players, we can count on data generated during manual tests, data from previous iterations of the same IP, replay data and others. Live games can benefit even more from this approach, since they are already in production and generating the data we require to train the models.
With AI driven tests we don't need the if statements. The AI driven tests are able to take decisions, deal with different game feedback and handle changes in design without requiring constant maintenance. This decreases the costs of development and maintenance and increases coverage, by consequence improving the ROI.
Striking the Right Balance: Scripted Tests and AI-Driven Tests
There is no silver bullet when it comes to testing games. However, based on my experience, this is the strategy that I recommend:
Write scripted automated tests to cover the more deterministic areas of the game, the ones that resemble regular software the most. The tests where you do A, then B should happen (given, when, then for those that like gherkin).
UI, Menus and Navigation
Game Progress and Retention
Player Profiles and File I/O
Identities and Cloud Save
Tutorials
Telemetry and Analytics
Settings and Configuration
In Game Purchases
Focus on AI driven tests when it comes to testing gameplay, multiplayer sessions, performance… They will provide more return over the investment in time and money. You will have more robust and flexible tests.
Use the data acquired from players for supervised learning. Keep the algorithm learning, the more data the better. Use the AI driven tests for repetitive work and let the human testers focus on the more creative work.
The photo is a random whiteboard discussion. Something I miss from the times before hybrid and remote work.
Enjoy life,
Leonardo Ribeiro Oliveira