MAuto: Automatic Mobile Game Testing Tool Using Image-Matching Based Approach

In the era of mobile computing and internet technology, mobile gaming has seen a huge increase, gaining almost all population groups. According to SmartInsights (Chaffey 2016), there are about 1.7 billion mobile users worldwide who averagely spend 3 h per day on their mobile phones. In another market study (see, Iqbal 2019) showed that 33% of mobile phone users regularly play games on their phones where 50% of them download apps to their phones.

Besides, with the development of smartphones, fast mobile broadband and platform availability, mobile gaming has moved deeper into the broader culture of individuals and communities. According to Intelligence Blog Sonders (2017), 62% of smartphone users download game applications within a week after purchasing their phones, which is higher than any other downloaded applications. This generated more than $60 billion of revenues, which are expected to exceed $100 billion by 2021 according to estimates raised by Newzo (2018). On the other hand, the ever-growing popularity of Android that constantly attract new developers and business, has unfortunately led to a wide discrepancy of distinct Android devices employed. As such, ensuring efficient and reliable testing of newly developed mobile game applications becomes of paramount importance and one of the toughest challenges faced by game developers, service providers as well as regulators. This is often referred to as quality assurance (QA) testing, which focuses on identifying technical problems with the games. Although QA techniques appear almost at every stage of the software development lifecycle, starting from requirement eliciting to product deployment and maintenance where a special interest is attributed to testing automation. The latter is an integral part of a continuous integration pipeline (Novak 2008) where simple automated tests are used for basic program elements, such as individual class methods or separate functions. Especially, test automation reduces the time, cost and resources, while enhancing the reliability through exposure to a large number of test cases that cannot be performed solely by human interaction in practice.

Compared to traditional desktop-applications, test automation for mobile applications bears additional challenges. First, through sandboxing, only limited access to internal processes is provided, which challenges the developers to optimize resource allocation. Second, the general user interface navigation of mobile apps is vulnerable and hard to control due to uncertainty pervading the response time of the interface (s). This includes, for instance, grab and hold like interactions. This problem is also referred to as the fragile test issue pointed out by Meszaros (2007). That is why it is recommended that the application functional logic should not be tested via the application’s user-interface, although such rules are often violated by developers themselves. Third, mobile devices are often in a steady movement, which can cause the currently executed test-case for an app to break down. Fourth, often the complexity of the allocation task together with resource limitations cause a change in size and resolution of the screen, which, in turn, makes any user-interface based test likely to fail. Fifth, despite the effort to harmonize the software-hardware configuration in mobile platforms, the number of distinct configurations is sharply increasing, which makes the application of a single testbed very difficult. For instance, the number of distinct Android devices is exponentially increasing (e.g., more than 24,000 distinct (Android) devices were reported in 2015¹). Therefore, it is nearly impossible to test an application on every distinct device in a real environment and, at the same time, provide the best user experience where the underlined application works flawlessly on all other devices. Sixth, mobile games bear additional inherent features that add extra difficulties. For instance, games involve a lot of graphics and other assets, which substantially increase the loading time. This, in turn, challenges the efficiency of the resource allocation policy. Besides, games have inherent hooks that are intended to make the player play the game again and again (Novak 2008). This makes it difficult to automate the process of accessing the various passes of the game. Finally, games bear a psychological factor referred to as fun factor (Novak 2008). Indeed, even in the case of a bug-free scenario, the game can fail because the players do not feel the fun factor so that their actions are random and not comply with game rules. Because of its inherent subjectivity and vulnerability from one player to another, it is almost impossible to automate the fun factor in testing.

Due to the above challenges and the lack of effective fully automated testing platforms, mobile app testing is still performed mostly manually, costing the developers and the industry significant amounts of effort, time, and money (Novak 2008; Choudhary et al. 2015; Kochhar et al. 2015). This requires attention from both the research community and practitioners. Although, setting up the test automation scheme would imply an additional investment, sometimes referred to as the “hump of pain” learning curve, the expected benefits gained from this process will return back such investment sooner or later (Crispin and Gregory 2011).

In this perspective, we present in this paper a new take on a mobile game application testing called MAuto. The latter aims to help the tester to create tests that work with Android games. The tests can then be re-run on any other Android device. The tool records tests from user-interactions and exports them to Appium framework (Appium 2012) for playback. MAuto belongs to the class of image-based recognition tests where AKAZE features (Alcantarilla et al. 2013) were used to recognize the objects from the screenshots. When the user performs the recording, MAuto generates a test script that reproduces the recorded events. MAuto then uses Appium framework to perform the replay of the test script task. To validate the developed MAutol, tests are created with the tool for Hill Climb Racing mobile game and successfully executed. The rest of this paper is organized as follows. Section 2 reviews the state of the art in the field of mobile testing. The description of the developed MAuto system is reported in Sect. 3 of this paper, while experimentation and exemplification using Hill Climb Racing game are examined in Sect. 4. Section 5 summarizes the key findings and ways forward.

MAuto tool has been tested and validated using Clash of Clans Android mobile game (version 8.551.4) (available from Supercell¹⁴). Clash of Clans (CoC) is a mobile Massive Multiplayer Online Game (MMO/M-MOG) where the player builds a community, trains troops and attacks other players to earn assets. The game has a tutorial that the player should pass to play the game. The tutorial guides the player to click certain elements to continue the game. Therefore, if the tutorial can be passed without serious bugs, it is likely that the game works properly. Besides, since the variations are quite limited in the tutorial, this makes it a good test subject for MAuto. During the recording phase of the MAuto, the browser pops up indicating the readiness to start the recording task as seen in Fig. 8.

The first view which requires user interaction in Clash of Clans is the important notice view, see Fig. 9. This view is accessible using the native object recognition tool. The object’s resource ID is android:id/button3 whose associated package is com.supercell.clashofclans.

Clash of Clans requires the user to select one Google Play account for the game, accessible using the native object recognition as well (see Fig. 10).

The subsequent views are no longer accessible using native object recognition, therefore, the use of image recognition is needed. See an example of this task in Fig. 11. The green circles are the calculated features and the red lines are the matching features in both images. Once we have the matching features, we calculate the average coordinate from the inliers to get the coordinate of the query image in the screenshot.

An example of a file created by MAuto in order to save the click coordinates, screenshots and query images is shown in Fig. 12, while an example of Appium script in MAuto is reported in the “Appendix” of this paper.

Next, the Record and Replay phase is carried out using Appium test script that takes into account the image recognition based approach. An example of Appium test script for this purpose is shown in “Appendix”.

It is very time consuming and prone to errors for human testers to take regular screenshots from the device, transfer them to the host machine and crop appropriate query image. However, such a repetitive task can more efficiently be automated. MAuto is designed to do so and, thereby, decrease the amount of required manual work. It can take the screenshots and transfer the images to the host machine automatically. More specifically, MAuto crops the images properly and creates reusable tests through the appropriate use of Appium. To demonstrate its feasibility and technical soundness, MAuto was used to create automated test scripts for Clash of Clans mobile game. Strictly speaking, although MAuto does not automate everything, still it can significantly improve the speed of test automation script creation. Nevertheless, the selected query images have a huge impact on test stability on other devices. Indeed, the query image must have a good layout for the AKAZE features to be matched appropriately on the screen. Figure 13a highlighted an example of the query image of click where MAuto and AKAZE found only 4 features so that most likely this query image cannot be found from the screen when the test is run (see Fig. 13b).

The current version of MAuto has a predefined box to crop the query image from the click-coordinate and, sometimes, the box size becomes too small to contain a usable number of features. To circumvent this limitation, the user needs to manually crop a better query image from the screenshot to expect enhanced results.

Initially, the user should use native object recognition whenever possible. Indeed, native object recognition is found to be relatively stable, thereby, once available, such an approach should be privileged over MAuto. The latter often takes shortly after the native object recognition step.

When creating the tests, the screenshot operation is quite slow. It can last few seconds to take the screenshot. This means the usability of the application in the browser is not the same as in the device without MAuto. It is also harder to play games through the browser than in the device.

MAuto cannot work with fast-paced mobile games, because it is too slow. It takes relatively too much time to transfer the screenshot from a mobile device to the host. Therefore, it is almost impossible to play fast-paced games with MAuto, because the game can end up in a couple of seconds without new inputs.

Many recording tools do far better when native object recognition can be used in the application. If a native object recognition cannot be used, then MAuto will take over. However, we should notice that it is not possible to give inputs to mobile device sensors through MAuto. This means that it is not possible to test directly games that use sensor data. Although Appium has some support for sensor inputs, MAuto cannot record those inputs.

Tools to create image-based recognition test scripts have not reached maturity yet and still are under development. This paper has introduced a testing tool, MAuto, to make it easier to create automated mobile game tests. The approach is based on a fruitful combination of AKAZE features, Appium, record and replay, and native object recognition. Evaluation and testing have been conducted using the Clash of Clans game. It is stressed that a good choice of query images is required to make the test stable for production use.

MAuto is a very raw approach to solve challenging mobile automatic testing problems. With some polishing, MAuto would work on slow games, but it does not work with fast games that require rapid user interactions.

MAuto allows the user to create Appium tests with image recognition without coding a single line of code. The main target is to help the developers to test mobile games, but the tool can be used for other application types as well.

As a perspective work, it would be a good idea to make the cropped image size dynamic. At the moment it is a static 10 pixel square around the click-coordinate. When cropping the query image we could calculate the number of features in the image and if there are fewer features than 20 for example, then the algorithm should increase the query image size and calculate the features again until the image has a good amount of features. This would decrease the manual work the user has to do to fix the low-quality query images. It should be quite easy to add iOS support to MAuto as well. Appium works al-ready for Android and iOS. The corner problems are to find a way to take a screenshot from the iOS device and to find a quality VNC client for iOS. The image recognition solution MAuto can therefore easily be extended to an iOS environment.

To overcome the slowness of MAuto, especially in the recording phase, one solution could be to compress the image in the mobile device and then send it to the host machine. Another solution consists of tapping into the Android operating system and remove the VNC solution completely. From an input–output perspective, MAuto takes the user inputs from a desktop browser, which is not an ideal way to interact with a mobile device. It would be better for instance to trap the inputs directly from the screen of the mobile device and transfer the clicks and images to the host machine after the test has been recorded. Indeed, if the test recording would be in the mobile device, MAuto might be able to trap the sensor inputs and write those inputs to tests as well.