Why UI automation testing

Mobile APP is a complex system with strong coupling between different functions, so it is difficult to guarantee the overall function only through unit test. UI testing is an important part of mobile application development, but the execution speed is slow and there is a lot of repeated workload. In order to reduce these workload and improve work efficiency, it is necessary to introduce a sustainable integrated automated test scheme.

Why Appium

Appium (Appium. IO/docs/cn/abo… Is an open source testing tool that can be used to test native and Web hybrid apps on Android /iOS/Windows.

  1. To cope with the rapidly iterating functionality of mobile apps, more and more apps are adopting a hybrid model, in which some functionality is handed over to Web pages embedded in the App. Appium can easily switch and test native applications or web pages embedded in apps, and has good support for Hybrid apps.

  2. Appium uses the testing framework provided by each platform itself, so there is no need to introduce third-party code or repackage the application.

    platform The test framework
    The Android 4.2 + UiAutomator/UiAutomator2 (default)
    The Android 2.3 + Instrumentation(provided by Selendroid)
    IOS 9.3 above XCUITest
    IOS 9.3 below UIAutomation

  3. Appium is open source on GitHub and has a high maintenance frequency and relatively high community activity. With the continuous efforts of the community, Appium has always been compatible with the latest version of mobile operating system and the official provided testing framework, and features have become more and more complete, including basic log collection, screen recording, OpencV-based image recognition, as well as the recent addition of iOS 13/Android 10 support.

  4. Appium supports custom plugins to find elements. There are also third-party plugins on GitHub, such as the AI-based Icon Recognition control sample project (github.com/testdotai/a…). ; You can also customize plug-ins to find page elements using image recognition, OCR, and so on.

Use Cucumber to organize cases

Appium supports a variety of programming languages, including Java and Python. However, it is not easy to use code to maintain case and it costs a lot to learn. Cucumber can be used to organize cases in a way closer to natural language. Cucumber is a tool that supports Behaviour Driven Development (BDD). It can customize syntax rule templates and convert steps described in text into steps executed by code. As Cucumber and Java 8 are compatible with Chinese text encoding, Chinese operation steps can be customized, which is easier to understand than English code. For example, to define a basic click operation, the expected syntax rule is “when [element name] is clicked”, then the following definition can be used:

   // Cucumber uses regular expressions to match the content in quotes as the type parameter@ when ("^ click \ [^ \] *) \" $")
   public void findElementAndClick(String type) throws Throwable {
       // Driver is an abstraction of Appium's treatment of the device under test
       // Type can be passed the string corresponding to the element ID. By. ID means that the element is searched By the resource-id element
       driver.findElement(By.id(type)).click();
   }
Copy the code

When writing case, the Page Object design pattern commonly used in UI automation testing is used, that is, a Page Object is defined for the UI Page to be tested in APP, which contains operable or verifiable elements on the Page and adds common methods.

Take the home page of Pepper as an example, you can create an object named “home page”, which contains “search”, “my”, “broadcast” and other elements corresponding to the search method (for example, the search button, The resource-id corresponding to the element to look up is com.huajiao:id/main_home_top_search. Since searching by entering a user UID on a search page is a common operation, you can define a “search” method for this purpose. All test cases, Page objects, elements and methods are saved and edited using the test background web Page, and the basic keyword completion function is realized.

Once the basic click, swipe, text input operations are defined above, and the appropriate pages and methods are established, a use case can be translated into a case description that looks like nature (# opening behavior comment line):

# "$homepage. Search "means to use the" search "element in the" homepage "Page when clicking on the $homepage. Search # "$search. Search ()" represents the search method invoked on the search page, with the search keyword parameter $search in parentheses. Search (43011080) when the predicate element appears $search. The search resultsCopy the code

Write code for complex custom operations

Using Cucumber to define common operations, such as clicking, sliding and verifying text, can reduce the workload of writing a test case and improve the readability of test cases. However, not all functions can be performed in the way of common operations. In particular, Cucumber only supports sequential execution of instructions step by step and cannot branch or loop instructions, so complicated operation logic requires coding in custom steps to complete the operation. The preparation of the code part encapsulates the reference to the official Android Espresso project, through the way of chain call “find – operation – check” process.

Take the Android client login as an example, click the bottom “home – my” element, if the current state is not logged in, it will pop up login pop-up, at this time the bottom “home – my” element is not visible, indicating that the state is not logged in.

As Cucumber is executed in sequence, it is impossible to exit the login when the “my” element is visible and close the login popup when it is not, so you need to write code to customize the login step:

@ when ("^ Log out $")
    public void logout(a) throws Throwable {
        // Click "home - my"
        onView(By.id("com.huajiao:id/bottom_tab_user")).perform(click());
        try {
            // If the current user is logged in, there will be no pop-up prompt to log in, and the "home - my" element will be visible
            onView(By.id("com.huajiao:id/bottom_tab_user")).check(matches(isDisplayed()));
            // Call the logout method
            logOut();
        }
        // Not logged in, "home - my" element does not exist, throws NoSuchElementException
        catch (NoSuchElementException e) {
            // Press the system back button to close the login popover
            onActions().pressKeyCode(AndroidKey.BACK, "1").perform(); }}Copy the code

Use Appium to find UI elements

  1. Basic lookup

    • By.id: searches by element’s resource-id.
    • MobileBy.AndroidUIAutomator(String code): search through the code text of UIAutomator2.codeIn order to conform to the code text of UIAutomator2 specification, Appium will parse the text and use reflection to call UIAutomator2 for search. As follows:UiSelectorFind text containstextThe elements:String code = "new UiSelector().textContains(\"" + text + "\");" ;

  2. Xpath can be used to find elements and attributes in AN XML document. Appium and Google’s official UIAutomatorViewer tool are all organized in XML. Xpath can accurately locate elements that cannot be located By by. id and BY. className alone:

    • TEXT is a sibling of the “TEXT” element whose resource-id is “id” :

      xpath://*[@text='TEXT')]/.. /android.widget.TextView[@resource-id='ID']
    • Resource-id is “id” and the child of the status element is selected, whose attr attribute is value:xpath://*[@resource-id='ID' and @selected='true']/*[@attr='value']

    Although xpath is more accurate in locating elements, the path of elements may be affected by layout changes, and the performance is poor on iOS. Therefore, it is recommended to use resource-ID to locate elements

  3. The image recognition lookup element Appium supports finding images By By = mobileby. image(base64ImageString) at the Selector level. Multi-element lookups are not currently supported and only the first element found is returned. Getting Appium to support image finding requires a little upfront work:

    1. NPM install -g opencv4nodejs to install the NodeJS OpenCV library

    2. Configure related parameters in Appium (see Blog for more configurations) :

      // Set the image recognition threshold to 0.4 by default. Try to find a balance between missing elements and finding elements that don't match
driver.setSetting(Setting.IMAGE_MATCH_THRESHOLD, 0.5);
// Image recognition takes a long time. You can save time by manipulating pairs without looking for images again
driver.setSetting(Setting.CHECK_IMAGE_ELEMENT_STALENESS, false);
Copy the code

StaleElementReferenceException: Appium elements to find, after attempts to operating elements, if the element is not on the current page DOM resources thrown when StaleElementReferenceException anomalies. When Appium uses UIAutomator2 to search for elements, it will retain the cache of elements, and when it operates on elements, it will directly hand the cached information to UIAutomator2 for clicking, sliding and other operations.

  • During the test, the following steps may occur: Page A is switched to page B. Click the element EL on page B. However, both pages A and B have elements with the same ID as EL. When attempting to operate element EL on page B, Appium directly uses the cache of page A, which will appearStaleElementReferenceException;
  • Since Appium uses HTTP requests to find and manipulate elements, the actual process of finding and manipulating elements is: POST to find elements -> Server cache elements ->POST to operate on cached elements, with time intervals. During the network request, if elements such as pop-up window on the APP side are blocked, it may also result inStaleElementReferenceException.

Overall workflow

  1. The hTest Client obtains the android package server download list and selects the latest APK installation package version. If the latest version is higher than the mobile version, it will cover the installation of the Chinese prickly ash APP on the mobile end, and automatically trigger the execution of BVT test cases (directly triggered from the test platform web page when executing a single case);
  2. The test platform selects the SET of BVT use cases described by Cucumber, searches the Page Page, escapes the elements and methods of the use case steps, and replaces them with element locators available to clients (id:It starts with the resource id,text:The header indicates a text lookup) and is returned to the client via an HTTP request (sent using a socket when executing a single case).
  3. In the process of executing the test cases, it may be in finding elements to meet the mobile phone terminal popup window covering Chinese prickly ash APP elements, etc., therefore, in the process of executing the test cases will detect mobile client, the test steps that may occur in the expected popup window, including the first filling play window, introducing present download window, etc., play again after close the popup window to find elements;
  4. The HTest client initializes the Appium driver, uses the Appium as a proxy to connect to the mobile phone, and performs basic operations in the test case on the mobile phone.
  5. If the test case fails to be executed, the system tries to execute the failed test case again. If the test case fails again, the system collects mobile logs, saves screenshots and screenshots, and saves the failure logs to the test platform. When a single case is executed, the socket is used to send the execution result, and the result is sent back to the test platform for display through the Htest Server. If BVT, the result data is sent back through the interface

Using the test platform webpage to execute a single test case:

Divided by modules, the whole framework is divided into:

  1. Test platform: Webpage, used to save and edit test cases based on Cucumber, manage Page pages, parse elements in use cases, send escaped use cases to clients, and display the actual execution results of clients;

  2. Htest Server: Java middleware, used by netty framework, is responsible for forwarding socket messages, that is, the test platform notifies the client to execute the use case messages, and the client to execute the results back to the test platform. Use:

    • Server side in the htest netty startup com. Htest. Server. The server. The BaseServer

      @Overridepublic void run(a) {
    if (bossGroup == null) {
        bossGroup = new NioEventLoopGroup();
        model.setBossGroup(bossGroup);
    }
    if (workerGroup == null) {
        workerGroup = new NioEventLoopGroup();
        model.setWorkGroup(workerGroup);
    }
    ServerBootstrap b = new ServerBootstrap(); 
    b.group(model.getBossGroup(),model.getWorkGroup())
        .channel(NioServerSocketChannel.class)
        .option(ChannelOption.SO_BACKLOG, 100)
        .option(ChannelOption.SO_KEEPALIVE, true)
        .handler(new LoggingHandler(LogLevel.INFO))
        .childHandler(getChildHandler());
    try {
        future = b.bind(SERVER_IP, getPort()).sync(); 
        LOGGER.debug("Service started successfully IP ={},port={}",SERVER_IP, getPort());
        future.channel().closeFuture().sync();
    } catch (Exception e) {
        LOGGER.error("Exception{}", e);
    } finally {
        Runtime.getRuntime().addShutdownHook(new Thread() {
            @Override public void run(a) { shutdown(); }}); }}Copy the code

    • HttpServer, JarServer, and WebsocketServer all use the same startup mode. The difference is that they listen on different ports and use different handlers to process data

    • HttpServer processor is com. Htest. Server handler. ServerHttpHandler, processing the messages are processed in accordance with the HTTP protocol

      @Override
protected void messageReceived(ChannelHandlerContext ctx, HttpRequest request) {
    try {
        this.request = request; headers = request.headers();
        if (request.method() == HttpMethod.GET) {
            QueryStringDecoder queryDecoder = new 
                QueryStringDecoder(request.uri(), Charset.forName("utf-8")); 
            Map<String, List<String>> uriAttributes = queryDecoder.parameters(); // Only request parameters are printed here (you can customize the processing according to your business needs)
            for (Map.Entry<String, List<String>> attr : uriAttributes.entrySet()){
                for (String attrVal : attr.getValue()) {
                    Logs.HTTP.debug(attr.getKey() + "="+ attrVal); }}}if (request.method() == HttpMethod.POST) {
            fullRequest = (FullHttpRequest) request;
            // Process the body data according to the different Content_Type
            dealWithContentType();
        }
        keepAlive = HttpHeaderUtil.isKeepAlive(request);
        writeResponse(ctx.channel(), HttpResponseStatus.OK, "Commence execution", keepAlive);
    } catch (Exception e) {
        writeResponse(ctx.channel(), HttpResponseStatus.INTERNAL_SERVER_ERROR, "Startup failed".true); }}Copy the code

    • JarServer processor is com. Htest. Server handler. ServerHandler, processing the messages are processed in accordance with the protobuf format

      @Override
protected void handleData(ChannelHandlerContext ctx, MessageModel.Message msg) {
    Connection connection = server.getConnectionManager().get(ctx.channel());
    connection.updateLastReadTime();
    server.getMessageReceiver().onReceive(msg, connection);
}
Copy the code

    • WebsocketServer processor is com. Htest. Server handler. ServerChannelHandler, it is also in accordance with the protobuf format processing the message, They differ from HttpServer in their ChannelInitializer

  3. Htest Client: Java client used to define Cucumber steps, update mobile APK, initialize Appium, and execute test cases. Usage: Run the java-jar htest-client.jar command on the CLI of the PC. The PC must have Appium and NodeJS OpencV environment, and use the YAML configuration file to control the test parameters. The specific working mode is as follows:

    • Function: This jar supports periodic check of the latest APK function. This function is disabled by default. You can configure whether this function is enabled through the YAML file. If the latest APK is found, it will be automatically installed to the mobile phone, and send a request to the Web server (the test platform for managing automated cases), triggering a specified module case set execution.
    • Download policy: The system downloads only the latest APK by default. If the apkVersion value in the local YAML configuration file is higher than the apkVersion value on the server. If it is smaller than the server, it is not downloaded.
    • Installation policy: After downloading, the system will first compare the versionName(resolved by AAPT) of apK in the mobile phone with the versionName size of the downloaded APK. If the downloaded APK is new, the system will install it. Otherwise, the system will not install it. You can also configure parameters to be installed on a specified mobile phone. If only one mobile phone is available, no parameter is required.
    • After the installation is complete, the apkVersion value is automatically updated for future verification.
    • After the installation is complete, the Web server will send an HTTP request to the Web server. After receiving the request, the Web server will trigger one time and send the case set task to the current mobile phone. The specific case set module is configured by the models parameter, and the email recipient is configured through mails.

  4. Appium: NodeJS client/server, used to connect to the mobile phone, through UIAutomator2/XCUITest, perform basic operations such as get elements/click/slide on the mobile phone;

Problems and Improvements

  1. At present, a client only supports the automatic case execution through USB connection to a single mobile phone. In addition, the granularity of case classification is not small enough to execute a complete test process in parallel with multiple mobile phones. The improved method is to use ADB TCPIP to connect multiple mobile phones through wireless network and execute case in parallel according to modules.
  2. The existing error collection mechanism after use case execution failure is not perfect enough. Due to the use of ADB for video recording, the compatibility is not good, and the longest operation video can only be recorded for 3 minutes. The improvement is to use SCRCPy for the execution of failed cases.
  3. Currently, element search methods such as ID and text provided by Appium are used. The success rate of standard controls is high. However, the search efficiency of xpath is low for UI elements that cannot obtain resource-ID, such as custom controls, and sometimes cannot be uniquely located. However, Appium’s image finding accuracy is not very good. In certain situations (such as the interface written by Flutter), it is difficult to locate elements by image recognition alone. The improvement method is the use of custom Appium plug-in, through image recognition, OCR and other ways to find and locate elements.