Dewen Zou is the director of mobile application development of Qunar, responsible for the management of pre-ticket sales and client. I joined Qunar in 2012 and successively worked in the Strategy and ticket Business Department. I am good at client and cross-end technical stacks RN and Flutter, and have in-depth research on device fingerprint. I led the domestic main process of RN migration of airline tickets and the development and promotion of TARS automated test system.
Cui Yu joined Qunar in 2018, mainly responsible for the main process of air ticket, air ticket iOS client, TARS UI automation system, iOS fingerprint reinforcement and other aspects.
1. Project approval background
Business pain points
Internet APP test in “rapid iteration” development mode, each iteration will be accompanied by a variety of tests, including functional integration testing for important and key aspect, and because of version iterations required features in the process of verification and existing regression function, requires the tester to test various features of sexual function, And to ensure that the original function is not affected. In frequent iterative engineering, the function and structure of APP become more and more complex, the influence of new code is more and more extensive, the regression range is more and more, and there are various models and system versions on the APP side, which leads to potential missed test risks and more and more repetitive work. Take qunar air ticket business for example, there are 300+ smoke Case test points and 1000+ monthly regression Case test points. Each monthly regression consumes a lot of energy of testers, and the quality of testing depends on the quality of testers, which may be missed. Every monthly regression requires at least 10 QA tests for one day. It is very inefficient, and there are often online bugs and faults.
In order to solve one of the biggest effort every time the regression test workload, improve work efficiency, reduce the omission of potential risks, reduce duplication of work, the project hope to be able to complete by means of a wysiwyg script recorded, even if the tester can’t program don’t understand the script, but can be by normal users click the drag-and-drop operation, such as The recording of scripts is automatically completed, thus greatly reducing the cost of automatic maintenance for enterprises and projects. According to the multi-terminal characteristics of mobile manufacturers, we need to develop a set of UI automation test solution. Supports automatic and repeated execution of TestCase. In compatibility tests of various models, the back-end response data can be controlled, and the request data, response data, and screen shots of the execution result are verified to assert the correctness of the execution result.
2. Scheme selection
2.1 Test Classification
Automated testing is a testing technique that develops along with software development technology. Software testing includes both white-box and black-box testing, and automated testing first uses unit testing in white-box testing. This testing method is more efficient and easy to implement. Black box automation came later, but is now also very popular. The principle of black box automation testing is to use programs and scripts to simulate user behavior, and to verify key checkpoints of software in an intelligent way. In the prosperous era of desktop software and Web technology, a large number of excellent automated test tools such as QTP and WebDriver have emerged. These tools have been gradually mature after continuous improvement, and can better meet the automation test requirements of traditional applications.
Automated testing of the APP includes the following:
1. Divide by test purpose:
Functional testing: Testing the various functions of products and modules.
Performance test: Test the performance indicators of the system.
Stress test: Test the load capacity of the software or system to detect potential hazards.
Compatibility test: Test the compatibility between products and hardware and software, such as the compatibility of software on different Android models.
Security testing: using different methods to find software security problems, such as information leakage, illegal use, malicious damage, etc.
Other specific tests: such as weak network test, power consumption test, fluency test, etc.
2, if divided according to the software development stage, each stage can do:
Unit testing: White-box testing of individual modules in a program to verify the correctness of the basic building blocks of software.
Integration testing: A composite test is performed on a unit module to verify that the interface between the unit modules is correct.
System test: complete test of the whole system to verify the correctness and compliance of the whole system.
Regression testing: When a software change occurs, verify the functional modules that may be affected by the change.
Acceptance testing: The final phase of testing to ensure software quality before release or release.
3. Other Common test concepts:
Smoke test: Smoke test is a simple test of the most basic functions of the software to determine whether the software can be measured at low cost.
The smoke test comes from the hardware test. When the circuit board is finished, it will be powered up first. If there is no smoke, the next test will be carried out.
Exploratory testing: Exploratory testing relies more on the tester’s personal experience or expertise and more on the tester’s initiative.
Exploratory testing can refer to the importance of game testing, thousands of players in a variety of unexpected circumstances in unexpected ways to make the game, so the game testers should not only master the system test methodology, advanced test instruments, have rich experience in the game and explore test thinking.
Functional integration testing, performance, compatibility, interface interaction, access rights, user experience, and so on, which a wide variety of models and versions of the device. Among them, functional integration testing, which requires verification of functional features and regression of existing functions in the process of each version iteration, requires testers not only to test various features of new functions, but also to ensure that the original functions are not affected.
As time goes on, the function of APP is more and more complete, more and more complex structure, every code change range is more and more wide, the influence of structure and some test conditions were very complex, the return of the corresponding scope will be as more and more, with frequent iteration, which makes the quality assurance to repetitive meaningful labor too much, In the process of implementation will also cause potential leakage risk to some extent. Due to the limited economic, human and other factors, it is very difficult to complete the compatibility test in a certain period of time.
To sum up: First, changes in functional testing may be difficult to implement; second, regression is repeated frequently; third, compatibility will multiply the above two situations. Therefore, in order to solve these problems and improve work efficiency, a set of APP automation solutions should be developed according to the characteristics of APP. Support automatic execution of test cases after packaging, in a variety of machine compatibility test, reduce the repeated work of testers, improve test quality, and can be executed to determine whether the results of the automated test system.
2.2 Development trend of UI automation
The application of smart phone brings a new revolution in software testing technology. Compared with traditional testing, mobile phone testing is separated from the traditional architecture of PC in hardware, and there is a big difference in the way of user interaction. Mobile phone software has unique operation mode in use, such as user’s sliding, touching and clicking, etc. How to realize automatic script execution on mobile phone client becomes the difficulty of this kind of test. In addition, due to the market pattern of IOS and Android platforms, most mobile applications need to support multiple platforms, and their infrastructures are different, which makes mobile automation testing frameworks have higher requirements for compatibility. The intelligence and popularization of mobile devices also make more and more people enjoy the convenient life brought by the innovation and development of science and technology. Especially since this year, the world’s first historic mobile network traffic than PC, it shows the trend of the future is mobile market will dominate, the most widely used on a mobile device is APP, because of its convenience operation, friendly interaction, and earnestly solve the user’s problems, the first is widely used by people. To automate mobile client testing, the first step is to address the robustness of test scripts and platform compatibility.
During the use of APP software, various problems encountered by users should be reduced as much as possible to avoid the inconvenience brought to users by APP bugs and make the original convenience become less “convenient”. So in the development process, it is necessary to put quality as the top priority, comprehensive and in-depth quality testing. And the Internet industry is changing rapidly, to meet the rapid changes in business and technology continue to bring forth the new. Therefore, the development mode of APP adopts the “rapid iteration” mode of software engineering to deal with the impact of rapid change. It is important to ensure comprehensive test coverage while using less time cost.
In recent years, mobile phone client automation tools have developed rapidly, and many automation frameworks have emerged, such as UiAutomator and Robotium based on Android platform, Instrument provided by IOS, etc., but these tools have not effectively solved the platform compatibility problem. Cannot be driven by a single framework.
Different platform tests, and deviations in identifying client-side content and elements, make the test scripts less stable and robust. The current tool with good compatibility and stable framework is Appium. Appium is an open source, multi-platform and multi-language testing framework. Compared with other frameworks, it does not need to recompile the source code when writing and running tests, and does not require too much programming language in the writing and implementation of scripts, and tests are more lightweight and flexible. In addition, Appium uses a C/S architecture to provide a unified external service interface that makes the interaction and control of the client or emulator transparent.
At present, there are some APP testing frameworks on the market, some cross-terminal ability is not strong, support WWW web page testing, some only support Android, IOS mobile terminal, while supporting Android, IOS, WWW, small programs are few and far between, some solve the problem of functional testing, However, due to rapid changes, the cost of maintenance is far greater than the cost of use, which does not solve the pain point well.
The TARS system can be cross-terminal and cross-platform by self-developed solution, using artificial intelligence image recognition technology, OCR text recognition technology and general UI element search engine. And based on C/S framework, by sending HTTP requests to control mobile devices, can achieve multi-language support. Combined with Jenkins and other continuous integration tools, there is no problem in building a set of self-developed UI automated test system.
The name of TARS comes from the famous science fiction movie Interstellar. TARS is a powerful intelligent robot in the movie, which assists the protagonist to complete many difficult tasks.
3. Frame building
3.1 Function Sorting
3.2 Frame Structure
Using TARS+Jenkins+STF as the overall control and scheduling system, the overall system is divided into application layer, control layer and equipment layer. Through the device cluster control system, select device clusters and run automated test cases in batches. Case entry system can determine the operation steps of Case according to the positioning and identification of pictures and UI elements. Almost all systems use Python as the programming language with less code and more flexibility over data.
The first is preparation: the control system selects the device to be tested through the device management platform, installs the client, and configures the test environment parameters, such as AB and MocK. The control system can control various data fields obtained by the APP side by modifying the response data, saving the response data to a testCase (for the next MocK), directly returning the server response, or obtaining the MocK data corresponding to the TestCase.
Then start testCase: call up the client on the specified device and execute the specified TestCase.
Finally, there is the Diff assertion phase, which compares the differences between returned results, request parameters, and style images.
- Control system, the correct control schedule each step to perform the action. The request can be correctly forwarded and the response intercepted.
- The Mock system can modify, save and return the back-end response data to the front end according to the correct configuration, or Mock data back to the front end and correctly save the various data of the request and response to complete the HTTP forwarding proxy.
- The equipment platform can judge the availability of the existing equipment mounted to the platform and provide the equipment for common test.
- Case entry allows users to manually input cases by screenshot, UI element recognition, OCR text recognition and other methods, which simplifies Case entry methods.
Image and data DIff system, which accurately diff test data and screenshots, excluding possible influences (such as data using Mock interfaces).
3.3 Element search scheme
Automated Case writing requires that the test framework layer provide functional support to operate on interface elements, most importantly finding and manipulating elements.
To write a Case or record a Case, one of the basic capabilities we need is how to obtain the page elements accurately and stably. It determines the cost of recording the Case and the cost of maintaining the Case.
If the search is not accurate enough, it will directly lead to unstable operation of our Case and affect the accuracy of the execution result.
If the structure of the page changes and the way we find elements is not stable enough, then the Case needs to be maintained again for each functional iteration. In that Case, the cost of maintaining the Case is far greater than the test cost saved by using automated tests, and automated tests are meaningless.
A good way to find elements is something that needs to be explored and improved in our UI automation testing practices.
At present, there are three main categories of popular element search schemes in the market:
- Look it up by XPath
The UI elements that we get are basically organized in XML. XPath is a language for finding information in XML documents. XPath can be used to find elements and attributes in XML documents. XPath can accurately locate an element, but it depends on the context of the XML document structure, can be affected by layout changes, and does not perform well on iOS.
- Find by identity
You can locate an element directly by adding an identifier (accessibility, testID, etc.) to the element, or you can find the element by matching the text content. This way, even if the structure of our page changes, we don’t need to change the identity of the element we need to operate on, so we don’t need to change the code. However, we need to add identifiers to specific elements, and there is a certain amount of work and maintenance costs.
- Search through image recognition
In contrast to the white-box method of finding elements, image recognition, which does not care about the structure of the elements on the page, seems to be the future of automated testing. By simply taking a screenshot of the element you want to manipulate, you can find the element based on the image and operate on it.
But at present, this way is still very immature, the accuracy of image search is poor, and different models, it is likely that the picture can not be universal, need to re-capture for the model.
| Element lookup | advantages | disadvantages |
|---|---|---|
| XPath | Flexible and convenient positioning method | Depending on the context, interface layout changes can affect positioning |
| logo | Accurate and stable positioning, not affected by layout changes | Repeating elements such as list items are difficult to distinguish |
| Image identification | More in line with people’s operation habits | The positioning is not accurate, and various resolutions need to be adapted |
In the actual use scenario, we use the above three elements positioning methods in combination to meet various situations. They are encapsulated through POCO libraries.
POCO
POCO is a cross-platform open source framework for UI automation. It encapsulates a variety of automation tools, such as WDA, UIAutomation, etc.
POCO uniformly encapsulates the function of other UI automation tools to find elements, providing a convenient, flexible and platform-consistent ELEMENT operation API for Case writing.
POCO can conveniently achieve the following functions:
- Element attribute or ID location
- Element relative position lookup
- Element regular matching
- Element interactive control
The POCO method of retrieving elements is implemented using the factory pattern, so it is also easy to extend custom features such as image recognition or OCR text recognition.
Since POCO is based on Python, it is easy to deploy and debug scripts.
3.4 Case Writing design patterns — Page Object Model
The functional support provided by the framework layer alone is far from enough to reduce the cost of Case maintenance to an acceptable range.
Because when we actually write a Case, we need to write a lot of code for the business logic. There are two common problems:
- Writing a Case involves more interface operations, and writing only one Case is time-consuming.
- Business logic or elements change, and all related cases have to be changed again. If there are more and more cases, it can be a disaster to maintain.
So we need a design that reuses the code as much as possible and abstracts the concrete business logic without having to write a lot of tedious element lookup code.
Page Object Model (POM) This design pattern is designed to create a page object (class) for each page to be tested, encapsulating the tedious positioning operations into the page object and providing only the necessary interface for external operations.
POM separates page positioning from business operations, and separates test objects from test scripts. If the UI changes the page, the test scripts do not need to be changed, only some code in the page object can be changed, improving maintainability.
For example, if we need to test the search results of the flight list page, we can directly call the function of the home page object to jump to the ticket page, and then call the function of the select city and date search of the ticket home page object to jump to the flight list page, and then make an assertion on the data we want to test in the flight list page.
How to implement POM design patterns
A Page Object has two characteristics:
- Its elements
- Realize the function
The self element is the actual page element, and the implementation function is the business function provided by the page object, such as user login.
This way, even if the page elements and interaction logic change, it is possible to change only a small amount of code to make the relevant cases work.
The POM design pattern has the following design specifications:
- The public method represents the functionality provided by the Page
- Try not to expose the internal details of the Page
- Don’t be an assertion
- Method can return other Page Objects
- Page Objects do not need to represent the entire Page, but can be any part of it
- The same operation, different results should be separated (correct login, wrong login)
Based on POM design pattern, we have achieved very low Case maintenance cost. Even if there is a major page overhaul, the Case does not need to change.
3.5 Json Mock scheme
During the actual operation of Case, we found that in addition to the problem of the device environment, there was another problem that greatly affected the success rate of Case execution, that is, the environment on the data line required by Case could not be provided stably.
In theory, this problem can be solved by providing a set of written dead data for Mock, but many cases need to verify the birth order. In this Case, the Mock data will fail, and to some extent, we hope to use real data for validation, so that we can find the problems of the online interface at the same time.
Therefore, we need to analyze the problems of data instability on a case-by-case basis.
Question 1: How to obtain real-time online data for testing
Because of the complexity of our business, data is not reusable to some extent. For example, a set of data, can only be used once, the next use will be invalid. In addition, if we want to test round-trip combined Case, we need to know which airline has this type of quotation. Based on this objective situation, our test team developed a set of main process parameter generation interfaces based on various conditional filters of online data.
This interface will filter the online data stored in history and return us the corresponding online quotation according to the data characteristics specified by us. In this way, we can accurately get specific data to test.
Question 2: How to process the online data in a specific way
For example, we often do some activities online, and some pop-ups on the page may not be expected, blocking our operation. If our Case does corresponding processing for these cases, it will greatly increase the complexity of Case, and judgment of various cases will also reduce the execution speed of Case.
In addition, we may need different types of passengers for the test ticket, so we must ensure that such types of passengers are included in the account.
In this Case, we need to make real-time modifications to the online data to ensure the data environment required for Case execution.
We do this with a JSON mock tool. When we start running the Case, the Case sends the processing configuration to the client. When the client gets the data, the Mock tool first intercepts the data and processes the data according to the processing configuration sent by the Case, such as removing the active popover node or replacing the passenger data. After processing, the data is returned to the business code. In this way, we can Mock the online data into the data we need to a certain extent to ensure the accuracy of Case execution.
3.6 Continuous Integration Solution
In the actual automation use scenario, we usually need to carry out automated testing in the daily development and testing process. The process generally consists of the following stages:
- Build the development branch’s latest beta package
- Install to automation equipment
- Setting the Running environment
- Executing automated Case
- End Send a result report
In addition, we will perform different tests according to different usage scenarios, such as:
- Smoke test — return to key cases to ensure the basic quality of the test
- Business regression testing — regression of business cases related to requirements, and assist QA in regression of historical cases
- Performance testing – To see if there is any deterioration in performance as a result of this change
- Compatibility testing – Tests stability on different devices
It’s definitely not practical for people to do this, so we need a continuous integration tool to automate this for us.
We used Jenkins as our continuous integration platform, and the whole process can be summarized as follows:
Automated tasks are automatically triggered during development testing:
4. Practice process
The roadmap
Access research and development process: 109 projects (57 in September) and 205 times (90 in September); automatic regression saves about 370 PD (about 180 PD in September); Access to the release process: Q3 temporary gray version, temporary version and monthly version issued 24 times (iOS 6 times and Android 18 times), the implementation of UI automation about 100 times (covering the main package and 8 Android channel packages), release of the return of manpower saving 150+ PD; DailyRun: Perform online inspection twice a day; Bug interception: found 2 crashes and 3 P1 bugs.
