In the process of rapid iteration and new release of ToB business, development and QA are faced with a large number of manual test scenarios. So how to achieve intelligent regression testing, test cases to achieve accurate record and bidirectional traceability of code logic? How to improve test efficiency more efficiently? This is the key exploration of automated test platform.

​

Article | Zhao Chong

Senior mobile terminal development engineer of NetEase Yunxin

​

background

Complexity of business scenarios

As b-side solution delivery scenarios increase, versions need to be rapidly iterated. The automation of single-terminal platform has been unable to improve efficiency, and artificial regression is more subjective to evaluate the impact of changes. In order to ensure version stability, the challenge of full regression testing will stretch the B-side SDK and scheme release planning. The result of traditional automated test is only a single check. Mobile terminal API and UI automation scenarios are complex, and use case management for compatibility between old and new versions is inefficient. Therefore, it is indispensable for NetEase Yunxin to test the multi-terminal interworking platform based on the full scenario of mobile terminal.

​

​

​

Thinking and Challenge

The platform abstracts the basic capabilities to support full process automation testing in a plug-in capability way.

​

 

Actuator link environment

  • Actuator information for running test cases;
  • Build links to the execution artifacts used to run test cases;
  • The on-line test equipment used to run the test cases is scheduled;

Cloud assertion rules

  • The verification of the execution results is compared according to the whole word end of the rule;
  • Unified definition of comparison rules for execution results, supported by keywords;
  • Persistent assertion parameters, based on multi-end data calculation verification results;

Meta-use case model

  • Recommend identification of basic use cases and AI use cases based on automatically generated interface metadata attributes;
  • According to the fragmentation reuse combination and custom use cases, business scenario model is constructed.
  • The upstream and downstream of multi-terminal and multi-platform use case parameters are based on platform persistent data.
  • Execute set modular configuration resource pack;

Statistical model of results

  • Report on defect location and classification of task execution results;
  • Coverage analysis;
  • Historical data aggregation model analysis;

​

The solution

Combined with background and thinking, the design strategy of automation platform includes: actuator link environment, cloud assertion rules, upstream and downstream of meta use cases, scenario modeling, coverage analysis, defect location and reporting.

Platform architecture Scheme

​

Meta use case extraction process

Code is automatically extracted and mapped to the corresponding base use cases.

By configuring resource packs and rule files, and combining tools to parse products under test, test codes and corresponding interface use cases are automatically generated. Select the methods and custom attributes of the elements to be tested on the platform and combine them into meta-use cases. The meta-use case method attributes are not editable, they are a mapping product of precisely tested code, and the subsequent building of the business model is based on the meta-use case metadata. The client interface cannot hot-update the interface behavior as the server can. The release API must be a fixed interface behavior.

​

Building a business model

Refine the scene business, do baseline precipitation, extract test execution set.

​

Running link flow

In combination with the QA local debugging process, the link of local debugging cases needs to support user-defined device resources for debugging cases.

​

Intelligent use case recommendation

The persistent data logs of real users lock the link execution track of use cases through keyword logs and generate corresponding use cases.

 

​

​

 

practice

Shortening the time between code completion and user feedback minimizes the cost of the work in progress, which is also required to deliver the code to the customer in a faster and safer version to maximize the benefits of the version.

Costs and Benefits of automation (ROI)

The benefits of automation are proportional to the number of iterations. Here’s a simplified formula for a new product:

Benefits of automation = number of iterations * (full manual execution cost - maintenance cost) - cost of first automationCopy the code

The ground project

It is currently implemented in Engineering Line Solutions and instant messaging (IM1 and IM2).

​

Daily hang up test

The upstream Job completes construction tasks, and the downstream Job automates tasks. Do not block Package engineering tasks if upper and lower jobs cannot be triggered, node nodes can be written to the service execution Pipeline to execute them

​

​

future

In the early stage, it only aimed at the exploration and practice of mobile terminal multi-platform in the field of ToB business solution. In the future, it hoped to improve the usability and expansibility of the platform while improving the real efficiency. I hope this article can inspire other teams and continue to explore together in the following time.

The authors introduce

Zhao Chong, senior mobile terminal development engineer of NetEase Yunxin, is keen on cross-platform development technology and engineering efficiency improvement. Currently, he is mainly responsible for the research and development of video conference component SDK

​