This article is from the official account of the project: “AirtestProject” Copyright notice: It is allowed to be reproduced, but the original link must be retained. Do not use it for commercial or illegal purposes
I. Introduction to private cloud Solutions
1. Development history
Airtest project (UI automation test framework) was launched in 2014, and after successfully serving dozens of projects within NetEase, it was invited to be jointly released with Google at GDC 2018. So far, the project has been used by tens of thousands of developers and thousands of companies.
But the internal practice process, we found only at the bottom of the test framework is far not enough to meet the needs of enterprises of large-scale automated tests, for example, how to set up stable and efficient equipment cluster, how to scientifically remote management and dispatching equipment, how to manage the test resources and continuous integration, etc., is a set of complete automation solutions need to solve the problem. In view of the above requirements, we have developed mobile phone cabinets, device management system DeviceKeeper, Airlab cloud test platform and other software and hardware, and built a complete set of automated test solutions, which not only serve NetEase, but also serve many external enterprises, and have achieved very good results.
2. Program introduction
The automated test cluster solution launched by NetEase includes complete solutions such as device cluster construction, device management, script writing, device scheduling, multi-machine script execution, report generation and continuous integration. It is a software and hardware “integration” product, which can help us quickly build a set of private mobile cloud testing platform inside the enterprise.
Figure NetEase automatic test solution-private cloud
Ii. Detailed introduction of scheme products
We will introduce our entire private cloud solution in detail from four aspects: cluster construction, cluster management, scripting and platform construction.
1. Cluster construction
Traditional device management is managed manually by device administrators. Testers also need to borrow and return devices frequently offline to complete their own complicated testing work. In addition, with the gradual increase of internal test equipment, equipment management will be very time consuming and manpower, and the utilization rate of equipment will be very low. Just imagine how long it would take one device manager to manually perform routine on-off maintenance on 100 devices.
So how do you manage test equipment efficiently? How do you provide stable equipment resources for testing? Our solution is to build a real computer cluster. In a real machine cluster, device resources can be centrally managed (device resource pooling) so that maintenance staff can manage and maintain these devices in batches online. It also allows testers to quickly and flexibly obtain specified devices or a large number of devices for regression/compatibility testing.
In order to build such a real phone cluster, NetEase independently developed industrial-level mobile phone cluster cabinet:
Figure 8*4 professional cabinet
Figure: Cabinet of NetEase automation test room
The capacity of the cabinet and the height of the cabinet can be customized flexibly. There are cabinets of different specifications, such as 8 x 4, 10 x 8, and 10 x 9. Each layer of equipment drawer can be pulled and support independent work. The well-ventilated cabinet design and built-in industrial-grade network solution ensure equipment stability and network fluency during continuous testing or batch equipment tasks. In addition to the cabinet type cabinets (suitable for use in the computer room), we have also developed portable portable cabinets:
Picture portable portable cabinet
Compared with the cabinet cabinet, the portable portable cabinet has a small carrying capacity of up to 10 units, but it is small and flexible. It can be mentioned and measured anywhere and is not confined to the equipment room. It is very suitable for some test effect demonstration scenes, outdoor environment test scenes, and even directly on the station test is a very good choice.
In addition, in order to solve the cabinet equipment prolonged online charging battery shell or a shorter service life, we are also within the cabinet on a customized upgrade USB HUB, optimization can become intelligent control electric version, through the program can conveniently control equipment of power and data on and off, timing to lightly put the operation of equipment, Greatly prolongs the battery life of the equipment and reduces the maintenance and updating cost of the equipment hardware.
Figure cabinet – Intelligent power control hub
2. Cluster management
After the successful establishment of the device cluster, we need to consider the problem of cluster management:
On the one hand, we need to be very convenient to monitor and maintain our device cluster, to ensure the long-term stability of the device cluster;
On the other hand, we also need to flexibly manage and allocate our device cluster, distinguish between public and project-specific devices, and guarantee the privacy of private devices while ensuring high utilization of devices. In order to solve the above pain points, we developed a device management system DeviceKeeper “matching” with cabinet hardware:
Figure NetEase DeviceFarm cluster solution
DeviceKeeper not only enables device administrators to remotely check the online status of device clusters, but also supports one-click batch maintenance and monitoring of our device clusters:
Figure DeviceKeeper- Check the online status of the cluster
Figure DeviceKeeper- Exception monitoring
It also manages and allocates the entire cabinet or devices above the cabinet to ensure that only specific project teams or users can access and use private devices (which is more in line with internal device requirements and privacy) :
Figure DeviceKeeper- Cabinet Management
It is also worth mentioning that the well-known cloud real machine platform is also integrated in our DeviceKeeper device management system, which is convenient for testers to carry out part of manual testing remotely on the web page:
Figure DeviceKeeper- Cloud server page
3. Scripting
The purpose of automated testing is to find product problems faster and better, and effectively improve product quality. Once we have solved the problem of equipment resources for automated testing, we need to obtain these equipment resources for automated scripting, script running, and test report generation to find product problems.
NetEase’s private cloud solution not only provides us with the underlying automated testing framework, but also provides us with tools such as the editor for rapid production scripts and test reports:
Figure Automated Testing – frameworks and tools
In the selection of automatic testing framework, we need to consider the versatility of the platform and the difficulty of getting started. The versatility of the platform can ensure that the framework is compatible with mainstream Android, iOS, small program, H5 and all kinds of game engines and other platforms. Low threshold to get started, it is convenient for enterprises to quickly access automated testing and quickly implement automated testing solutions.
NetEase’s Airtest and Poco frameworks have been developed based on these pain points. With the special editor AirtestIDE, zero-based testers can produce simple automated test scripts very quickly (AirtestIDE). Airtest.netease.com/) :
Figure out pain points and solutions for automated testing
Figure Airtest- Framework principles and examples
Figure Poco- Framework principles and examples
In addition, the inside of the private cloud solution enterprise IDE on the basis of community edition IDE, developed a more advanced features, such as batch operation (small batch compatibility test/regression test), a key generation polymerization report, hd mode connection devices (such as good compatibility, more fluent), more in line with the enterprise level of automation scripts written demand.
Figure Enterprise IDE- Batch run tasks
Figure Enterprise IDE- Overview of aggregated reporting
In addition, private cloud solutions provide a Klay unified installer for scripting, which is an enhanced tool library. It encapsulates some of the usual features, such as automatic pop-ups, swiping by direction, and sliding to unlock the screen, as well as some enhancements to the Airtest API. The library reduces some of the scripting difficulties by allowing testers to call the API provided for similar scenarios without additional encapsulation.
4. Platform building
While the enterprise IDE supports small batches of compatibility/regression testing (with a dozen or so devices), larger regression and device compatibility tests require an efficient automated test case running platform.
The Airlab cloud test platform provided by NetEase Private cloud solution not only supports the scheduling and running of basic batch test scripts, but also supports the unified management of test resources, automatic test report analysis and continuous integration.
Figure Airlab- Platform function diagram
Using Airlab cloud testing platform, testers only need to prepare test resources and submit task configuration to start the test task and run the task on a large number of equipment.
Figure Airlab- Regression test task configuration
There is no need for manual intervention in the whole test process. After the task is completed, a complete operation report of the test task will be automatically provided, including report overview, performance report, error report, screen recording of test cases, script log, etc., which can help testers find product problems.
Figure Airlab- Regression Test task report overview
For regression test and compatibility test of stable operation function, only need to write the test script once and save it on the platform, it can run automatically repeatedly, which can improve test efficiency and save test labor cost to the greatest extent.
In addition, it is worth mentioning that Airlab automated test platform can support batch equipment automatic package, Patch down and login requirements, helping testers reduce the operation time of package installation, application startup and login process.
The team only need to provide testing inclusions, login cases, login script and test account to test resources, including Airlab automated test platform, and configure a test on the Airlab test automation platform login compatibility test task, select need to pack for equipment, platform automatically login to complete the installation, the patch test application and operation, After the task is completed, inform the project team to test:
Figure Airlab- Packing task
3. Advantages of the scheme
1. Flexibility in setting up device clusters
We support a variety of forms, a variety of sizes of real machine cluster construction, can meet the different cluster construction needs of enterprises: If you need a small cluster and the cluster can be flexibly placed in the office/workstation, you can choose our portable portable cabinet, which can accommodate 10 devices, supporting Android and iOS, but one portable portable cabinet can only deploy devices of one platform.
For medium – and large-sized clusters, you can choose our industrial cabinets. Common specifications include 4 x 8 (32 sets of equipment), 8 x 10 (80 sets of equipment), and 9 x 10 (90 sets of equipment). We also support cabinet specifications customization based on the actual equipment scale of the enterprise. In addition, industrial cabinets also support customized height, which can meet the needs of enterprises to mount special equipment, such as cars and machines, translation pens, special tablets, etc.
Of course, our industrial level cabinet also supports Android and iOS devices, and supports mixed deployment by row. For example, there are 4 rows of 4*8 industrial level cabinets, with 8 devices in each row. We can choose the first row to deploy aS iOS nodes and mount iOS devices. The remaining three lines deploy Android nodes and mount Android devices.
2. Cabinet professionalism
Our industrial cabinet has gone through several generations of hardware trial and error, finally confirmed the existing hardware solution, can guarantee the stability of cluster operation to a large extent; And we are still constantly improving and optimizing our cabinets. For example, in 2021, we have upgraded the ordinary USB Hub to an intelligent control USB Hub, which greatly extends the battery life of the equipment with the intelligent control system.
Another example is the intelligent cabinet door lock scheme we are testing and using, which can increase the security of the cabinet and facilitate the test colleagues to borrow equipment from the door line by scanning the code.
3. Scalability of the underlying testing framework
Our image recognition framework Airtest and control recognition framework Poco are both pure Python framework packages; It can be used to encapsulate quadratic functions and scripts freely, and can organize test sets flexibly.
In addition, our framework also supports the combination of our own testing services (such as game command service), free call instructions within the script, combined with the API of the testing framework to complete more in-depth testing functions.
4. Integrity of tool chain
The NetEase private cloud solution provides detailed solutions from cluster building, cluster management, script writing, automatic task running to report generation, that is, from the hardware layer to the business layer.
Enterprises can flexibly purchase software and hardware products provided by the private cloud solution based on their own requirements, including cabinet customization and software solution selection.
5. Privacy of privatized deployment
Now there are a lot of enterprises on the market can provide public equipment testing services, charging per time/use time, but also to meet the needs of enterprises with certain testing machines.
However, the disadvantage of public equipment service is that it can not guarantee that the equipment models can meet the testing needs of enterprises; Secondly, the privacy of the whole test process cannot be guaranteed. After all, we need to upload the test package body and scripts and other resources to a public platform, so the problem of confidentiality cannot be underestimated.
So now, many companies are becoming more and more in the equipment of the cluster deployment privatisation, netease provide private cloud solution, also is the cabinet and software platform are deployed within the enterprise, effectively guarantee the equipment use and test process of privacy, at the same time, the privatization deployment can guarantee enterprise can flexible large-scale regression/compatibility testing tasks.
Iv. Program procurement and implementation process
Generally, external enterprises need to go through the following process to purchase our private cloud solutions:
Figure 163.com private cloud solution procurement process
Among them, enterprise requirement confirmation is to understand the different private cloud deployment/automation requirements of each enterprise. After clearly understanding the requirements of the enterprise, our operation/business students can give the solution suitable for the enterprise and the quotation of the solution.
When the enterprise confirms that the plan can meet its own needs, it will carry out quotation confirmation and contract signing and other matters. After the contract is signed and the enterprise pays the advance payment, we will arrange the delivery of the hardware (cabinet and cabinet accessories, etc.) to the door, and then we will send engineers to the door to deploy the hardware and software to ensure that the system is successfully deployed inside the enterprise and can be used normally.
At this time, the enterprise can carry out the acceptance of our products, and then make the final payment after the acceptance. At this point, the program is purchased and landed inside the enterprise. Later, we will provide specialized technical support groups for enterprises to facilitate timely response and solution of questions or other problems in the process of using the solution.
V. Contact information
Enterprises interested in NetEase private cloud solution can contact us through the following ways: official website: airlab.163.com/b2b