Author: Jiang Wei chief architect of netease Yunxin Client

Introduction to the

In 2020, the mobile Internet DevOps field shows a trend of large-scale operation. Most of the big App r&d companies are equipped with r&d teams of hundreds of people. The demand for compilation and calculation is increasing day by day, and the behavior of saving manpower with automated testing is also emerging. Mobile DevOps faces challenges in mass compilation, automated testing, and delivery, but in the context of the COVID-19 pandemic, research and development budgets tend to be conservative. How to solve the contradiction between high computing demand and low budget in this era has become an urgent challenge.

This article introduces Proxmox’s practical application in netease Yunxin, a subsidiary of netease Intelligence, and provides best practices for reducing costs and improving efficiency. We’ll start with the challenges of working with Mobile DevOps, then introduce Proxmox in practice, and finally give some concrete results in conjunction with CI/CD continuous integration and delivery for Mobile application development.

What is Mobile DevOps?

DevOps should be familiar to many backend developers today. It is literally a combination of development and operations. In fact, it is a great software delivery concept that aims to accelerate software development and deliver a complete workflow of continuous improvement and innovation. As a result, this concept is often associated with the Agile work culture, and it can be said that Agile is inseparable from DevOps, which is the practice of Agile concepts.

The world of mobile application development is significantly different from server-side development, but the philosophy, goals and goals of DevOps are the same and universal. Mobile DevOps is a set of practices designed to accelerate the development and delivery process of Mobile applications and respond to the changing needs of the market as quickly as possible. In 2020, it’s safe to say that Mobile DevOps is the must-have option if you want to stand out in the crowded marketplace.

The current state and challenges of Mobile DevOps

There are some challenges in the implementation of Mobile DevOps. Due to the differences between Mobile and server scenarios, the implementation of Mobile DevOps is different from that of server DevOps, mainly due to the following reasons:

Mobile builds are naturally fragmented

All compilation of iOS App relies on MacOS system computers: Mac Mini/Mac Pro with different shapes and sizes. Usually, after buying more than 10 PCS, they need to build their own Mac room, which is a huge challenge to maintainability and stability.

Android App relies on the tool chain with strong generality, which can use cheap PC (Win/Linux) to meet the requirements, relying on more different versions of Android SDK, NDK, Gradle support.

Mac App compilation depends on different operating system versions, and some teams have specific requirements for Xcode versions.

Windows operating system version, SDK various operating system version matching Visual Studio version is also very complex.

The faster the mobile app development iteration cycle, the better

Most of the Top 10 teams are capable of releasing official releases in 2 weeks and hotfixes for serious issues in 1 day. This requires Mobile DevOps to be able to fully complete the automated test within one day. Tests of different development teams and features corresponding to branch are conducted concurrently every day. Therefore, DevOps must address the need for high computation.

Maintaining CI equipment is becoming more and more expensive

A team of 10 people needs at least one automated compilation and testing machine, and the number grows exponentially as teams reach hundreds or thousands of people. Compilation machine here refers to Mac Mini, Mac Pro, PC, Linux server, automatic test mobile phone Android, iPhone, etc.

Take several units that THE author has worked for as an example, the annual purchase of automated test and compilation machines costs millions, including equipment depreciation and renewal, and this cost has actually become a fixed expense every year.

Not only is the cost of purchasing equipment high, but the more machines you build, the more maintenance costs you have. Every operation on Mobile DevOps needs to be copied and pasted to dozens of different machines, and with every machine upgrade and device update, Mobile DevOps people are as busy as ants in a hot nest.

Why consider enterprise private cloud solutions?

Mobile build tasks “going cloud” may sound surreal, but once you get rid of mobile build fragmentation, you can enjoy the same benefits as the back-end development DevOps cloud. Let’s talk about a few thought processes, challenges, and solutions for considering private clouds.

Why consider DevOps solutions for private clouds?

The main advantages of a private cloud DevOps solution are:

  • Security: Private cloud means absolute code Security, all compilation takes place in the physical room, no external access Security vulnerabilities;
  • High availability: Private clouds and Git repositories have unmatched advantages over the same network. Taking netease as an example, the speed of Intranet checkout is as high as 150M/s, and a WebRTC code is as large as 20GB. It is almost a nightmare to use the public cloud solution, and many projects have natural complexity, which may refer to multiple Intranet submodules or external library dependencies. The cost of migration and transformation is high and the income is low.
  • Scalability: Like many public cloud solutions, private clouds can be clustered, and a new build machine can be easily created using copy and paste.

The challenges of private clouds

It’s not hard to find software that supports virtualization. It’s hard to find products that are all-platform virtualization, scalable, stable, and free.

We tried various combinations of options and listed the software vendors on the market, as shown below:

Support platform stable Easy to scale cost
Hyper-V Windows is easy pay
vSphere Hypervisor Windows,Linux,MacOS is easy pay
Oracle VM Windows,Linux,MacOS is difficult free
Proxmox VE Windows,Linux,MacOS is easy free

Naturally, Proxmox VE came to our attention with high stability, scalability, and support for multiple platforms. After a screening survey, we decided to use Proxmox VE to implement netease yunxin’s private cloud solution and support Mobile DevOps. Let’s make a brief introduction to it first.

Introduction of Proxmox VE

Proxmox VE is a powerful enterprise virtualization open source software that manages containers, defines machine cpus, disk capacity, network configuration, and manages them in a cluster using a simple Web interface. Proxmox VE virtualizes Windows, Linux, and a variety of other operating systems. Here we are based on Proxmox V6.2.

Here is the first post above, intuitive experience of its management interface.

Mobile DevOps works differently

The installation process will be detailed at the end of this article. Let’s take a look at what happened to Mobile DevOps for private cloud solutions after using Proxmox.

Pried open the beast of sex

The disk read/write performance of a common Lenovo PC is shown on the left, and the file read/write performance of a Proxmox Windows RAID disk is shown on the right. File read/write performance is improved by 8-10 times after Proxmox is used.

Due to the high efficiency of memory sharing, the normal 16GB tight machine, directly share up to 256GB of memory, completely free of pressure. The CPU can also be set to a very high configuration, allowing Proxmox to be dynamically allocated to maximum performance.

The following is a comparison of the Jenkins Job before and after hardware. The build time goes directly from 15 minutes to just 3 minutes!

Maintenance is easy to manage

Remember how Mobile DevOps manages configuration and maintenance on a daily basis across different machines? Not only are mistakes easy to make, scripts are not executed properly, but uncontrollable risks can occur.

With Proxmox VE, such a hassle is no longer necessary. Only need to complete the configuration of a sample machine, copy and paste directly, a exactly the same machine is completed, only need to modify the IP Settings of the node machine, you can complete the machine configuration. If you’re using DHCP, it’s maintenance-free.

Cost savings

The price of two R730 is about 15W, which is equivalent to how many machines the team buys?

  • 2 new 2020 Mac Pros (20W);
  • 20 Lenovo desktop PCS (20W);
  • 20 Linux compilation machines (15W);
  • An additional 2-3 DevOps maintenance staff;

The price advantage is obvious, and buying all kinds of machines is a serious waste of resources in spare time, which is far less efficient than sharing private cloud servers.

Yunxin uses Proxmox VE for detailed installation process

In this article, we introduce the Proxmox process from the beginning of the purchase process to the completion of the installation.

Purchasing server machines

Simply put, Proxmox VE is a Debian operating system + virtualization software package. Software is generally installed on the server, put in the computer room to IT unified hosting, after all, we all know UPS power supply, unified refrigeration, these are not small hard requirements. Therefore, purchasing blade servers to put in the machine room is the first step.

Then the purchase of how much server configuration became the first step careful choice, here can talk about our configuration calculation.

  • A standard 2020 Intel CPU is 8-core, i9, 2.3GHz, 16GB of RAM;
  • A Lenovo Qitian commercial PC is almost 6 core, i5-9500, 3.0GHz, 16GB of memory;

So the basic calculation can be used: number of team members x 0.5 to get the number of cpus required to compile at any given moment, memory requirements, etc.

Netease cloud letter, for example, we passed the measure, the average need iOS/Android/Windows/MacOS each platform three concurrent Job ability at the same time, it is need to calculate the force of 12 MBP, so we chose the two Dell R730 online room was discharged from the machine. (Save the cost of the company, skill points get ~)

The more memory the better, high concurrency compilation computations are very memory intensive, we simply upgrade to 256GB memory.

Disk space is configured on demand. In general, MacOS must be configured with SSDS. Windows devices can use Raid disk matrices and can be very fast, as illustrated later. In our case, each machine was configured with 1TB SSD and 8TB RAID HDD.

Well, without further ado, the procurement process is completed, and the next step is to enter the stage of installing server software.

I’m not going to tell you at this point that the server room was freezing, for reasons I won’t go into, but I was wearing a T-shirt.

The installation process

Step 1 here proxmox.com/en/download… Download ISO, burn to USB disk as boot installation disk, enter the boot screen after the installation process, click Next and fill in the steps.

Generally, the equipment room requires a fixed IP address. You are advised to configure the IP address carefully. In addition, you are advised to use a network cable with a bandwidth of 10GB for the network port.

So far screen, installed, automatic restart, you can leave the shivering machine room.

Set up a virtual server cluster

After all the machines installed, log on first each machine corresponding management interface, such as https://192.168.1.141:8006, only need to find a as the main controller, click on the Create Cluster, copy and paste to join the information, to other machines on paste, can be a key to join the Cluster management. At this point, you can manage all your clusters on any management interface.

Disk allocation

Go to this interface first, familiarize yourself with the disk configuration partition information of the machine, and prepare for the following system installation.

How much disk, HOW much CPU, and how much memory will be needed for each operating system needs to be planned in advance. Our plan for the first phase is as follows:

The VM has been installed

The ISO file used to install the OPERATING system is stored in the **/var/lib/vz/template/ ISO ** directory. You can log in to the Proxmox terminal using SSH to download the OPERATING system directly, or mount remote disks using SFTP or EVEN NFS.

When the system download is complete, you can see the ISO installation disk in the content of the local disk.

Vm installation

Start by giving your virtual machine a name. OS+Version+IDE toolchain +Version is recommended, such as Ubuntu18.04+AndroidR10.

Check the Advanced option and Start at Boot so that your virtual machine will Start automatically if the Proxmox machine restarts.

Drop down to select start ISO file:

The disk capacity is set to HDD or SSD. The default Cache mode is used. Generally speaking, the Write Back mode provides better performance.

It is worth mentioning that SCSI provides the best performance, but not all operating systems support by default, Linux is supported by default, Windows is not supported, you need to download a good driver in the boot disk area in advance, add a second CD ROM, and dynamic search driver to install SCSI hard disk support.

Cpus typically have 2 cores, Sockets mean the number of physical cpus, so the actual Total Core is multiplied by them. If you need super performance, you can set Proxmox to 32 cores or more, and Proxmox will allocate CPU power dynamically, up to your set value if only your task is executing.

Memory configuration, you can choose elastic shared memory, minimum 2GB up to 16GB memory. This saves resources. Generally speaking, Ubuntu only needs 16GB of MEMORY, while Windows needs 16-32GB.

Network adapter, choose the default or RealTek network adapter is available.

For final confirmation, remember to check Start After Created.

You have learned how to install and configure a VM. The OS installation is basically the same as the OS installation in a single vm.

conclusion

This paper briefly introduces the throttling efficiency improvement process of netease Yunxin through Proxmox on Mobile DevOps.

After adopting the Proxmox VE mobile private build cloud solution, netease Yunxin significantly improved compilation speed, continuous integration became more frequent, iteration cycles became shorter, the proportion of automated testing was further increased, and r&d equipment costs were significantly controlled.

Proxmox VE is still unknown in China, with many applications still unproven, such as device backup, disk expansion, and maintenance. Netease Yunxin has been deep in the field of technology, looking forward to more peers to exchange views.

The authors introduce

Jiang Wei, chief architect of netease Yunxin Client, cross-platform r&d expert. After graduation in 2006, he worked as a senior R&D manager in ZTE, Orange and RingCentral. He joined netease in 2020 and is now the technical leader of netease cloud Messaging client group. He is fully responsible for the architectural design and development of NETEASE Cloud messaging client IM and AUDIO and video SDK. He has years of experience in cross-platform research and development, audio and video, and DevOps, and is keen on functional programming and Continuous DevOps integration.

For more technical content, please pay attention to the wechat public account of netease Smart Enterprise Technology +