The privatized deployment and transformation practice of Android component-based system on JD m-PAAS platform

Cloud on the system is an effective way for technology enterprises and traditional enterprises to reduce costs and increase efficiency and cooperate with office. This paper records and shares the process of the privatization deployment and transformation of Aura platform (Android component platform) under THE M-PAAS platform of JD in t-PAAS environment.

Jingdong M-PAAS platform is an enterprise-level mobile RESEARCH and development platform, combining the accumulated experience and best practices of “Jingdong Department” APP research and development, providing one-stop solutions for mobile development. Aura is a componentized and modular solution for Android under M-PAAS platform, providing efficient decoupling, parallel development, independent debugging, rapid construction, flexible integration and other capabilities.

T-paas platform is the basic environment for JD’s private deployment, aiming to simulate various PaaS applications and make it easier to conduct commercial output in various customer environments. Its access specification completely follows the cloud native standard to ensure that PaaS service can easily cooperate with ecological applications and customer businesses. Application logic is realized in the way of cloud native container and operator, and packaged in the way of Helm standard, unified deployment and management on Kubernetes.

Requirement analysis and scheme selection

1/ Requirements analysis

Requirement: Deploy existing internal platform system Aura on t-PAAS platform.

Analysis: T-PaAS platform uses Kubernetes to deploy containers and manage Docker containers. Aura privatization deployment needs to solve the following problems:

Compile component Docker images
Write K8s layout files based on Kubernetes
Deploy on the T-PAAS platform

2/ Scheme selection

1) Existing schemes

The basic technology group defined a set of standard access methods, which simplified the learning of Docker and K8s. Based on the standard scheme, the rapid privatization deployment of the front and back ends can be completed, saving the learning cost of the R & D students in operation and maintenance. Standard access solutions address several important issues:

Automatically generate dockerfiles based on standard templates
Parameter configuration
Compile image automation

2) Scheme selection

Based on the existing scheme, it can only meet the requirements of image generation and related function deployment of the front and back ends. Dockerfile cannot be generated for complex CI/CD processes or containers based on complex basic environment. However, Aura platform’s system architecture is relatively complex, and the existing scheme cannot meet the reform of privatization scheme. The customized transformation based on the existing scheme requires a large investment in both time and labor costs. Based on the above reasons, the private deployment of JD Android component system needs to find a new way to reduce the threshold of transformation and improve the development efficiency.

Concrete transformation plan

The system architecture of Aura platform based on Kubernetes is modified as follows:

1/ Mirror partition

Aura Platform system architecture:

According to the above architecture, it is divided into three images:

Aura2Web: includes front end and back end
Aura2JenkinsMaster: Task scheduler
Aura2JenkinsSlave: CI build node

Because Aura2Web and Aura2JenkinsSlave use a complex basic environment and rely on third-party software, we decided to write a custom DockerFile based on Centos7.2 for image output.

2/DockerFile writing overview

DockerFile is written based on Centos7.2, and the experience of developing Docker is summarized. DockerFile:

Choose the simplest base image
Reduce the number of mirrored layers
Clean up the mirror build intermediates
Pay attention to optimizing network requests
Use the build cache whenever possible

Install the following software based on Centos7.2: JDK, nginx, Python, Maven, Git, Tomcat, JQ etc. Business source code to binary package to mirror: Mirror is to run our own services, so we need to put the platform package into the mirror. This requires a rule that makes it easy to record the process from source code to mirror and can be traced back.

The customization rules are as follows:

Front end:

The front end uses Vue, which needs to be compiled and built, and the product after construction is put into the mirror;
First in the source code Tag, Push to the server, triggered by the WebHook continuous integration, compiled front-end;
The front-end product into a ZIP package, on the jingdong cloud storage, write down the link address for backup.

The backend:

The back end needs to be confused and encrypted, and the encrypted product is similarly classified into a ZIP package, and then put it on the cloud storage of JINGdong, and write down the link address for backup.

3/ Unified configuration transformation

The code in the image uses a large number of configuration files, such as Aura2Web images, which have as many as six. Therefore, it is particularly important to unify the configuration file management. After research, confD software is very suitable for this scenario and can help configure efficient transformation. The specific usage is as follows:

Confd introduction:

Confd is a lightweight configuration management tool. By querying Etcd or other back ends and configuring template engines, you can keep local configurations up to date, and have periodic detection mechanism to automatically reload configuration changes. The data types supported by the backend are Etcd, Consul, Vault, EnvironmentVariables, Redis, ZooKeeper, Dynamodb, StackEngine, and Rancher. However, Confd and Etcd are commonly used together.

The Aura system doesn’t need a backend, just a unified configuration management using its template rendering.

The procedure for using Confdg is as follows:

Step 1: Download Confdg

Download the confD binary file at github.com/kelseyhight…

Step 2: Put confD into the image and add the following statement directly to the dockerfile

1 RUN set -ex 
2 && wget http://$storage_domain/our-tools/confd 
3 && mv ./confd /usr/bin 
4 && chmod a+x /usr/bin/confd
Copy the code

Step 3: Create the ConfD configuration file and template file

As shown in the figure, you can create multiple configurations and templates based on your needs, but they need to be matched one by one.

Example: frontend_domain toml

1
2 [template]
3 src = "frontend_domain.template"
4 dest = "/opt/servers/nginx/conf/domains/frontend_domain"
5 keys = [
6 "/aura/frontend/domain_inner",
7 "/aura/frontend/domain_outer",
8 ]
Copy the code

Frontend_domain.template

1 server 2 { 3 listen 80; 4 server_name {{ getv "/aura/frontend/domain_inner" }} {{ getv "/aura/frontend/domain_outer" }}; 5... 6}Copy the code

Step 4: Copy the configuration file and template file to the image in the Dockerfile

1COPY render /etc/confd
Copy the code

Step 5: Execute on Entery’s shell script to generate the actual configuration file

1 /usr/bin/confd-onetime -backend file –file ${config_file_path
Copy the code

4/ The middleware configuration involved

1) Database

For details, see the documentation for t-Paas. Set the required Host on the local computer and log in to phpmyadmin.tpaas.local (the user name and password are obtained from the documentation). Create a new database and customize the database name, say: auradb. Download the initialization SQ of the Aura you built earlier and import the SQL. Record the following information and add it to configMap.

Url and port number
The database name
User Name Password

2) GitLab

Refer to the website of the middleware information, find the WEBSITE of GitLab, log in to the website, use the user name and password provided on the middleware information or create an account. Here is an example of creating an account: aura and password: XXXXX, record it, and then put it into configMap.

3) Maven Private Server (NexusRepository OSS)

Refer to the url of the middleware information, find the address and username password, login.

Build the following two repositories (the deployment policy parameter was set to allow uploads when created)

libs-releases-local
libs-snapshots-local

If the anonymous access permission is enabled, create a user and record its account and password. Then add the user to configMap.

4) MiniO

Refer to the url of the middleware information to find the address and username password;
Log in to create a bucket and set the access policy to read/write.

5) Double domain name transformation

Since the environment of privatized customers is divided into internal and external environment, domain names accessed by the platform are divided into internal domain names, internal domain names are used for inter-service calls, and external domain names are used for direct access by users. The Key point of double domain name transformation is to classify the services that are invoked only by internal services and those that need to be invoked by users. After analyzing these services, add Key values in configMap and modify Confd configurations to adapt to related domain names. For example, the front-end domain name configuration is shown in the Confd section.

6) K8s arranges files

After the image file is generated, it’s time to write the K8s choreography file, and then deploy the image to the K8s platform. The following layout files need to be configured:

configMap
ingress
service
deployment
PersistentVolumeClaim

ConfigMap is used to store required parameters and send them to ConfD in the image for rendering.

PersistentVolumeClaim is primarily used to mount external files or directories to mount the AndroidSDK so that multiple build nodes can share the SDK and save space.

JenkinsSlave images will use the mounted PVC as input to the Android SDK. Multiple JenkinsSlave nodes share the Android SDK in the same PVC to save storage space.

The PVC mount directory is /usr/local/aura/auraCfs, or you can mount it to another directory (for example, /mnt/auraCfs) and use /usr/local/aura/auraCfs as a soft chain.

Decompress the SDK file, there are two files:

aura-Cfs-mini-without-gradlecache.tar.xz
aura-Cfs-mini-with-gradlecache.tar.xz

The difference between the two is a 14GB Gradle cache. The cache may or may not be used. If the cache is not used, it will be automatically downloaded from the network, which only lengthens the first build time.

The steps are as follows:

Unzip file Xz to the root directory of PVC.
Choose to use gradle cache.
Pre-built Gradle caches can be used to speed up the first build, or instead of pre-built caches, dependent packages can be automatically downloaded from the network during the build process. To use the Grade cache, follow these steps;
Ensure that JenkinsSlave images have sufficient storage space (greater than 200GB);
Zip package using with-gradlecache;
Create an empty file use_gradle_cache in the root directory of the PVC disk.
After decompressing, the image startup script will output “Gradle UserHome cache recovery completed”.

experience

This article mainly introduces the Aura platform (Android component platform) into Docker image, image compilation and deployment process. In the process of privatization deployment, three main points are summarized as follows:

1) Dockerfile compilation and image compilation

2) Unified configuration management

3) Compilation of K8s layout files

If other platforms have the same requirements related to this paper, they can focus on the above key points and smoothly promote the deployment of privatization transformation.

M-paas is one of the products of JD.com’s T-PAAS family of cloud technologies. T-paas is the best practice and technological precipitation of JD.com’s digital technology for many years. Enterprise customers are provided with 16 categories of full-stack PaaS services that have undergone large-scale and rigorous production verification.