The background,

In daily development, we often have release requirements and encounter various environments such as Online environments, Staging environments, Dev environments, etc. The simplest way is to manually build and upload the server, but this way is too cumbersome, continuous integration can be a perfect solution to this problem, I recommend you to read Jenkins. There are many different ways to build Jenkins projects, and one of the most popular is the freeform style of software projects (a Jenkins build approach that combines SCM and build systems to build your projects, even systems outside of software). This method is sufficient for simple construction of a single project, but it is difficult to meet the requirements for multiple similar and different projects, or a large number of jobs will be needed to support, which exists. A small change will require modification of many jobs, which is difficult to maintain. We’ve had this problem before.

Currently, our team is responsible for developing and maintaining multiple Android projects, and each project needs to be built, with a very similar but somewhat different build process. For example, the construction process might look like this:

  • Clone code;
  • Static code inspection (optional);
  • Unit tests (optional);
  • Compile packaged APK or hot patch;
  • APK analysis, obtain VersionCode, package Hash value (apkhash), etc.
  • Reinforcement;
  • Upload test distribution platform;
  • Archive (optional);
  • Trigger automated tests (optional);
  • Inform responsible person of build results, etc.

The whole process is basically the same, but there are some differences. For example, some builds can have no unit tests, some builds don’t trigger automated tests, and build results are notified by different principals. If you use the normal build of a free-style software project, each project creates a job to handle the process (other jobs may be called).

This would have worked, but you have to take into account the possibility of new processes being added (such as secondary signatures), bugs in the build process, and so on. In either case, once the main build process is modified, the jobs of each project need to be modified and tested, and a significant amount of time is bound to be wasted. In view of this situation, we use the construction method of Pipeline to solve it.

Of course, if you have a project that integrates React Native, you’ll need to build the JsBundle. JsBundle may not be updated after Native modification. If JsBundle is built together during the construction of Native, it will cause a lot of resource waste. Putting large files such as jsbundles directly in a Native Git repository is not particularly appropriate.

This article is to share the experience of using a Pipeline to solve this kind of problem.

Two, the introduction of Pipeline

A Pipeline is a construction Pipeline, best described by programmers as the use of code to control the construction, testing, deployment, and so on of a project. The benefits of using it are many, including but not limited to:

  • Pipelines can be very flexible to control the entire build process;
  • You can clearly know the time used in each construction phase, which is convenient for the optimization of construction;
  • Build error, using stageView can quickly locate the error stage;
  • A job can handle the entire build, facilitating management and maintenance.

Stage View

Three, use Pipeline to build

Create a new Pipeline project and write the Pipeline build script like this:

  • Pipeline packaging scripts of multiple projects cannot be shared, resulting in one script for each project, which is more troublesome to maintain. A change that requires modifying scripts for multiple jobs;
  • Multiple people maintaining build jobs may overwrite each other’s code;
  • If the script fails to be modified, it cannot be rolled back to the previous version.
  • Build scripts cannot be versioned, old fixes need to be built, and the current job version may not be the same, and so on.

Four, write the Pipeline as code

Since there are defects, we need to find a better way to manage the Pipeline script. In fact, Jenkins provides a more elegant way to manage the Pipeline script, when configuring the project Pipeline, select the Pipeline script from SCM, like the following:

Thus, our build data sources are divided into three parts: job UI, general Pipeline scripts for the warehouse, and special configurations for the project. Let’s take a look at them respectively:

Job UI interface (Parameterized build)

When configuring the job, select the parameterized build process, passing in the project repository address, branch, build notifier, and so on. You can also add additional parameters that may require frequent modification, such as flexibility in choosing which branch of code to build.

Project configuration

In the project project, put the configuration for the project, which is usually a fixed parameter of the project and seldom changed, such as the project name, as shown below:

Injecting build information

When QA mentions a Bug, we need to determine which build it is, or know commitId so we can locate it. Therefore, build information can be injected into APK at build time.

  1. Inject the property intogradle.properties
CI_BUILD_NUMBER=0 CI_BUILD_NUMBER=0 CI_BUILD_NUMBER=0 The default value is 0 CI_BUILD_TIMESTAMP=0Copy the code
  1. Set buildConfigField in build.gradle
BuildConfigField "String", "APP_ENV", "\"${APP_ENV}\"" buildConfigField "String", "CI_BUILD_NUMBER", "\"${CI_BUILD_NUMBER}\"" buildConfigField "String", "CI_BUILD_TIMESTAMP", "\"${CI_BUILD_TIMESTAMP}\"" buildConfigField "String", "GIT_COMMIT_ID", "\"${getCommitId()}\"" // Get Git commitId String getCommitId() {try {def commitId = 'Git rev-parse HEAD'.execute().text.trim() return commitId; } catch (Exception e) { e.printStackTrace(); }}Copy the code
  1. Display the build information. In the App, find a suitable location, such as developer options, and display the previous information. When QA mentions bugs, ask them to bring this information with them
mCIIdtv.setText(String.format("CI build number :%s", BuildConfig.CI_BUILD_NUMBER));
mCITimetv.setText(String.format("CI build time :%s", BuildConfig.CI_BUILD_TIMESTAMP));
mCommitIdtv.setText(String.format("Git CommitId:%s", BuildConfig.GIT_COMMIT_ID));
Copy the code

Generic Pipeline scripts for the warehouse

A generic script is an abstract build process. When encountering anything related to the project, it needs to be defined as a variable and read from the variable. Do not write in the generic script.


node {
	try{
		stage('Check out code') {// Check out code from git repository
	    	git branch: "${BRANCH}".credentialsId: 'xxxxx-xxxx-xxxx-xxxx-xxxxxxx'.url: "${REPO_URL}"
	       	loadProjectConfig();
	  	}
	   	stage('compiled') {// Here is the build, you can call the job entry or project configuration parameters, such as:
	   		echo ${APP_CHINESE_NAME}"
	   		// Can be determined
	   		if (Boolean.valueOf("${IS_USE_CODE_CHECK}")) {
	   			echo "Static code review required"
	   		} else {
	   			echo "No static code checking required"
	   		}

	   	}
	   	stage('archive') {// This demo Android project, in actual use, please determine according to your own product
	       	def apk = getShEchoResult ("find ./lineup/build/outputs/apk -name '*.apk'")
	       	def artifactsDir="artifacts"// The folder where the product is stored
	        sh "mkdir ${artifactsDir}"
	       	sh "mv ${apk} ${artifactsDir}"
	       	archiveArtifacts "${artifactsDir}/*"
	   	}
	   	stage('Notify responsible person'){
	   		emailext body: "Build project :${BUILD_URL}\r\n Build completed".subject: 'Build result notification [success]'.to: "${EMAIL}"}}catch (e) {
		emailext body: "Building a project: ${BUILD_URL} \ r \ n build failures, \ r \ n error message: ${e. oString ()}".subject: 'Build result notification [failed]'.to: "${EMAIL}"
	} finally{
		// Clear the workspace
        cleanWs notFailBuild: true}}// Get the shell command output
def getShEchoResult(cmd) {
    def getShEchoResultCmd = "ECHO_RESULT=`${cmd}`\necho \${ECHO_RESULT}"
    return sh (
        script: getShEchoResultCmd,
        returnStdout: true
    ).trim()
}

// Load the configuration file in the project
def loadProjectConfig(){
    def jenkinsConfigFile="./jenkins.groovy"
    if (fileExists("${jenkinsConfigFile}")) {
        load "${jenkinsConfigFile}"
        echo "Found package parameter file ${jenkinsConfigFile}, loaded successfully"
    } else {
        echo "${jenkinsConfigFile} does not exist, please configure package parameters in project ${jenkinsConfigFile}"
        sh "exit 1"}}Copy the code

Lightly click Build with Parameters -> start building and wait a few minutes for the message to arrive.

Other construction structures

The above is only a good solution to the problem we are currently encountering. It may not be suitable for all scenarios, but it can be adjusted according to the above structure, for example:

  • Separate different Pipeline scripts according to the stages, so as to facilitate the maintenance of CI. One or several people maintain a stage in the construction.
  • Make the stages in the build process plainFree style software projectsJob, take them as basic services, call these basic services in Pipeline, etc.

6. When reacting Native

When React Native was introduced into the project, due to the technology stack, the React Native page was developed by the front end team, but the container and Native components were maintained by the Android team, and the construction process also changed.

Scheme comparison

plan instructions disadvantages advantages
Manually copy After the JsBundle is built, copy the completed product into Native project manually 1. Manual operation is troublesome, inefficient and error-prone

2. When it comes to cross-end collaboration, go to the front end team to take JsBundle every time

3. Git is not suitable for managing large and binary files		 Simple and crude
Use the SubModule to save the built JsBundle The JsBundle is directly placed in a submodule of the Native repository and updated by the front-end team. Each time the Native is updated, the latest JsBundle is directly obtained 1. Simplicity and no development costs

2. It is not convenient to control the version of JsBundle separately

3. Git is not suitable for managing large and binary files		 The front-end team can proactively update the JsBundle
Manage source code for JsBundle using subModule The source code of JsBundle is directly placed in a submodule of The Native repository, which is developed and updated by the front-end team. When building Native, JsBundle is constructed first 1. It is not convenient to control the version of JsBundle separately

2. Even if the JsBundle is not updated, it still needs to be built, which is slow and wastes resources		 Convenient and flexible
Build separately and archive the artifacts JsBundle is built separately from Native. The JsBundle is built separately from Native. When Native is built, you can directly download the built version of JsBundle 1. Configure and manage the JsBundle to liberate Git

2. It is convenient for Jenkins to dynamically configure the required version of JsBundle during Jenkins construction		 1. It takes time to establish processes

2. You need to develop the JsBundle download plug-in for Gradle

The front-end team develops the page, builds the JsBundle, and the Android team takes the front-end built JsBundle and packages it together to produce the final product. In our development process, JsBundle modification does not necessarily need to modify Native, and it does not necessarily need to rebuild JsBundle every time during Native build. In addition, these two parts are responsible for by two teams, and they should be independently developed. They should also be built independently during construction, and should not be merged together.

Comprehensive comparison, we choose to use separate build way to achieve.

Separate build

Since the release version needs to be separate, the JsBundle construction and Native construction need to be separated and completed by two different jobs. In this way, the two teams can operate independently and avoid mutual influence. JsBundle construction can also be done by referring to the construction method of Pipeline mentioned above, which will not be described here. Once you build it independently, how do you put it together? After the JsBundle is built, the versions of the JsBundle are stored in a place for Native to download the required version of the JsBundle during construction. The general process is as follows:

This process has two cores, one is the JsBundle archive store built and the other is downloaded at Native build time.

JsBundle archive storage

plan disadvantages advantages
File it directly on Jenkins 1. JsBundle cannot be viewed in summary

Many people may want to download, the name has version number, time, branch, etc., the name is not uniform, it is not convenient to build the download address

3. Downloading Jenkins’ products requires login authorization, which is quite troublesome		 1. Simple implementation, one line of code to fix, low cost
Build your own storage service 1. Large project and high development cost

2. It is troublesome to maintain		 Scalability, high flexibility
MSS

(Meituan Storage Service)		 There is no 1. Large storage space

2. High reliability and fast download speed with CDN

3. Low maintenance cost and low price

Here we chose MSS. To upload files to MSS, you can use s3CMD, but after all, not every Slave is installed on it, so it is not universal. In order to ensure stability and reliability, here based on THE MSS SDK to write a small tool, relatively simple, a few lines of code can be done.

private static String TenantId = "mss_TenantId==";
private static AmazonS3 s3Client;

public static void main(String[] args) throws IOException {
	if (args == null|| args.length ! =3) {
		System.out.println("Please enter inputFile, bucketName, objectName.");
		return;
	}
	s3Client = AmazonS3ClientProvider.CreateAmazonS3Conn();
	uploadObject(args[0], args[1], args[2]);
}

public static void uploadObject(String inputFile, String bucketName, String objectName) {
	try {
		File file = new File(inputFile);
		if(! file.exists()) { System.out.println("File does not exist:" + file.getPath());
			return;
		}
		s3Client.putObject(new PutObjectRequest(bucketName, objectName, file));
		System.out.printf("Uploading %s to MSS succeeded: %s/v1/%s/%s/%se", inputFile, AmazonS3ClientProvider.url, TenantId, bucketName, objectName);
	} catch (AmazonServiceException ase) {
		System.out.println("Caught an AmazonServiceException, which " +
				"means your request made it " +
				"to Amazon S3, but was rejected with an error response" +
				" for some reason.");
		System.out.println("Error Message: " + ase.getMessage());
		System.out.println("HTTP Status Code: " + ase.getStatusCode());
		System.out.println("AWS Error Code: " + ase.getErrorCode());
		System.out.println("Error Type: " + ase.getErrorType());
		System.out.println("Request ID: " + ase.getRequestId());
	} catch (AmazonClientException ace) {
		System.out.println("Caught an AmazonClientException, which " +
				"means the client encountered " +
				"an internal error while trying to " +
				"communicate with S3, " +
				"such as not being able to access the network.");
		System.out.println("Error Message: "+ ace.getMessage()); }}Copy the code

After we build directly in the Pipeline, we can call this tool. Jsbundles do not need to be permanently stored and can be deleted after a period of time. You can refer to MSS Life Cycle Management when deleting. So, we add a parameter to the job that builds the JsBundle.

// Upload to different buckets according to TYPE
def bucket = "rn-bundle-prod"
if ("${TYPE}"= ="dev") {
	bucket = "rn-bundle-dev" // There is life cycle management, automatically deleted after a period of time
}
echo "Start uploading JsBundle to MSS"
// The jar address needs to be replaced with your own
sh "Curl - s - s - http://s3plus.sankuai.com/v1/mss_xxxxx==/rn-bundle-prod/rn.bundle.upload-0.0.1.jar - o upload. L jar"
sh "java -jar upload.jar ${archiveZip} ${bucket} ${PROJECT}/${targetZip}"
echo ${archiveZip} ${archiveZip}
Copy the code

JsBundle downloads for Native builds

To enable automatic download at build time, we wrote a Gradle plugin. The first step is to configure the plugin dependencies in build.gradle:

classpath 'com. Zjiecode: rn - bundle - gradle - plugin: 0.0.1'
Copy the code

Apply plugins to required Modules:

apply plugin: 'mt-rn-bundle-download'
Copy the code

Configure JsBundle information in build.gradle:

RNDownloadConfig {
    // Remote file directory, because there are multiple types, so this can be filled in multiple.
    paths = [
            'http://msstest-corp.sankuai.com/v1/mss_xxxx==/rn-bundle-dev/xxx/'.'http://msstest-corp.sankuai.com/v1/mss_xxxx==/rn-bundle-prod/xxx/'
    ]
    version  = "1"// Version number, which is used to package JsBundle BUILD_NUMBER
    fileName = 'xxxx.android.bundle-%s.zip' // The name of the remote file,%s will be filled with the version above
    outFile  = 'xxxx/src/main/assets/JsBundle/xxxx.android.bundle.zip' // The downloaded storage path relative to the project root directory
}
Copy the code

The plug-in inserts a downloaded task before the Task of the package. The task reads the configuration information and checks for the existence of this version of JsBundle during the packaging phase. If it doesn’t exist, we’ll go to the archive’s JsBundle and download the JsBundle we need. Of course, version here can be injected as a job parameter using the build information described above. This allows Jenkins to dynamically fill in the required JsBundle version when building Native. The Gradle plugin is already in the Github repository, and you can make changes based on it. Of course, PR is welcome. Github.com/zjiecode/rn…

Six, summarized

We split a build into several parts, providing the following benefits:

  • Core build process, only need to maintain one, reduce maintenance work;
  • It is convenient for multiple people to maintain and build CI, avoiding Pipeline code being covered;
  • Easy to build job versioning. For example, to fix a released version, it is easy to switch to the version of the Pipeline script used in the released version.
  • For each project, the configuration is flexible. If the project configuration is not flexible enough, you can try to define more variables.
  • Build process visualization to facilitate targeted optimization and error location.

Of course, Pipeline also has some disadvantages, such as:

  • The Syntax is not friendly, but Jenkins provides a powerful tool to help.
  • Code testing is tedious, there is no local runtime environment, every test requires a job to be committed and run, and so on.

When a project integrates React Native with a Pipeline, we can upload the JsBundle build artifacts to the MSS archive. When building Native, it can be downloaded dynamically.

Seven, the author

Zhang Jie, senior Android engineer of Meituan-Dianping, joined chengdu R&D Center of catering platform in 2017, mainly responsible for b-side application development of catering platform. Wang Hao, Senior Android engineer of Meituan-Dianping, joined chengdu R&D Center of catering platform in 2017, mainly responsible for b-side application development of catering platform.

8. Recruitment ads

The writer is from Meituan Chengdu R&D Center (yes, we are building a R&D center in Chengdu). We have a number of openings for back end, front end and test positions in Chengdu. Please send your resume to [email protected].