This article is from OPPO Internet Basic technology team, please note the author. At the same time, welcome to follow our official account: OPPO_tech, share with you OPPO cutting-edge Internet technology and activities.

1. The background

The development of distributed technology has profoundly changed the way we program and the way we think about software. Distributed is a good solution to performance expansion, reliability, component availability and other problems, but the single-machine transformation into distributed has increased the complexity of the system, for component development, testing, deployment, release have put forward higher requirements. So, how to guarantee the software quality and system stability for the complex distributed system? First of all, let’s take a look at the general process of traditional software product activities. The simplified process is about 3 chunks:

Development -> QA -> Grayscale online!

The general picture is as follows:

There is a big problem with the process. The quality depends entirely on QA testing, and the docking depends entirely on human resources. The communication cost is high, and there are many missing problems.

  1. It is difficult for QA to test comprehensively every time. After all, QA is a human being, and the subjective factors of human beings are too big. Sometimes human judgment feels simple, and it is likely to miss the parts that need not be tested. Or think that the modification point is too simple, think not to go wrong, no longer comprehensive test. That there may be basic functional problems;
  2. Slow testing, low efficiency, and WASTE of QA resources. If QA requires full testing every time, there will be too much repetitive work, low efficiency, and poor effect. For these repetitive work, it could be better and faster, so that QA does not have the energy to do more things just to test such a few things;
  3. Even code that doesn’t compile well can miss QA;
  4. The closed loop is too slow. If there is a problem with the development function, it is too slow to wait until QA detects it and then report back to the development of the closed loop. Not to mention, if problems leak online and then feed back to developers, then Dai Jiang will be even bigger;
  5. When multiple developers are working in parallel, work can interfere with each other, minor problems can accumulate, and feature integration can be time-consuming;

Our proposed improvements:

  1. One of the core points: the problem should be found early, the earlier the discovery, the smaller the cost;
  2. One of the core points: the problem to close the loop faster, the faster the loop, the higher the efficiency;
  3. Repetitive work automation, reduce people’s invalid work;
  4. When there are more developers, the function is continuously integrated, and the problems are small and discovered in advance;

The core point is: “automated closed-loop problem”.

Today’s complex software systems put forward higher requirements for quality and efficiency, so the response software activities must be highly automated to meet the requirements. Automatic trigger, automatic test, automatic closed loop, automatic release, automatic card point and a series of guarantees, all predictable and solidifiable behavior should be automated, improve efficiency and quality, and let people do smarter things.

Our thinking: In recent years, there are some theories and practices of Continuous Integration, Continuous Delivery and Continuous Deployment for automation. Among these three, the word “continuous” is highlighted. “continuous” is to achieve the purpose of “fast”. “fast iteration”, “fast response”, “fast closed loop” and “fast” are the core competitiveness. Generally, the consensus process is classified as follows:

For developers, it is more about Continuous Integration. CI solidifies the process through automation to ensure orderly and reliable code Integration, controllable version and traceable problem. Automation reduces the subjective error rate and improves the speed in code activities. Improve release quality and efficiency.

2. What is CI?

CI is Continuous Integration, which is a crucial link in today’s software activities. CI is usually triggered by a developer submitting a code change, and is automatically validated in an intermediate environment. After CI validation, basic quality assurance has been achieved, further software activities can be allowed.

Continuous integration is simply a software development practice in which team members integrate as quickly as possible, and each integration is verified by an automated build (including compilation, release, and automated testing) to detect integration errors as early as possible. The earlier a problem is discovered, the less it costs to fix.

In general, continuous integration requires several steps:

  1. Code Commit (Git)
  2. Task Building (Jenkins)
  3. Deployment tests (Ansible, shell, Puppet)

Highlight: CI processes are triggered by code activities.

Code activities focus on two points in time:

  • Pre-merge: code changes are triggered before the main trunk branch. The object of integration is the code after the Merge of the code changes and the latest code in the trunk. The purpose of integration is to verify whether the code changes can Merge with the main trunk.
  • Post-merge: triggered when code changes Merge with the main trunk branch. The integration object is the latest trunk branch code, which is used to verify that the trunk works properly after the modified code is added.

Pre-Merge and Post-Merge have different concerns. What’s the difference? If there is only one developer, pre-merge and post-merge test objects are the same. Pre-merge and post-merge can be different when multiple developers submit code, with different CI test subjects.

In other words, pre-merge is parallel. Every development branch that wants to Merge the main branch fires a Pre-merge CI. The TEST object of the CI is < development + trunk >.

3. Four reflections on CI

3.1 How can CI be triggered?

Code activity trigger:

Generally, there are two trigger points, pre-merge and post-merge, respectively, before the code merges into the main trunk and after the trunk code merges.

Timed trigger.

3.2 What to do after CI triggers?

What happens when CI triggers? To put it bluntly, the build task does what it does. There are generally several processes:

  1. Checkout, pre-merge code — check whether MR Merge is valid;
  2. Code compilation – verifies that code compilation is valid;
  3. Static check – verifies whether static syntax is valid.
  4. Unit testing – regression tests function unit validity;
  5. Smoke test — simple test whether the system is normal;
  6. Interface test – test whether the user interface is normal;
  7. Performance benchmarking – Tests whether performance meets expectations;

3.3 How to close the loop?

Think about what might go wrong first:

  1. Pre-merge code conflicts;
  2. Code compilation failed;
  3. Static check failed.
  4. Unit test regression test failed;
  5. Smoke test step passed, interface test failed… ;

Submit a code MR submit may encounter the above problems, so how to quickly close the loop this problem?

First, there has to be a way to inform developers

Solution:

  1. Comments of MR are automatically added to MR in the form of comments after CI activities fail.
  2. The email, triggered by a MR, failed to be sent to the relevant person by email;

Again, there has to be a way to let developers know about problems

Solution:

  1. Once a developer knows that his MR triggers a CI failure, he needs to know how to troubleshoot the problem — test reports,
    • For example, unit test failure should have unit test report, interface test failure should have interface test report;
  2. Keep archive clues for troubleshooting on each build task;

3.4 What is the output of CI build activity?

  1. Unit test report;
  2. Interface test report;
  3. Code coverage reports;
  4. Interface coverage report;
  5. Build the release pack (required for sustained deployment);

4. CI platform selection

Generally speaking, there is no need for most companies to develop a CI platform by themselves. There are many excellent open source CI platform tools, and there is no absolute difference between the tools, so the selection will not be conducted here. We will use a complete example of Jenkins to illustrate the methods and skills of using CI. The code repository we use the Gitlab CI platform and we use open source Jenkins as a demonstration. Step by step we need to complete several module functions.

Platform selection: Code platform Gitlab, CI platform Jenkins;

5. CI process practice

CI is the main gatekeeper for code activities and generally has two triggers:

  1. Before the code is combined with the main stem, CI test is triggered to verify whether the integration meets the quality expectation. If not, the code is not allowed to be combined with the main stem.
  2. After the code is merged into the main trunk, CI tests are triggered to verify whether the latest trunk branch meets the quality expectations.

Pre-merge trigger process:

  1. Develop code to submit a Merge Request.
  2. MR automatically triggers CI build events
  3. Run static checks, Merge checks, unit tests, smoke tests, and integration tests before the code allows the Merge to Merge into the main trunk.
  4. Proceed to the next software activity

Post-merge trigger process:

  1. The administrator approves the Merge Request, and codes Merge into the main trunk, triggering the post-merge event.
  2. CI platform automatically builds CI after receiving the event;
  3. After the construction is completed, proceed to the next software activities;

6. Jenkins platform construction

6.1 Jenkins platform construction

Jenkins is Java program development, installation is very convenient, go to the official website to download a WAR package, and then pull up the background to run. Run the following command:

Start the

nohup /usr/bin/java -jar jenkins.war --httpPort=8888 >> jenkins.log 2>&1 &
Copy the code

So the Jenkins platform goes up. Super easy.

6.1.1 Initializing the Platform

The original key

The platform needs to be configured for the first time. Find an “initial key” in the log and pay attention to the prompt:

* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * ************************************************************* Jenkins initial setup is required. An admin user has been created and a password generated. Please use the following password to proceed to installation: 5ddce31f4a0d48b4b7d6d71ff41d94a8 This may also be found at: /root/software/jenkins/workhome/secrets/initialAdminPassword * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *Copy the code

This is the initial superuser password that Jenkins will use later, so grab a notebook and write it down.

The login page

Now open your browser and log in to Jenkins’ page. Let’s initialize Jenkins:

Click “Continue” to proceed to the next step, which is to customize some plug-in installations.

First custom plug-in installation

This is optional, this is based on their own needs to choose the plug-in, to save trouble, choose the first kind of good, and then it is also easy to download plug-ins on the platform.

Plug-in installation process (Jenkins almost completely assembles functionality from plug-ins) :

Successful updates will appear green. After the plug-in is installed, the next step is to configure the first superuser.

Once the configuration is complete, click “Save and Continue” and as a final step, configure the URL and click Finish:

Now that the basic configuration is complete, it’s time to have fun with Jenkins.

6.1.2 Use tips

Chinese cultural disposition

After the Jenkins platform is set up, the default version is in English, which may not be necessary in China. We can install a Chinese culture plug-in to show our Jenkins in a more friendly way. There are two steps:

Step 1: Install the plug-in “Locale” :

“Manage Jenkins” -> “Manage Plugins” -> “Available”

Step 2: After the installation, Configure Configure

Account configuration

6.2 Jenkins plug-in installation

6.2.1 Plug-in Update address

Here recommend the domestic plug-in source address, because the official website network access is not very stable. For example, here is tsinghua’s mirror source.

https://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json
Copy the code

6.2.2 Mandatory Plug-in Installation

Jenkins functions are all provided by plug-ins, and some plug-ins must be equipped to provide complete CI functions, such as Pipeline. Here are a few key plug-ins to use.

pipeline

Jenkins essential plugins, pipelined plugins, can be very convenient for you to define processes, scheduling nodes, allocating resources, processing results, etc.

blue ocean

Pipeline visualization plug-in, pipeline or declarative code writing, if you want to make people more convenient to use, then need a visualization tool, Blue Ocean is born for this.

junit

A parsing plug-in for test reports, which is also a more common test report format.

Cobertura Plugin

Coverage shows a plug-in. To complete a single test run, there needs to be a means of knowing the coverage and a convenient closed-loop process.

  1. Display coverage rate;
  2. Code coverage details for developers to close the loop (down to each line of code);
GitLab

Our demo uses Gitlab as an example, and we need to interact with Gitlab, so we need to install plug-ins to accept Gitlab events and feedback CI results.

6.3 Jenkins Task Creation

6.3.1 Creating a Task (Item)

Item is a CI item that is statically created and configured by the administrator and triggered as a job. The job number increases with each trigger. Click On New Item to create a “pipeline” Item.

6.3.2 Creating a View (View)

What is a View? A View can summarize tasks that have business significance and display them in a list. You can create it by clicking “New View”.

The view will display the item, which you can optionally check.

6.4 Jenkins assembly line

Pipeline frameworks describe processes through Pipeline scripts. Pipelines can be created in two ways, with two syntax:

  1. Declarative pipelining syntax
  2. Scripted pipeline syntax

Now the official recommendation is declarative pipelining. So, what does declarative syntax look like? One feature of declarative syntax is that the top layer must start with Pipeline {}.

7. Jenkins interacts with Gitlab

This step is the most important thing. Once Jenkins has built it, if it’s just an island platform, it doesn’t mean anything. It has to be involved in the software development process to make it work. The interaction diagram is as follows:

We see that The code activities of Gitlab need to trigger Jenkins in the form of events. After Jenkins completes a series of activities, he needs to feed back the results to Gitlab and influence the next activities of Gitlab. So Gitlab and Jenkins need to configure relationships with each other.

7.1 Gitlab configuration

Why do I need to configure GitLab? Because we need a route from Gitlab to Jenkins. As a code warehouse, GitLab mainly generates events related to the project code, such as Merge Request, Push Commit, etc. When gitLab generates these events, it needs to automatically Push the event to Jenkins, so as to get through the triggering interaction.

7.1.1 Configuring Web Hook Events

Operation steps:

  1. Open the code repository
  2. Go to Setting -> Integrations
    1. Fill in the URL
    2. Fill in the Secret Token
    3. Check Trigger event

Where did the URL and Secret Token come from? This corresponds to item.

On the Jenkins platform, open the corresponding item, open Configure, check Build Triggers and find “Build when a change is Pushed to GitLab” and there it is.

To add a Secret token further down, click Generate.

Fill in these two correctly, and you can get through the first step: the Gitlab to Jenkins trigger. After filling in, GitLab can send a test event to test.

Return 200 and you’re done.

7.1.2 Configuring pre-Merge Card Points

Code pre-merge CI does not Merge the main stem. The key is for GitLab to support Hook behavior on code Merge eve.

  1. First of all, we agree on a code of conduct: all codes merged with the master trunk must be submitted to MR, and only after passing the MR CI test can the master trunk be merged.
  2. Next, check Settings -> General -> Merge requests to select “Only allow Merge requests to be merged if the pipeline succeeds”;

7.2 Jenkins configuration

Jenkins mainly looks at Pipeline configuration, Pipeline configuration open Configure as follows:

Take a look at a complete Pipeline shelf that defines the phases (you can just copy it and run it to see the effect) :

Pipeline {agent any stages {stage(' code checkout') {steps {echo "------------"}} stage(" static check ") {steps {echo "-- -- -- -- -- -- -- -- -- -- -- --"}} stage (" the code to compile ") {steps {echo "-- -- -- -- -- -- -- -- -- -- -- --"}} stage (" unit tests ") {steps {echo "-- -- -- -- -- -- -- -- -- -- -- --"}} Stage (" packaging ") {steps {echo "-- -- -- -- -- -- -- -- -- -- -- --"}} stage (" smoke test ") {steps {echo "-- -- -- -- -- -- -- -- -- -- -- --"}} stage (" integration testing ") {steps { Echo "-- -- -- -- -- -- -- -- -- -- -- --"}} stage (" benchmark performance test ") {steps {echo "-- -- -- -- -- -- -- -- -- -- -- --"}}} post {always {echo "-- -- -- -- -- -- -- -- -- -- -- --"} success { echo "------------" } failure { echo "------------" } unstable { echo "------------" } } }Copy the code

Run out of the effect:

Blue Ocean effects:

Next, let’s break down several key stages for analysis.

7.2.1 code checkout

The Checkout code, which is our test object, is different for pre-merge and post-merge. The pre-merge card point is triggered by the Merge Request event. We need to Checkout out the “code change” + “latest trunk branch” code. Post-merge is relatively simple; we simply Checkout out the latest trunk branch. How to do?

Pipeline directly supports this use.

Stage (' code checkout') {steps {dir(path: "${code path you want to place}") {checkout Changelog: true, poll: true, SCM: [$class: 'GitSCM', branches: [[name: "*/${env.gitlabSourceBranch}"]], doGenerateSubmoduleConfigurations: false, extensions: [ [$class: 'PreBuildMerge', options: [fastForwardMode: 'FF', mergeRemote: 'origin', mergeStrategy: 'DEFAULT', mergeTarget: "${env.gitlabTargetBranch}" ]], [ $class: 'UserIdentity', email: "${env.gitlabUserEmail}", name: "${env.gitlabUserName}" ] ], submoduleCfg: [], userRemoteConfigs: [[ credentialsId: "${env.OCS_GITLAB_CredentialsId}", url: "${env.gitlabSourceRepoHttpUrl}" ]] ] } } }Copy the code

It is specified here:

  • The path to place from checkout (dir path configuration);
  • Env.gitlabsourcebranch specifies the checkout branch. Env.gitlabsourcebranch is set automatically when the MR event triggers a CI.
  • Specifies the checkout behavior, PreBuildMerge behavior (which also applies to post-merge PUSH events);
  • Specify gITLAB credential (credentialsId);

The syntax above applies to both pre-merge and post-merge.

7.2.2 Static Check

A static check is a static check on the syntax of your code. For example, golang can use go Vet, or Go FMT. After this check, you can ensure that your code has eliminated the most basic syntax and formatting errors.

7.2.3 Unit Tests

It is necessary to unit test the smallest functions to get a coverage picture of the project. At this stage we get two things:

  • Unit test case reports
  • Coverage report

How do I get the test report? Take Golang as an example. When running a single test, turn on the coverage switch and output the standard output to a file:

go test -cover -coverprofile=cover.output xxx | tee ut.output
Copy the code

Two files are generated:

  • Ut.output: the file used to generate the unit test report;
  • Cover. output: the file used to generate the coverage report;

Single test report generation

This file was first parsed into an XML file and then reported to Jenkins in junit for display.

sh "go-junit-report < ut.output > ut.xml"
junit 'ut.xml'
Copy the code

Where does go-Junit-Report come from? This is an open source tool designed specifically for unit test parsing.

The effect shown on Jenkins is as follows:

Coverage report generation

Parse the coverage output file to generate an XML file:

gocov convert cover.output | gocov-xml > cover.xml
Copy the code

This XML file is submitted for Jenkins platform display:

step([
        $class: 'CoberturaPublisher', 
        autoUpdateHealth: false, 
        autoUpdateStability: false, 
        coberturaReportFile: '**/cover.xml', 
        failUnhealthy: false, 
        failUnstable: false, 
        maxNumberOfBuilds: 0, 
        onlyStable: false, 
        sourceEncoding: 'ASCII', 
        zoomCoverageChart: false
    ]
)
Copy the code

Click into file to see details of code coverage:

7.2.4 Interface Tests

Do some interface level testing on the entire system, such as simulating user behavior and testing the interfaces that users call, to ensure basic functionality. The report output can also be in junit format, which can be reported to Jenkins. The analysis is shown in the figure below:

7.2.5 Sending Emails

If the test passes or fails, you need to send an email with the result.

configFileProvider([configFile(fileId: '5f1e288d-71ee-4d29-855f-f3b22eee376c', targetLocation: 'email.html', variable: 'content')]) {script {template = readFile encoding: 'utF-8 ', file: "${content}" emailext(subject: "CI ") ${currentBuild.result? : 'Unknow'}", to: "[email protected]", from: "[email protected]", body: """${template}""" ) } }Copy the code

7.2.6 Gitlab status interaction

{gitLabConnection('test-gitlab') gitlabBuilds(builds: ['jenkinsCI'])} // Gitlab updateGitlabCommitStatus name: 'jenkinsCI', state: 'success' addGitLabMRComment comment: "" "CI Jenkins automatically build details * * * * \ n | | | item value | -- -- -- -- -- - | -- -- -- -- -- - | | | results ${currentBuild. The result? : 'Unknow'} | | MR LastCommit | ${env.gitlabMergeRequestLastCommit} | | MR id | ${env.gitlabMergeRequestIid} | | Message Title ${env. GitlabMergeRequestTitle} | | | build task ID | ${env. BUILD_NUMBER} | | | build details link [${env.RUN_DISPLAY_URL}](${env.RUN_DISPLAY_URL})"""Copy the code

If CI succeeds or fails, this state needs to be sent to GitLab. We will show the results with a Comment and attach the jump link of Jenkins task, which can help developers to close the loop in the fastest way.

Only after success is allowed to join:

Gitlab CI

7.2.7 Building an Archive

Package log:

Gz ${SERVICEDIR}/run/*.log" // Jenkins archiveArtifacts allowEmptyArchive: true, artifacts: "log.tar.gz", followSymlinks: falseCopy the code

8. Jenkins advanced Techniques

8.1 Resources are Mutually Exclusive

Sometimes, when multiple tasks are running, it is possible to use a resource concurrently, and if the resource is limited, it may be necessary to use some mutex to guarantee it. For example, two tasks might use mongodb, and if mongodb only has one set, multiple tasks must be executed sequentially or the logic will run wrong. How to do?

This can be defined by going to Configure System -> Lockable Resources Manager:

Then use the lock resource in the Pipeline script:

Stage (" unit test ") {steps {lock(resource: "UT_TEST", quantity: 1) {echo "= = = = = = unit test = = = = = = = = = = = =" echo "= = = = = = unit tests to complete the = = = = = = = = = = = ="}}}Copy the code

You can also see which Resources are occupied by which tasks on the Dashboard -> Lockable Resources:

By defining Lock resources properly, we can make the task concurrent, but the key race resources are mutually exclusive, so that CI building tasks are more flexible and efficient. (This can be analogous to the effect of Lock granularity in code. If you don’t use Lock resources, It is likely that the only way to protect a race resource is to set node concurrency to 1.

8.2 Node Scheduling

Jenkins allows you to schedule specified tasks to appropriate nodes. If there are multiple nodes, you may want task A to be fixed to node1 for execution. You can use the agent command to specify this.

Each node is assigned a label name when it is defined, and then the node can be specified when it is run:

agent { label "slave_node_1" }
Copy the code

8.3 File Transfer between Nodes

We use stash and unstash to do this. Here is an example of lossless transfer of build/ directory between node1 and node2:

Stage (" package ") {agent {label "slave_node_1"} steps { stash (name: "buildPkg", includes: "Build /**/*")}} Stage (" Smoke deployment ") {agent {label "slave_node_2"} steps {// Unstash ("buildPkg")}}Copy the code

8.4 Post-Clearing nodes

When switching between multiple nodes in the pipeline, you need to pay attention to which node you are in, so as not to get confused.

Pipeline {agent {label "master"} stages {stage (" test ") {agent {label "slave_node_1"} steps {} Always {// Clean the slave_node_1 build space cleanWs()}}}} // Pipeline total post {always {// only clean the master build space cleanWs()}}}Copy the code

When you have multiple nodes, remember to clean up the nodes separately.

9. To summarize

In modern software development activities, CI is an essential process by using a reasonable technical platform to reasonably connect people and things. Developers are involved in IT. Starting from code activities, CI can quickly respond to the construction results to the corresponding people, and provide means for the corresponding people to solve the problem quickly. We demonstrated the complete setup process through Jenkins (CI platform) + Gitlab (warehouse), showing a practical process. It’s all about software development efficiency and release quality.