CI/CD Half bucket of water (four)

Column address: CI/CD assembly line

CI/CD Half bucket of water (1), CI/CD half bucket of water (2), CI/CD half bucket of water (3)

Clfeng

Learners’

Now that you’ve covered the most basic concepts and usage of CI/CD, let’s look at some of the more advanced features: Webhook and GitLab API calls. The overall structure of the article is to put forward two specific functional requirements, and then step by step to achieve.

Demand background

Feature 1: Automatically remind code reviewers to review code through internal chat tool

In the actual project development process, when we finish developing a function, we will push our code to the remote warehouse, put forward a merge request and assign it to our leader, who will help us review the code and merge it into the trunk. Therefore, after we submit the merger request, we may send the link to the leader through the internal chat tool and ask the leader to help us quickly review and merge it into the main trunk. So can we automate this step through CI/CD?

Function two: three party library upgrade card control, remind developers to do a good job of self-testing

We expect to analyze developer changes through CI/CD when merging requests. For some high-risk changes (such as component library upgrade, third-party library upgrade), we expect to automatically add comments to let developers pay attention to, and do the corresponding self-test and safeguard measures.

So how do we analyze whether a developer has made a component library upgrade? Take the Element-UI library as an example:

#The developer's personal branch is update-component-clF
#The branch to merge into is master
#So we can use the following script to determinegit diff HEAD origin/master ./yarn.lock | grep "element-ui"; if [ $? -eq "0" ]; Then echo "Update elmenent- UI library" then echo "Update Elmenent - UI library" fiCopy the code

From the above example, it is not difficult to imagine that git commands can be used in the pipeline of merge requests to determine whether an update to the Element-UI library has taken place. In order to be able to analyze with the above command, we need to know which branch is the target of the merge request. Is there any way to get that? If your GitLab version is older than 11.6, this is easy to get with the CI_MERGE_REQUEST_TARGET_BRANCH_NAME default variable. But what if GitLab’s version is lower? Webhook and GitLab API calls may help you solve the problem.

Let’s implement our requirements step by step:

Create a local Node service

#Create and enter the directory
mkdir gitlab-server;
cd gitlab-server;

#Install dependencies
yarn init -y;
yarn add koa koa-bodyparser koa-route;

#Create an
touch app.js;
Copy the code

# app.js
const Koa = require('koa');
const route = require('koa-route');
const bodyParser = require('koa-bodyparser');

const app = new Koa();
const port = 3000;

app.use(bodyParser());

app.use(route.post('/merge_events'.async (ctx) => {
  console.log(ctx);
}));


app.use(route.get('/'.ctx= > {
  ctx.body = 'Hello World';
}));

app.listen(port);

console.log(`Example app listening at http://localhost:${port}`);
Copy the code

Start the Node Server

#In the gitlab-server directory, however, it is recommended to start in vscode debug mode so that you can view the relevant content of the request directly
node app.js
Copy the code

Add merge request hook

To generate the Secret token

Add the hook

A solution found on the official website

Emmmm cannot send requests directly to the local network due to security issues. Releasing restrictions requires access to the Admin panel. But because I only registered a common account in GitLab to use, there is no admin permission, let alone admin panel, it seems that I can only build a GitLab on the server.

Build GitLab

Supported systems:

Ubuntu (16.04/18.04/20.04)
Debian (9/10)
CentOS (7/8)
Packages Leap (15.2)
SUSE Linux Enterprise Server (12 SP2/12 SP5)
Red Hat Enterprise Linux (please use the CentOS packages and instructions)
Scientific Linux (please use the CentOS packages and instructions)
Oracle Linux (please use the CentOS packages and instructions)

Note: The GitLab server is installed on a centos: 7 system running on a local VIRTUAL machine, so the following installation operations are performed on centos

The official website installation tutorial:about.gitlab.com/install/

Install SSH and process firewalls

sudo yum install -y curl policycoreutils-python openssh-server perl

#Command to check the status of the SSHD service
sudo systemctl status sshd

#If the service is not up, run the following command
sudo systemctl enable sshd
sudo systemctl start sshd

#Here is the firewall related processing, here directly paste the relevant command of the official website, but suggest that if it is a beginner, the server is not very understanding and learning the server (local VIRTUAL machine),
#Can close the firewall directly, trouble to all sorts of problems

#Close firewall, the author in order to save trouble directly off
sudo systemctl stop firewalld

#The following is an excerpt from the official website
# Check if opening the firewall is needed with: sudo systemctl status firewalldsudo firewall-cmd --permanent --add-service=http sudo firewall-cmd --permanent --add-service=https sudo systemctl reload  firewalldCopy the code

Add the GitLab package repository and install the packages

1. Add the GitLab software package repository

curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh | sudo bash
Copy the code

2. Install the software package and set the exposed address

#Replace http://192.168.56.101 with the address of your serverSudo EXTERNAL_URL="http://192.168.56.101" yum install -y gitlab-eeCopy the code

And then go back to our local machine (my MAC)

3. Visit GitLab

After the installation is successful, open a browser and visit http://192.168.56.101

4. Log in

The account is root

The password is randomly generated when you run the sudo EXTERNAL_URL=”http://192.168.56.101″ yum install -y gitlab-ee command.

The randomly generated password is stored in /etc/gitlab/initial_root_password. You can run the cat /etc/gitlab/initial_root_password command to view the password

Note: Check the password on the centos server where GitLab is installed

Note: The initial password will be cleared within 24 hours, so please change the password in time

Environment to prepare

Allows requests to be made to the local network from Web hooks and services

Remember why we built our own GitLab? To handle webhook’s inability to send requests to the local network. Now that you have a super administrator account, let’s follow the tips on the website and let go of the restrictions

The project creates and adds SSH keys

Create a project

Since the newly built project is empty, let’s create a new project

Create and add an SSH Key

Specific operations can see docs.gitlab.com/ee/ssh/inde…

Generally, SSH keys are generated locally, after all, we’ve been pulling project push code in previous chapters. So you can go to the.ssh directory to view the corresponding public key.

cd .ssh
ls
#Here are the files from the search
id_ed25519	id_ed25519.pub	id_rsa		id_rsa.pub	known_hosts

#Copy the output and paste it into the input box below
cat id_rsa.pub
Copy the code

Add webhook

Then we will go back to the operation of adding merge request hook. According to the previous tutorial, we will do the same operation on the GitLab built by ourselves. The main steps are as follows:

Registered runner
Add Pipeline Triggers and get the security token
Add webhook

Based on the previous steps, we will eventually come to the following steps

Start the local service and make a breakpoint

Query merge request packets

Then we add the.gitlab-ci.yml file to the project and add the merge request to see what the local Node service receives

# Content is our first assembly line
# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

build_job:
  stage: build # specify stage to which the job belongs
  image: centos:7 # specify the docker image used to execute the job
  tags: 
    - clf-runner # specify the runner that executes the job.
  script: Script run during job execution
    - echo "build project"

test_job:
  stage: test
  image: centos:7
  tags: 
    - clf-runner
  script:
    - echo "test project"

deploy_job:
  stage: deploy
  image: centos:7
  tags: 
    - clf-runner
  script:
    - echo "deploy project"

Copy the code

Make a merge request

Viewing request Packets

Message analysis

There are many and complete messages in the received message, so let’s take a look

Note: The message is quite long, so it would be good to know what the message might be

// This is the packet information in ctx.headers
{
  "content-type": "application/json"."user-agent": "GitLab / 14.2.3 - ee"."x-gitlab-event": "Merge Request Hook".// What is the event
  "x-gitlab-token": "839fa63721bcb59a246d991410b30e".// Here is the trigger token we added earlier.
  connection: "close",
  host: "192.168.1.4:3000"."content-length": "3696",}Copy the code

// Here is the message information in ctx.request.body
{
  object_kind: "merge_request",
  event_type: "merge_request",
  user: {
    id: 1,
    name: "Administrator",
    username: "root",
    avatar_url: "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon",
    email: "[email protected]",
  },
  project: {
    id: 2,
    name: "cicd-demo",
    description: "cicd-demo",
    web_url: "http://192.168.56.101/root/cicd-demo",
    avatar_url: null,
    git_ssh_url: "[email protected]: root/cicd - demo. Git." ",
    git_http_url: "http://192.168.56.101/root/cicd-demo.git",
    namespace: "Administrator",
    visibility_level: 0,
    path_with_namespace: "root/cicd-demo",
    default_branch: "main",
    ci_config_path: null,
    homepage: "http://192.168.56.101/root/cicd-demo",
    url: "[email protected]: root/cicd - demo. Git." ",
    ssh_url: "[email protected]: root/cicd - demo. Git." ",
    http_url: "http://192.168.56.101/root/cicd-demo.git",
  },
  object_attributes: {
    assignee_id: 1,
    author_id: 1,
    created_at: "2021-09-11 04:15:57 UTC",
    description: "Feat: Add. Gitlab-ci. yml configuration",
    head_pipeline_id: null,
    id: 1,
    iid: 1,
    last_edited_at: null,
    last_edited_by_id: null,
    merge_commit_sha: null,
    merge_error: null,
    merge_params: {
      force_remove_source_branch: "1",
    },
    merge_status: "preparing",
    merge_user_id: null,
    merge_when_pipeline_succeeds: false,
    milestone_id: null,
    source_branch: "feat-cicd-clf",
    source_project_id: 2,
    state_id: 1,
    target_branch: "main",
    target_project_id: 2,
    time_estimate: 0,
    title: "Feat: Add. Gitlab-ci. yml configuration",
    updated_at: "2021-09-11 04:15:57 UTC",
    updated_by_id: null,
    url: "http://192.168.56.101/root/cicd-demo/-/merge_requests/1",
    source: {
      id: 2,
      name: "cicd-demo",
      description: "cicd-demo",
      web_url: "http://192.168.56.101/root/cicd-demo",
      avatar_url: null,
      git_ssh_url: "[email protected]: root/cicd - demo. Git." ",
      git_http_url: "http://192.168.56.101/root/cicd-demo.git",
      namespace: "Administrator",
      visibility_level: 0,
      path_with_namespace: "root/cicd-demo",
      default_branch: "main",
      ci_config_path: null,
      homepage: "http://192.168.56.101/root/cicd-demo",
      url: "[email protected]: root/cicd - demo. Git." ",
      ssh_url: "[email protected]: root/cicd - demo. Git." ",
      http_url: "http://192.168.56.101/root/cicd-demo.git",
    },
    target: {
      id: 2,
      name: "cicd-demo",
      description: "cicd-demo",
      web_url: "http://192.168.56.101/root/cicd-demo",
      avatar_url: null,
      git_ssh_url: "[email protected]: root/cicd - demo. Git." ",
      git_http_url: "http://192.168.56.101/root/cicd-demo.git",
      namespace: "Administrator",
      visibility_level: 0,
      path_with_namespace: "root/cicd-demo",
      default_branch: "main",
      ci_config_path: null,
      homepage: "http://192.168.56.101/root/cicd-demo",
      url: "[email protected]: root/cicd - demo. Git." ",
      ssh_url: "[email protected]: root/cicd - demo. Git." ",
      http_url: "http://192.168.56.101/root/cicd-demo.git",
    },
    last_commit: {
      id: "38719961be0cfa04d15cf4cca5441883219162c0",
      message: "Feat: add. Gitlab-ci. yml config \n",
      title: "Feat: Add. Gitlab-ci. yml configuration",
      timestamp: "2021-09-11T12:14:16+08:00",
      url: "http://192.168.56.101/root/cicd-demo/-/commit/38719961be0cfa04d15cf4cca5441883219162c0",
      author: {
        name: "clfeng",
        email: "[email protected]",
      },
    },
    work_in_progress: false,
    total_time_spent: 0,
    time_change: 0,
    human_total_time_spent: null,
    human_time_change: null,
    human_time_estimate: null,
    assignee_ids: [
      1,
    ],
    state: "opened",
    action: "open",
  },
  labels: [
  ],
  changes: {
    merge_status: {
      previous: "unchecked",
      current: "preparing",
    },
  },
  repository: {
    name: "cicd-demo",
    url: "[email protected]: root/cicd - demo. Git." ",
    description: "cicd-demo",
    homepage: "http://192.168.56.101/root/cicd-demo",
  },
  assignees: [
    {
      id: 1,
      name: "Administrator",
      username: "root",
      avatar_url: "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon",
      email: "[email protected]",},],}Copy the code

Need to implement

So far our environment is really ready. So let’s go back and look at the two requirements that we originally wanted to implement.

When a merge request is made, the internal tool automatically alerts the delegate to review the code
Third party library upgrade card control and additional variables supplement

Automatically alerts merge code

Earlier we added webHook to the merge request, which automatically sends a POST request to our Node service when the merge request occurs. We can then parse the information in the request to know who assigned the merge request, and then send a message to it through the internal chat API.

Modify the node service app.js file

# app.js
const Koa = require('koa');
const route = require('koa-route');
const bodyParser = require('koa-bodyparser');

const app = new Koa();
const port = 3000;

app.use(bodyParser());

app.use(route.post('/merge_events'.async (ctx) => {
  const body = ctx.request.body;

  // Merge the requested URL
  const mergeReqUrl = body.object_attributes.url;

  // Assign the person's information, such as id, username, email
  const assignee = body.assignees[0];

  // With this information we can find the person assigned to review the merge request;
  dim(assignee.id, 'There is a merge request for your review:${mergeReqUrl}`)}));// This function encapsulates the ability to call the chat API to notify the person with the corresponding ID
// The implementation depends on the internal chat tool
function dim (userId, message) {
  console.log(` [${userId}] :${message}`)
}

app.use(route.get('/'.ctx= > {
  ctx.body = 'Hello World';
}));

app.listen(port);
console.log(`Example app listening at http://localhost:${port}`);
Copy the code

This completes the first requirement: the internal tool automatically alerts the delegate to review the code when a merge request is made

Tripartite library upgrade card control

As mentioned earlier, we can implement our functionality with the following commands

git diff HEAD origin/master ./yarn.lock | grep "element-ui"; if [ $? -eq "0" ]; Then echo "Update elmenent- UI library" then echo "Update Elmenent - UI library" fiCopy the code

If we want to execute the above command in job, we need:

Git commands can be executed in the container where the job resides
We know the destination branch of the merge request (origin/master written above), but our merge request could be Branch_A merging into Branch_B

In order to execute git commands in a container, let’s build our own container image.

Build the mirror

In the future, you need to push the image to the remote server, which requires your own Docker account. If not, please register it first. Link:hub.docker.com/

We created a docker directory under cicD-Demo project to store the docker-related scripts we wrote

1. Write Dockerfile

# Dockerfile
FROM centos:7
RUN yum install git -y
Copy the code

2. Write a build script

# build.sh
#! /bin/bash

#Clfeng/CLF-CICD-centos :1.0 is a complete image name
#Clfeng is the author's Docker account name (it should be replaced with my own account name here)
#Clf-cicd-centos indicates the image name defined by the author
#The 1.0 is the image identifier, which can also be understood as the version numberDocker build-t clFENG/CLF-CICD-centos :1.0.Copy the code

3. Build an image

Upload the entire Docker directory to the server to build the image

4. Push the image to the remote repository

#Since using the default mirror source is always unable to push successfully, here first set the source
vi /etc/docker/daemon.json

#Edit the following
{
	"registry-mirrors":["https://registry.docker-cn.com"]
}
Copy the code

#After finishing the above editing, restart docker
systemctl restart docker
Copy the code

#Log in to your Docker account, execute the following command, and then enter your account and password as prompted
docker login 

#Push the built image to the remote repositoryDocker push clfeng/CLF - cicd - centos: 1.0Copy the code

5. Test whether the image is available

Change the.gitlab-ci.yml file and change the first job to use the image we just pushed

Build_job ()
build_job:
  stage: build
  image: Clfeng/CLF - cicd - centos: 1.0 # Here is the key
  tags: 
    - clf-runner
  script:
    - echo "build project"
    - git --version
Copy the code

Add card controller logic

1. Write card control scripts

#! /bin/bashcd .. ; echo "cur dir: $(pwd)"; git diff HEAD "origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" ./yarn.lock | grep "element-ui"; if [ $? -eq "0" ]; Then echo "Update elmenent- UI library" then echo "Update Elmenent - UI library" ; fiCopy the code

Add analysis job

stages:
  - analyse
  - build
  - test
  - deploy

analyse_job:
  stage: analyse
  image: Clfeng/CLF - cicd - centos: 1.0
  tags: 
    - clf-runner
  only:
    - merge_requests
  script:
  	Add -x here to facilitate debugging
    - cd build && bash -x analyse.sh

build_job:
  stage: build
  image: Clfeng/CLF - cicd - centos: 1.0
  tags: 
    - clf-runner
  script:
    - echo "build project"
    - git --version

test_job:
  stage: test
  image: centos:7
  tags: 
    - clf-runner
  script:
    - echo "test project"

deploy_job:
  stage: deploy
  image: centos:7
  tags: 
    - clf-runner
  script:
    - echo "deploy project"
Copy the code

Note: it can be added before pushing related processing of card [email protected] into the project, then upgrade the Element-UI and push the card control logic

At this point we can see if there has been an update to the Eleme-UI library, but it’s not obvious from the CICD panel. Can we add a hint to the comments?

Of course, GitLab is very powerful. We can call GitLab API to realize the requirements mentioned above.

3. Add reminders to the comments section

In order to be able to call the GitLab API we need to add an access token for authentication on request.

The corresponding API found on the official website

#There are several parameters in there
#Id: indicates the ID of the project
#Merge request: ID
#The link https://docs.gitlab.com/ee/api/notes.html#create-new-merge-request-note
POST /projects/:id/merge_requests/:merge_request_iid/notes
Copy the code

Where do I look at these ids?

The project id

Merge request ID

Example Modify the card control script

#! /bin/bashcd .. ; echo "cur dir: $(pwd)"; git diff HEAD "origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME" ./yarn.lock | grep "element-ui"; if [ $? -eq "0" ]; Then curl -x POST \ -h "private-token: $ACCESS_TOKEN" \ -d "body=" $ACCESS_TOKEN" \ "http://192.168.56.101/api/v4/projects/$CI_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_ID/notes" fiCopy the code

The execution result

Here we go to implement the two requirements we mentioned earlier! But we also know from the previous learning that quality card control needs to rely on the preset variable CI_MERGE_REQUEST_TARGET_BRANCH_NAME of GitLab, but if the version of GitLab is too low, it may not be able to get this variable, so how should we deal with this problem?

Compatible with lower versions

Since the internal version of GitLab in my company is relatively low, I encountered a problem that I could not get the target branch through the CI_MERGE_REQUEST_TARGET_BRANCH_NAME variable.

To solve these problems, when the Node service receives a request, we send a request that triggers the pipeline and carries the corresponding variable.

Modifying node Services

const Koa = require('koa');
const route = require('koa-route');
const bodyParser = require('koa-bodyparser');
const axios = require('axios').default;

const app = new Koa();
const port = 3000;

app.use(bodyParser());

app.use(route.post('/merge_events'.async (ctx) => {

  const body = ctx.request.body;

  // Merge the requested URL
  const mergeReqUrl = body.object_attributes.url;

  // Assign the person's information, such as id, username, email
  const assignee = body.assignees[0];

  // With this information we can find the person assigned to review the merge request;
  dim(assignee.id, 'There is a merge request for your review:${mergeReqUrl}`);

  // Manually trigger pipeline related logic
  triggerPipeline(ctx);
  ctx.body = 'trigger success';
}));

// This function encapsulates the ability to call the chat API to notify the person with the corresponding ID
// The implementation depends on the internal chat tool
function dim (userId, message) {
  console.log(` [${userId}] :${message}`)}// Manually trigger pipeline related logic
function triggerPipeline(ctx) {
  const token = ctx.headers['x-gitlab-token']; // Secret token added when adding webhook
  const {
    project: {
      id / / project id
    },
    object_attributes: {
      action, // The specific behavior that triggers the merge request event
      source_branch, / / the source branch
      target_branch, // Target branch
      iid, // The iID of the merge request
    },
    repository: {
      homepage // Project address
    }
  } = ctx.request.body;
  
  const domain = (homepage.match(/^(http:\/\/[^/]+)\/*/))1];

  if (['open'.'reopen'.'update'].includes(action)) {
    const variables = {
      CLF_MERGE_REUEST: 'true'.// Use this scalar to identify the merge request pipeline that we trigger through the API when we merge requests
      CLF_MERGE_REQUEST_TARGET_BRANCH_NAME: target_branch,
      CLF_MERGE_REQUEST_ID: iid
    }
  
    const varStr = Object.entries(variables).map(([key, value]) = > {
      return `variables[${key}] =${value}`;
    }).join('&');
    
    // View the format of the relevant API from the Pipeline Triggers section
    / / http://192.168.56.101/api/v4/projects/2/ref/REF_NAME/trigger/pipeline? token=TOKEN&variables[RUN_NIGHTLY_BUILD]=true
    const triggerPipelineUrl = `${domain}/api/v4/projects/${id}/ref/${source_branch}/trigger/pipeline? token=${token}&${varStr}`;
  
    // Call API to trigger pipeline
    axios({
      url: triggerPipelineUrl,
      method: 'post',
    }).then(response= > {
      console.log(response);
      console.log('created pipeline success');
    }).catch((error) = > {
      console.log(error);
      console.log('created pipeline fail');
    });
  }

}

app.use(route.get('/'.ctx= > {
  ctx.body = 'Hello World';
}));

app.listen(port);
console.log(`Example app listening at http://localhost:${port}`);
Copy the code

Modify. Gitlab – ci. Yml

stages:
  - mytest
  - analyse

mytest_job:
  stage: mytest
  image: Clfeng/CLF - cicd - centos: 1.0
  tags: 
    - clf-runner
  only: $CLF_MERGE_REUEST is set to true when the job is triggered, i.e. the pipeline triggered by our Node service is executed
    variables:
      - $CLF_MERGE_REUEST = = "true"
  script: # check if we can get the job we want
    - echo "$CLF_MERGE_REUEST"
    - echo "$CLF_MERGE_REQUEST_TARGET_BRANCH_NAME"
    - echo "$CLF_MERGE_REQUEST_ID"

# comment out the job temporarily with a
.analyse_job:
  stage: analyse
  image: Clfeng/CLF - cicd - centos: 1.0
  tags: 
    - clf-runner
  only:
    variables:
      - $CLF_MERGE_REUEST = = "true"
  script:
    # - cd build && bash analyse.sh
    - echo "analyse_job"
Copy the code

The execution result

As you can see, this way we can access the target branch and carry as many values into the pipeline as we need.

conclusion

In this chapter, we realized the function of notifies corresponding reviewers to conduct code review in time when there is a merge request through Webhook. It also calls the GitLab API to add prompts to the comments section when developers upgrade the component library. Combined with the previous three articles, we have covered the most basic environment building and basic concepts to more advanced Webhook and GitLab API call. This is the end of CI/CD series articles, if there are worthy of sharing the actual combat will be shared, but may not be updated for a long time. I hope this series of articles has been helpful to you.

Learners’

Demand background

Create a local Node service

Add merge request hook

To generate the Secret token

Add the hook

Build GitLab

Install SSH and process firewalls

Add the GitLab package repository and install the packages

1. Add the GitLab software package repository

2. Install the software package and set the exposed address

3. Visit GitLab

4. Log in

Environment to prepare

Allows requests to be made to the local network from Web hooks and services

The project creates and adds SSH keys

Create a project

Create and add an SSH Key

Add webhook

Start the local service and make a breakpoint

Query merge request packets

Make a merge request

Viewing request Packets

Message analysis

Need to implement

Automatically alerts merge code

Tripartite library upgrade card control

Build the mirror

1. Write Dockerfile

2. Write a build script

3. Build an image

4. Push the image to the remote repository

5. Test whether the image is available

Add card controller logic

1. Write card control scripts

Add analysis job

3. Add reminders to the comments section

The project id

Merge request ID

Example Modify the card control script

The execution result

Compatible with lower versions

Modifying node Services

Modify. Gitlab – ci. Yml

The execution result

conclusion

Related Posts

Vultr starts at $2.50 for 15 rooms worldwide, but…

Some of the best cross-platform frameworks for audio and video

ShaderJoy — Introduction to GLSL Debugging techniques