An overview of the
Continuous integration is a software development practice in which team development members frequently integrate their work, meaning that by each member integrating at least once a day, multiple integrations may occur per day. Each integration is verified by an automated build (including compilation, release, and automated testing) to catch integration errors early.
Continuous Deployment is the rapid delivery of high-quality products through an automated build, test, and deployment cycle. To some extent, it represents the degree of engineering of a development team. After all, the labor cost of a fast-running Internet company is higher than that of machines. Investment in machines to optimize development process can also improve people’s efficiency and maximize engineering productivity.
1. Environment preparation
This experiment is based on Centos 7.3 and Docker 17.032-CE. Get docker CE for CentOS Get Docker CE for CentOS
1.1. Docker starts Gitlab
The startup command is as follows:
docker run --detach \
--hostname gitlab.chain.cn \
--publish 8443:443 --publish 8080:80 --publish 2222:22 \
--name gitlab \
--restart always \
--volume /Users/zhangzc/gitlab/config:/etc/gitlab \
--volume /Users/zhangzc/gitlab/logs:/var/log/gitlab \
--volume /Users/zhangzc/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce
Copy the codePort, hostname, and Volume can be set as required
1.2. Docker starts Gitlab-runner
The startup command is as follows:
sudo docker run -d /
--name gitlab-runner /
--restart always /
-v /Users/zhangzc/gitlab-runner/config:/etc/gitlab-runner /
-v /Users/zhangzc/gitlab-runner/run/docker.sock:/var/run/docker.sock /
gitlab/gitlab-runner:latest
Copy the codeVolume Set this parameter based on site requirements
1.3. Image creation for integration deployment
Our integration and deployment needs to be in a container, so we need to make an image and install the necessary tools for integration and deployment operations. So far our projects are based on Golang 1.9.2, so this is a specific image based on golang:1.9.2.
Dockerfile contains the following contents:
# Base image: https://hub.docker.com/_/golang/The FROM golang: 1.9.2 USER root# Install golint
ENV GOPATH /go
ENV PATH ${GOPATH}/bin:$PATH
RUN mkdir -p /go/src/golang.org/x
RUN mkdir -p /go/src/github.com/golang
COPY source/golang.org /go/src/golang.org/x/
COPY source/github.com /go/src/github.com/golang/
RUN go install github.com/golang/lint/golint
# install docker
RUN curl -O https://get.docker.com/builds/Linux/x86_64/docker-latest.tgz \
&& tar zxvf docker-latest.tgz \
&& cp docker/docker /usr/local/bin/ \
&& rm -rf docker docker-latest.tgz
# install expect
RUN apt-get update
RUN apt-get -y install tcl tk expect
Copy the codeGolint is a tool for checking the golang code style. Docker is required to use the host docker command in the container, so it is necessary to install a Docker executable file, and then mount the host /var/run/docker.sock file to the same location in the container when the container is started. Expect is a tool used to automatically log in to remote servers using SSH. This tool is installed to enable application deployment on remote servers
In addition, when installing Golint, you need to go to Golang.org to download the source code. Because of the wall, the go Get command cannot be executed. In order to deal with this problem, first download the relevant source code through other channels, put it in the specified path, and then copy it to the image, and install it.
The following script is used to generate the image:
#! /bin/bash
echo "Extract files needed to build the image"
source_path="source"
mkdir -p $source_path/golang.org
mkdir -p $source_path/github.com
cp -rf $GOPATH/src/golang.org/x/lint $source_path/golang.org/
cp -rf $GOPATH/src/golang.org/x/tools $source_path/golang.org/
cp -rf $GOPATH/src/github.com/golang/lint $source_path/github.com
echo "Build a mirror"Docker build-t go-tools:1.9.2.echo "Delete files needed to build the image"
rm -rf $source_path
Copy the codeAfter the image is generated, it is pushed to the image repository and pulled from the server of Gitlab-Runner
Gitlab and Gitlab-Runner in this test run under docker of the same server.
2. Runner registration and configuration
Registered 2.1.
When the environment is ready, run the following command on the server to register runner:
docker exec -it gitlab-runner gitlab-ci-multi-runner register
Copy the codeEnter the relevant information as prompted
Please enter the gitlab-ci coordinator URL:
# gitlab url, such as https://gitlab.chain.cn/
Please enter the gitlab-ci token for this runner:
# gitlab-> Your project -> Settings -> CI/CD ->Runners Settings
Please enter the gitlab-ci description for this runner:
# example: demo-test
Please enter the gitlab-ci tags for this runner (comma separated):
# Example: demo
Whether to run untagged builds [true/false] :# true
Please enter the executor: docker, parallels, shell, kubernetes, docker-ssh, ssh, virtualbox, docker+machine, docker-ssh+machine:
# docker
Please enter the default Docker image (e.g. ruby:2.1):
# go-Tools :1.9.2 (previously made by myself)
Copy the code
Once successful, you can see the following at the bottom of the Gitlab -> Your project -> Settings -> CI/CD ->Runners Settings page:
2.2. The configuration
After the registration is successful, you need to perform some specific configurations on the original configuration, as follows:
[[runners]]
name = "demo-test"
url = "https://gitlab.chain.cn/"
token = "c771fc5feb1734a9d4df4c8108cd4e"
executor = "docker"
[runners.docker]
tls_verify = false
image = "Go - tools: 1.9.2."
privileged = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
extra_hosts = ["Gitlab. Chain. Cn: 127.0.0.1."]
network_mode = "host"
pull_policy = "if-not-present"
shm_size = 0
[runners.cache]
Copy the codeGitlab-runner starts a container according to the configuration above, namely go-tools:1.9.2, b in which all the startup parameters are configured under [runners. Docker]. Including mounting, networking, and so on. After the container is successfully started, the container will be used to pull the code on GitLab, and then it will be checked according to the rules defined by itself. After all the tests are successful, it will be deployed.
Volumes: To run the host docker command in the container.
Extra_hosts: Add a host map to Gitlab to map to 127.0.0.1
Network_mode: Make the container’s network consistent with that of the host so that gitLab can be accessed through 127.0.0.1.
Pull_policy: If the specified image does not exist, it will be pulled by docker pull
3. Define rules
In gitlab project root directory to create. Gitlab – ci. Yml file, fill in the runner rules, specific grammar class official documents: docs.gitlab.com/ee/ci/yaml/
3.1. Go Integration command
Here are a few common golang integration commands
- Package list As described in the official documentation, the GO project is a collection of packages. Most of the tools described below will use these packages, so the first command we need is the methods that list the packages. We can do this with the go list subcommand
go list ./...
Copy the codeBe aware if we want to avoid applying our tools to external resources and limit it to our code. So we need to remove the vendor directory, the command is as follows:
go list ./... | grep -v /vendor/
Copy the code- Unit tests These are the most common tests you can run in your code. Each.go file needs one that can support unit testing
_test.goFile. You can run tests for all packages using the following command:
go test -short $(go list ./... | grep -v /vendor/)
Copy the code- Data contention this is often a problem that is hard to avoid solving, and the GO tool has it by default (but only on Linux/AMD64, FreeBSD/AMD64, Darwin/AMd64, and Windows/AMD64)
go test-race-short $(go list. /... | grep - v /vendor/)Copy the code- Code coverage this is an essential tool for assessing the quality of code and showing which parts of code are unit-tested and which are not. To calculate code coverage, run the following script:
PKG_LIST=$(go list ./... | grep -v /vendor/)
for package in ${PKG_LIST}; do
go test -covermode=count -coverprofile "cover/${package##*/}.cov" "$package" ;
done
tail -q -n +2 cover/*.cov >> cover/coverage.cov
go tool cover -func=cover/coverage.cov
Copy the codeIf we want to get coverage reports in HTML format, we need to add the following command:
go tool cover -html=cover/coverage.cov -o coverage.html
Copy the code- Finally, once the code has been fully tested, we compile the code to build executable binaries.
go build .
Copy the code- Linter This is the first tool we used in our code: Linter. It checks for code style/errors. This may sound like an optional tool, or at least a “nice” one, but it does help maintain a consistent code style across projects. Linter is not part of Go itself, so you’ll need to install it manually to use it (we’ve already installed the previous Go-Tools image). It’s fairly simple to use: just run it on the code package (you can also point to a.go file):
$ golint -set_exit_status $(go list ./... | grep -v /vendor/)
Copy the codeNote the -set_exit_status option. By default, golint only prints style problems with a return value (with a zero return code), so CI does not consider it an error. If -set_exit_status is specified, golint’s return code will not be 0 in case of any style problems.
3.2. The Makefile
If we don’t want to write too complicated in the.gitlab-ci.yml file, we can pack all the tools we use in the continuous integration environment into a Makefile and call them in a uniform way.
In this case, the.gitlab-ci.yml file will be more concise. Of course, makefiles can also call *.sh script files
3.3. Configuration Example
3.3.1.. gitlab – ci. Yml
Image: Go-tools :1.9.2 stages: - build -test
- deploy
before_script:
- mkdir -p /go/src/gitlab.chain.cn/ZhangZhongcheng /go/src/_/builds
- cp -r $CI_PROJECT_DIR /go/src/gitlab.chain.cn/ZhangZhongcheng/demo
- ln -s /go/src/gitlab.chain.cn/ZhangZhongcheng /go/src/_/builds/ZhangZhongcheng
- cd /go/src/_/builds/ZhangZhongcheng/demo
unit_tests:
stage: test
script:
- make test
tags:
- demo
race_detector:
stage: test
script:
- make race
tags:
- demo
code_coverage:
stage: test
script:
- make coverage
tags:
- demo
code_coverage_report:
stage: test
script:
- make coverhtml
only:
- master
tags:
- demo
lint_code:
stage: test
script:
- make lint
build:
stage: build
script:
- pwd
- go build .
tags:
- demo
build_image:
stage: deploy
script:
- make build_image
tags:
- demo
Copy the code3.3.2 rainfall distribution on 10-12. Makefile
PROJECT_NAME := "demo"
PKG := "gitlab.chain.cn/ZhangZhongcheng/$(PROJECT_NAME)"
PKG_LIST := $(shell go list ./... | grep -v /vendor/)
GO_FILES := $(shell find . -name '*.go' | grep -v /vendor/ | grep -v _test.go)
test: ## Run unittests
@go test -v ${PKG_LIST}
lint: ## Lint the files
@golint ${PKG_LIST}
race: ## Run data race detector
@go test -race -short ${PKG_LIST}
coverage: ## Generate global code coverage report
./scripts/coverage.sh;
coverhtml: ## Generate global code coverage report in HTML
./scripts/coverage.sh html;
build_image:
./scripts/buildDockerImage.sh
Copy the code3.3.3. coverage.sh
#! /bin/bash
#
# Code coverage generation
COVERAGE_DIR="${COVERAGE_DIR:-coverage}"
PKG_LIST=$(go list ./... | grep -v /vendor/)
# Create the coverage files directory
mkdir -p "$COVERAGE_DIR";
# Create a coverage file for each package
for package in ${PKG_LIST}; do
go test -covermode=count -coverprofile "${COVERAGE_DIR}/${package##*/}.cov" "$package" ;
done ;
# Merge the coverage profile files
echo 'mode: count' > "${COVERAGE_DIR}"/coverage.cov ;
tail -q -n +2 "${COVERAGE_DIR}"/*.cov >> "${COVERAGE_DIR}"/coverage.cov ;
# Display the global code coverage
go tool cover -func="${COVERAGE_DIR}"/coverage.cov ;
# If needed, generate HTML report
if [ "The $1"= ="html" ]; then
go tool cover -html="${COVERAGE_DIR}"/coverage.cov -o coverage.html ;
fi
# Remove the coverage files directory
rm -rf "$COVERAGE_DIR";
Copy the code3.3.4. buildDockerImage.sh
#! /bin/bashCheck GOPATH echo"Testing GOPATH"
if [ -z "$GOPATH"]; then echo"GOPATH not set"
exit 1
else
echo "GOPATH=$GOPATH"Fi # Initializes data echo"Initialize data"
new_version="1.0.0"
old_version="1.0.0"
golang_version="1.9.2"
app_name="application"
projust_root="demo"
DOCKER_IMAGE_NAME="demo"
REGISTRY_HOST="xxx.xxx.xxx.xxx:5000"
path="/go/src/_/builds/ZhangZhongcheng/demo"Replace the current container with the old tag echo"Replace current container with old label"Docker rmI $REGISTRY_HOST/$DOCKER_IMAGE_NAME:$old_version1.9.2 Mirror started container instance to compile the binary executable program echo of this project"Compiling the binary executable of this project based on golang:1.9.2 Mirror-started container instance"
cd $path
go build -o $app_name
echo "Check $app_name application"
FILE="$path/$app_name"
if [ -f "$FILE"]; then echo"$FILE ready"
else
echo $FILE application does not exist
exit 1CD $path/scripts echo CD $path/scripts echo CD $path/scripts echo"Extract files needed for build time"cp .. /$app_name $app_name # build echo based on Dockerfile in current directory"Build image based on Dockerfile in current directory"
echo "docker build -t $REGISTRY_HOST/$DOCKER_IMAGE_NAME:$new_version ."Docker build -t $REGISTRY_HOST/$DOCKER_IMAGE_NAME:$new_version"Delete the generated executable and the files required for the build"rm -rf $app_name rm -rf .. /$app_name # check mirror echo"View mirror"Docker images | grep $# DOCKER_IMAGE_NAME push mirror echo"Push image"
echo "docker push $REGISTRY_HOST/$DOCKER_IMAGE_NAME:$new_version"
docker push $REGISTRY_HOST/$DOCKER_IMAGE_NAME:$new_version
echo "auto deploy"
./automationDeployment.sh $new_version $old_version
Copy the code3.3.5. automationDeployment.sh
#! /usr/bin/expect# set shebang = shebang3Seconds set IP xxx.xxx.xxx. XXX set password"xxxxxxx"
set new_version [lindex $argv 0]
set old_version [lindex $argv 1]
spawn ssh root@$ip
expect {
"*yes/no" { send "yes\r"; exp_continue}
"*password:" { send "$password\r" }
}
expect "# *"
send "cd /root/demo/\r"
send "./docker_run_demo.sh $new_version $old_version\r"
expect eof
Copy the code3.3.6. Dockerfile
FROM golang:1.9Alpine #ENV TIME_ZONE Asia/Shanghai ADD application /go/ SRC /demo/ WORKDIR /go/ SRC /demo ADD run_application.sh /root/ RUN chmod755 /root/run_application.sh
CMD sh /root/run_application.sh
EXPOSE 8080
Copy the code3.3.7. Run_application. Sh
#! /bin/bash# mapping IP cp/usr/share/zoneinfo/Asia/Shanghai/etc/localtime CD/go/SRC/demo /. / applicationCopy the codeResults 4.
The following is a screenshot after the deployment is successful:
After the
Golang is based on a Gitlab CI/CD deployment solution