I. Background introduction

In the context of rapid update and iteration of Internet applications, traditional manual or simple scripts can no longer adapt to this change. At this time, Devops provides us with a good solution. The good application of CI/CD can greatly facilitate our daily work. Automated rapid continuous integration/continuous delivery gives us faster application opening, better stability and greater reliability.

2. Topology environment

2.1 Architecture Topology

As shown in the example above, the process topology is simply spent:

  • Webhook automatically triggers Jenkins to build the application after the developer pushes native code into GitLab-Server
  • Git Clone from GitlabServer source code, and start the application
  • The front end can be placed lb for high availability
  • Database Connects to the cloud database
  • Logs can be stored later in the log and posted to elK for log visualization
  • Build completion email notification to stakeholders (test or open)

2.2 System Software Version

The name of the version
Linux system CentOS7.3 64
Docker 1.13
Django 2.0

Three, installation and deployment

3.1 Jenkins Installation and deployment

For Jenkins installation and deployment, please refer to Jenkins Notes

  • Add the Docker target server after installation

  • Configure the mail sending server

3.2 Docker Installation and deployment

Docker installation and deployment and Dockerfile preparation can refer to: container Docker details

3.3 Gitlab installation and deployment

GitLab is installed on the Linux server on the public network. If the public network is not available, you need to manually modify the external_url ‘http://own Intranet IP’ file in /etc/gitlab/gitlab.rb.

yum install -y libsemanage-static libsemanage-devel policycoreutils openss
h-server openssh-clients postfix
systemctl enablePostfix && systemctl start postfix wget HTTP: / / https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-8.0. 0 - ce. 0. El7. X86_64. RPM RPM -i gitlab - ce - 8.0.0 - ce. 0. El7. X86_64. RPMObtain the public IP address
PUBLICIP=$(curl http://ipv4.icanhazip.com)
# modified
sed -i "s/gitlab-server/${PUBLICIP}/g" /etc/gitlab/gitlab.rb

gitlab-ctl reconfigure
gitlab-ctl restart

echo "Username:root"
echo "Password:5iveL! fe"
Copy the code

3.4 Configuring the Release Process

  • Jenkins is building a free style software project

  • Parametric builds facilitate subsequent deployment of source ports and releases of Docker incoming mappings

  • The source code comes from the Django project at GitLab

  • Webhook gitlab and Jenkins

Jenkins install plugins:

  • When the developer automatically triggers the Jenkins construction project after the local push code, there is the Git pull code written in the Dockerfile, so Jenkins does not need to distribute the code to the docker host again, and Jenkins is used as the trigger docker construction

  • Configure mail after the build is complete

Build Notification :${BUILD_STATUS} – ${PROJECT_NAME} – Build # ${BUILD_NUMBER}! Building notification templates:

<! DOCTYPE html> <html> <head> <meta charset="UTF-8">
<title>${ENV, var="JOB_NAME"}- the first${BUILD_NUMBER}The build log </title> </head> <body leftmargin="8" marginwidth="0" topmargin="8" marginheight="4"
    offset="0">
    <table width="95%" cellpadding="0" cellspacing="0"
        style="font-size: 11pt; font-family: Tahoma, Arial, Helvetica, sans-serif"> <tr> < TD >(this email is automatically issued by the program, do not reply!) </td> </tr> <tr> <td><h2> <font color="#0000FF"> < span style = "box-sizing: border-box! Important${BUILD_STATUS}</font>
                </h2></td>
        </tr>
        <tr>
            <td><br />
            <b><font color="#0B610B"</font></b> <hr size="2" width="100%" align="center"/></ tr> <tr> < TD >< ul> <li>${PROJECT_NAME}</li> <li${BUILD_NUMBER}</li> <li>SVN version:${SVN_REVISION}</li> <li>${CAUSE}</li> <li> Build log: <a href="${BUILD_URL}console">${BUILD_URL}Console </a></li> <li> build Url: <a href="${BUILD_URL}">${BUILD_URL}</a></li> <li"${PROJECT_URL}ws">${PROJECT_URL}Ws </a></li> <li> Project Url: <a href="${PROJECT_URL}">${PROJECT_URL}</a></li>
                </ul>
            </td>
        </tr>
        <tr>
            <td><b><font color="#0B610B">Changes Since Last
                        Successful Build:</font></b>
            <hr size="2" width="100%" align="center"/ > < / td > < / tr > < tr > < td > < ul > < li > history change record: < a href ="${PROJECT_URL}changes">${PROJECT_URL}changes</a></li>
                </ul> ${CHANGES_SINCE_LAST_SUCCESS,reverse=true, format="Changes for Build #%n:

%c

",showPaths=true,changesFormat="
[%a]

%m


",pathFormat=" %p"}

            </td>

        </tr>

        <tr>

            <td><b>Failed Test Results</b>

            <hr size="2" width="100%" align="center" /></td>

        </tr>

        <tr>

            <td><pre

                    style="font-size: 11pt; font-family: Tahoma, Arial, Helvetica, sans-serif">$FAILED_TESTS</pre>

                <br /></td>

        </tr>

        <tr>

            <td><b><font color="#0B610B"</font></b> <hr size="2" width="100%" align="center"/></td> </tr> <! -- <tr> <td>Test Logs (if test has ran): <a

                href="${PROJECT_URL}ws/TestResult/archive_logs/Log-Build-${BUILD_NUMBER}.zip">${PROJECT_URL}/ws/TestResult/archive_logs/Log-Build-${BUILD_NUMBER}.zip</a>

                <br />

            <br />

            </td>

        </tr> -->

        <tr>

            <td><textarea cols="80" rows="30" readonly="readonly"

                    style="font-family: Courier New">${BUILD_LOG, maxLines=100}</textarea>

            </td>

        </tr>

    </table>

</body>

</html>

Copy the code 


The trigger type can be set as required. Set always to send emails regardless of success or failure



    




    

  • View files inside the remote Docker server
    • 




    




    

  • Django deployments have made their own Docker images using conda packaged with the project’s Python3.6 environment package
    

    
      
    

    • 



Before using a pure Python3.6 system, PIP was used to install the requirements. TXT module during each build. However, in the long run, since the environment changes rarely, each PIP installation takes time, so conda was used to make the packaged Python environment into a customized environment. In order to reduce the deployment time of the environment, you can also mount the local disk in the environment through the -v parameter of Docker image production, and build the local Conda each time to complete the rapid deployment of the environment.


Check the Dockerfile


FROM 87a69025db6a
MAINTAINER kaliarch

# define the working directory in docker
ENV WORK_DIR /work/
Create a working directory inside the docker
RUN mkdir $WORK_DIR
Define the mapping port

EXPOSE 80

WORKDIR $WORK_DIR 
RUN git clone http://123.xxxx.xxxxx.245/Devops/go2cloud.git

Add a startup service script
ADD *.sh ${WORK_DIR}

CMD `which bash` /work/start_all.sh && tail -f /work/logs/server-$(date +%F).log
Copy the code


Check out the Django startup script


#! /bin/bash
BASEPATH=$(cd `dirname $0`;pwd)

PY_CMD=/python3/bin/python


Service entry file
#MAIN_APP=${BASEPATH}/go2cloud/manage.py 
Migrate script entry files
SCRIPTS_APP=${BASEPATH}/go2cloud/scripts/migrate_task_schdule.py
# delete script entry file
DELETE_APP=${BASEPATH}/go2cloud/scripts/delete_transfer_server.py


# log directory

LOG_DIR=${BASEPATH}/logs/
[ ! -d ${LOG_DIR} ] && mkdir ${LOG_DIR}

# start service
# nohup ${PY_CMD} - u ${MAIN_APP} runserver then executes 0.0.0.0:80 > > ${LOG_DIR} server - $(date + % F.) log > & 1 & 2
Start script migration scheduling script
echo "-- -- -- -- -- -- -- -- --$0 $(date) excute----------" >> ${LOG_DIR}task-script-$(date +%F).log
nohup ${PY_CMD} -u ${SCRIPTS_APP} >> ${LOG_DIR}script-$(date +%F).log 2>&1 &

Start migration delete script
echo "-- -- -- -- -- -- -- -- --$0 $(date) excute----------" >> ${LOG_DIR}delete-script-$(date +%F).log
nohup ${PY_CMD} -u ${DELETE_APP} >> ${LOG_DIR}delete-script-$(date +%F).log 2>&1 &

Copy the code


Check out the Jenkins deployment script


#! /bin/bash
release=The $1
port=$2

BASEPATH=$(cd `dirname $0`;pwd)

# Build go2cloud-platform image
cd /dockerwork
docker build -t go2cloud-platform-mini:$release .

IMGNAME=$(docker images|awk -v release=$release '{if($1=="go2cloud-platform-mini" && $2==release) print $3}')

echo $IMGNAME
# start container
docker run -d -p ${port}:80 -v /testlog/:/work/logs ${IMGNAME}

Copy the code


Log persistence is stored on the Docker host using the -v parameter


Iv. Test and display


4.1 Test Construction


Build tests manually



    




4.2 check the log





    



    



    




4.3 Viewing the Docker Container





    




4.4 test the app





    




Fifth, reflection and improvement


    

  • At present, the database is connected to the database built for cloud server, and the later database also uses Docker, and multiple groups use Docker-compose for unified deployment and management
    • 

  • Later, public cloud K8S cluster can be used for convenient testing
    • 

  • At present, the logs generated by the Docker container are stored on the Docker host, which can be stored on COS in the later stage and then delivered to elK cluster for log visualization processing
    • 

  • Manage images in a unified manner and create a local image repository
    • 

  • Gitlab adds code Review and combines it with automated testing
    •