Jenkins deploys the Python project into action

The background,

Jenkins is often used to deploy Java code in our work, because of its flexible plug-in features, such as JDK, Maven, Ant, etc., which enables Java projects to be compiled and deployed online without any delay. Similarly, for scripting language types such as Python, Jenkins’ powerful plug-in functions can be used to easily achieve CI/CD. However, if deploying multiple projects to the same server involves environmental consistency problems, Docker can be used to solve this problem, or Python virtual environment such as Virutalenv or Conda and other excellent tools can be used to solve this problem. In this case, it is slow to install dependency packages according to requirements. In the later stage, the whole Python environment needs to be packaged, and the conda tool is used to manage the project environment, which is convenient for rapid transplantation.

In this article, a detailed Django project is deployed, including using the Conda tool for Python multi-environment management, the Pylint tool for code inspection, and the Nose framework for unit testing and coverage.

Two, necessary knowledge

Jenkins base installation and deployment

Reference: blog.51cto.com/kaliarch/20…

Conda software package management system

IO /en/latest/m… conda. IO /en/latest/m… Basic commands:

Conda update conda conda upgrade --all 2. Conda install package_name You can specify the version conda install package_name=1.10 3. Remove the package conda remove package_name 4. Query the package conda list 5. Query the package conda search package_name 6. Source configuration Because anaconda's server is abroad, sometimes the speed will be slow, so you can change to a domestic source, such as TUNA of Tsinghua University. conda config --show-sources# Check the current source of useConda config --remove channels Source name or linkDelete the specified sourceConda config --add Channels source name or linkAdd the specified source

Add Anaconda TUNA image
conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
The mirrored address in the help of # TUNA is quoted, so it needs to be removed
 
Set the channel address to display when searchingConda config --set show_channel_urls yes Conda env list conda env list Conda create -n env_name list of packages eg:conda create -n py2 python=2.7 Request 3. Enter python :(Windows does not need to addsource)
source activate env_name
eg:sourceActivate Py2 4. Exit the environment:sourceDeactivate 5. View the installed package conda list -n python2 6. Conda create --name <new_env_name> --clone <copied_env_name> 7 Delete environment: conda remove -n go2cloud-api-env --all 8 When sharing code, you also need to share the running environment with others. First enter the environment and run the following command to save the package information of the current environment into the YAML file named environment.Copy the code

conda env export > environment.yaml

Similarly, when executing someone else’s code, you need to configure the environment accordingly. You can use the shared YAML files to create the same runtime environment

conda env create -f environment.yaml

The Shell foundation

Reference: myshell – note. Readthedocs. IO/en/latest/I…

GIT basis

Reference: blog.51cto.com/kaliarch/20…

Three, optimize

Generate the theme

Theme tools generate links: afonsof.com/jenkins-mat… Has been uploaded in topics like the color of the tool to generate logo to download the theme of the generated to Jenkins Jenkins server home directory, generally for installing the Jenkins system user’s home directory. Jenkins/userContent/material /, if there is no this directory to the new directory, CSS file move to the directory, for example, / root /. Jenkins userContent/material/blue CSS.

Configure the topic on Jenkins

a.Install Jenkins Simple Theme Plugin b. C. Click Configure System D. Find the Theme configuration e. cssurl fill in local themes, such as: http://localhost:8080/jenkins/userContent/material/blue.css f. lick the Save g. Save to view the effect.

Pay attention to

The URL of the local CSS can be opened by the browser for test access. If the access fails, Jenkins will be loaded by default. The default theme of the later migration is better to write localhost as the URL.

Fourth, deploy actual combat

4.1 Server List

The name of the	IP	software	note
Jenkins-server	10.57.61.138	miniconda	Jenkins server
Des-server	172.21.0.10	miniconda	Project deployment server

4.2 architecture diagram

4.3 Preparations

Installing dependency packages

Pylint: Python static code review, reference: pylint.pycqa.org/en/latest/u…
Mock: Used to generate test data.
Nose: Python unit test package.
Coverage: Python code coverage package.

Conda create -n python=3.6; conda env list is used to check the environment. To avoid contamination, install the PIP install PyLint Mock Nose Coverage software within the project environment using the PIP tool.

Install Jenkins plugins

JUnit: Allows junit-format test results to be published.
Cobertura Plugin: This Plugin integrates Cobertura coverage reports to Jenkins.
Violations plugin: This plugin does reports on Checkstyle, CSSLint, PMD, CPD, FXCop, PyLint, jcReport, FindBugs, And perlcritic violations), reference: a wiki. Jenkins. IO/display/JEN…
Git Plugin: This plugin allows GitLab to trigger Jenkins to build and display their results in the GitLab UI.
Git Parameter: Add ability to choose Branches, tags or Revisions from Git repositories configured in project.

4.4 Creating a Task

Create a free-style software project

New Task -> Build a free style software project, fill in the description, here because the later use of PyLint code check, give the message type of code check, can be fixed according to the message type.

Configure a parameterized build

Git branch is selected for configuration, and you can customize the port. Note that the Name of the parameterized build has changed, which will be used in the future.

Source code management

Gitlab has been certified. Note that Branches were selected in the parameterized build, and the variable $branch needed to be referenced in Branches to build.

Build configuration
- Execution of the shell

This shell is executed on Jenkins server, so it needs to configure Python virtual environment in advance and conduct code review on it. Nosetests. XML file is generated by unit test and coverage. XML file is generated by code coverage test. The PyLint test generates the pylint.xml file.

The following is a sample shell script for this project, which needs to be modified according to your actual situation, paying attention to the Python project structure and the identifiers that need to be checked by code.

base_dir=/root/.jenkins/workspace/
project=go2cloud-api-deploy-prod/
project_env=go2cloud-api-env
# Switch the Python environment
source activate ${project_env}
$(which python) -m pip install mock nose coverage
Update the Python environment
echo "+++ Update Python environment +++"

if [ -f ${base_dir}${project}requirements.txt ];then
    $(which python) -m pip install -r ${base_dir}${project}requirements.txt && echo0 | |echo 0
fi

# Code review/unit test/code test coverage
echo "+++ code check +++"
cd ${base_dir}
# pylint generated XML
$(which pylint) -f parseable --disable=C0103,E0401,C0302 $(find ${project}/* -name *.py) >${base_dir}pylint.xml || echo 0

echo "+++ + unit test +++"
# generate nosetests. XML
#$(which nosetests) --with-xunit --all-modules --traverse-namespace --with-coverage --cover-package=go2cloud-api-deploy-prod --cover-inclusive || echo 0
$(which nosetests) --with-xunit --all-modules --traverse-namespace --with-coverage --py3where=go2cloud-api-deploy-prod --cover-package=go2cloud-api-deploy-prod --cover-inclusive || echo 0
echo +++ code coverage +++"
# generate coverage. XML
Copy the code

python -m coverage xml –include=go2cloud-api-deploy-prod* || echo 0

Sends files and commands to the target server

#! /usr/bin/env bash
# current directory
BASEPATH=$(cd `dirname $0`;pwd)

# Python interpreter specific path
PYTHON_BIN=The $1

# mananger File path
MAIN_APP=$2

# python
SERVER_PORT=$3

[ $# -lt3] &&echo "Missing parameters" && exit 1

LOG_DIR=${BASEPATH}/logs/
[ ! -d ${LOG_DIR} ] && mkdir ${LOG_DIR}

OLD_PID=`netstat -lntup | awk -v SERVER_PORT=${SERVER_PORT} '{the if ($4 = = "0.0.0.0:" SERVER_PORT) print $NF}'|cut -d/ -f1`
[ -n "${OLD_PID}" ] && kill9 -${OLD_PID}

echo "-- -- -- -- -- -- -- -- --$0 $(date) excute----------" >> ${LOG_DIR}server-$(date +%F).log

# start service

nohup ${PYTHON_BIN} -u ${MAIN_APP}Runserver then executes 0.0.0.0:${SERVER_PORT}& > >${LOG_DIR}server-$(date +%F).log 2>&1 &
Copy the code

Post-build action
- The JUnit plug-in implements unit test reporting and needs to specify nosetests.xml

The Cobertura Plugin implements coverage testing

The Violations plugin performs code audits that require the generated Pylint.xml file on jenkins-server.

Note that the file path is Jenkins server pylint.xml and the encoding of the generated file.

Email Notification Configuration

Select the email Content as THE Content Type is HTML, so that you can write the email HTML template, to generate a better-looking email notification template. Notice You can select a trigger alarm type. The alarm is sent when the construction fails for several times or when the construction fails. You can configure the alarm as required.

<! DOCTYPE html> <html> <head> <meta charset="UTF-8">
<title>${ENV, var="JOB_NAME"}- the first${BUILD_NUMBER}The build log </title> </head> <body leftmargin="8" marginwidth="0" topmargin="8" marginheight="4"
    offset="0">
    <table width="95%" cellpadding="0" cellspacing="0"
        style="font-size: 11pt; font-family: Tahoma, Arial, Helvetica, sans-serif"> <tr> < TD >(this email is automatically issued by the program, do not reply!) </td> </tr> <tr> <td><h2> <font color="#0000FF"> < span style = "box-sizing: border-box! Important${BUILD_STATUS}</font>
                </h2></td>
        </tr>
        <tr>
            <td><br />
            <b><font color="#0B610B"</font></b> <hr size="2" width="100%" align="center"/></ tr> <tr> < TD >< ul> <li> Project name & NBSP; : & have spent${PROJECT_NAME}</li> <li> build number & NBSP; : & have spent The first${BUILD_NUMBER}</li> <li>SVN&nbsp; Version: & have spent${SVN_REVISION}</li> <li> trigger cause: &nbsp;${CAUSE}</li> <li> Build log: &nbsp; <a href="${BUILD_URL}console">${BUILD_URL}Console </a></li> <li> build &nbsp; &nbsp; Url&nbsp; : & have spent <a href="${BUILD_URL}">${BUILD_URL}</a></li> <li> Working directory & NBSP; : & have spent <a href="${PROJECT_URL}ws">${PROJECT_URL}Ws </ A ></ LI >< LI > project & NBSP; &nbsp; Url&nbsp; : & have spent <a href="${PROJECT_URL}">${PROJECT_URL}</a></li>
                </ul>
            </td>
        </tr>
        <tr>
            <td><b><font color="#0B610B">Changes Since Last
                        Successful Build:</font></b>
            <hr size="2" width="100%" align="center"/ > < / td > < / tr > < tr > < td > < ul > < li > history change record: < a href ="${PROJECT_URL}changes">${PROJECT_URL}changes</a></li>
                </ul> ${CHANGES_SINCE_LAST_SUCCESS,reverse=true, format="Changes for Build #%n:

%c

",showPaths=true,changesFormat="[%a]

%m
",pathFormat="        %p"}

            </td>

        </tr>

        <tr>

            <td><b>Failed Test Results</b>

            <hr size="2" width="100%" align="center" /></td>

        </tr>

        <tr>

            <td><pre

                    style="font-size: 11pt; font-family: Tahoma, Arial, Helvetica, sans-serif">$FAILED_TESTS</pre>

                <br /></td>

        </tr>

        <tr>

            <td><b><font color="#0B610B"</font></b> <hr size="2" width="100%" align="center"/></td> </tr> <! -- <tr> <td>Test Logs (if test has ran): <a

                href="${PROJECT_URL}ws/TestResult/archive_logs/Log-Build-${BUILD_NUMBER}.zip">${PROJECT_URL}/ws/TestResult/archive_logs/Log-Build-${BUILD_NUMBER}.zip</a>

                <br />

            <br />

            </td>

        </tr> -->

        <tr>

            <td><textarea cols="80" rows="30" readonly="readonly"

                    style="font-family: Courier New">${BUILD_LOG, maxLines=100}</textarea>

            </td>

        </tr>

    </table>

</body>

</html>

Copy the code 
4.5 Build tests

Trigger a build

Select the corresponding branch and port

    


Check the consle log

Pylint check




    


Nosetests unit tests and code coverage


    

    


The target server views the specific project


    

4.6 Viewing Construction Results

The overview to see


    


View code coverage


    

(C) convention. Violates the Coding Style Standard (R) refactor refactoring. Very poorly written code. (W) Warning Warning. Some Python-specific problems. (E) Error. It's probably a bug in the code. (F) Fatal error. An error that prevents Pylint from running further.Copy the code


    


Check email


    

5. Pipeline deployment
5.1 Basic concepts of Pipeline

What is the pipeline

Pipeline is a workflow framework running on Jenkins. The configuration information in the project is put into a script in the form of steps to connect tasks that originally run independently on a single or multiple nodes, so as to realize complex process choreography and visualization that is difficult to be completed by a single task.

Basic concept

Stages: Tell Jenkins what to do. A Pipeline can be divided into stages, each representing a set of operations. Note that Stage is a logical grouping concept that can span multiple nodes.
A Node is a Jenkins Node, either Master or slave, which is the specific runtime environment for executing a Step.
Step: Step: detailed construction operation of each Step. Step is the most basic operation unit, ranging from creating a directory to building a Docker image, which is provided by various Jenkins Plugins.


Language hair tools

Pipeline provides a set of extensible tools for Pipeline as Code (Jenkinsfile stored in the project’s source Code base) through Pipeline Domain Specific Language (DSL) syntax.
Reference: wiki. Jenkins. IO/display/JEN…
5.2 Characteristics of pipeline
Based on Jenkins Pipeline, users can quickly implement the complete process of a project from build, test to release in a JenkinsFile, and save the definition of the Pipeline.

Code: Pipelines are implemented as code, usually checked into source control, enabling the team to edit, review, and iterate over their CD process.
Sustainability: Pipeline jobs are not affected by Jenklins restarts or outages.
Pause: A Pipeline can choose to stop and wait for worker input or approval before resuming Pipeline operation.
Multifunctional: Pipelines support complex real-world CD requirements, including fork/join subprocesses, the ability to loop and execute work in parallel
Extensible: The Pipeline plug-in supports custom extensions to its DSL and multiple options for integration with other plug-ins.

5.3 pipeline grammar

declarative

pipeline {
/* insert Declarative Pipeline here */
}
Copy the code
Basic statements and expressions that are valid in declarative pipelining follow the same rules as Groovy’s syntax, with the following exceptions:

The top of the assembly line has to be oneblock, in particular: pipeline {}
Without semicolons as statement separators, each statement must be on its own line.
A block can consist only of segments, instructions, steps, or assignment statements. * Attribute reference statements are treated as no-parameter method calls. For example, input is treated as input()

Example:
Jenkinsfile (Declarative Pipeline)
pipeline {
    agent none 
    stages {
        stage('Example Build') {
            agent { docker 'maven:3-alpine' } 
            steps {
                echo 'Hello, Maven'
                sh 'mvn --version'
            }
        }
        stage('Example Test') {
            agent { docker 'openjdk:8-jre' } 
            steps {
                echo 'Hello, JDK'
                sh 'java -version'}}}}Copy the code
Jenkins. IO /zh/doc/book…

The script type

A scripted pipeline, like [declarative], is built on a subsystem of the underlying pipeline. Unlike declarative, the scripted pipeline is actually a generic DSL built by Groovy [2]. Most of the functionality provided by the Groovy language is available to users of scripted pipelining. This means that it is a very expressive and flexible tool through which to write continuous delivery pipelines.
Example:
node('master') {// the master node runs. The following stages can also be specified'Prepare'// Clear the publication directory bat' ''if exist D:\\publish\\LoginServiceCore (rd/s/q D:\\publish\\LoginServiceCore) if exist C:\\Users\\Administrator\\.nuget  (rd/s/q C:\\Users\\Administrator\\.nuget) exit'' '// Pull git repository stage'Checkout'
        checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], &emsp; &emsp; &emsp; &emsp; &emsp; &emsp; &emsp; submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'c6d98bbd-5cfb-4e26-aa56-f70b054b350d', 
            url: 'http://xxx/xxx/xxx'[]]) // Build stage'Build'
        bat ' ''cd "D:\\Program Files (x86)\\Jenkins\\workspace\\LoginServiceCore\\LoginApi.Hosting.Web" dotnet restore dotnet build dotnet publish --configuration Release --output D:\\publish\\LoginServiceCore'' '/ / deployment stage'Deploy'
        bat ' '' cd D:\\PipelineScript\\LoginServiceCore python LoginServiceCore.py '' '// Automatic test (python code implementation) stage'Test'
        bat' '' cd D:\\PipelineScript\\LoginServiceCore python LoginServiceCoreApitest.py '' '   
}
Copy the code
5.4 the sample
The above project is published here using pipeline

Create a free-style software project

New Task -> Pipeline, fill in the task description.

    


    


Pipeline

If you are not familiar with Pipeline syntax, tool generation can be used

    

pipeline {
  agent any
  parameters {
    gitParameter branchFilter: 'origin/(.*)', defaultValue: 'master', name: 'BRANCH'.type: 'PT_BRANCH'
  }
  stages {
    stage('checkout src code') {
        steps {
            echo "checkout src code"
            git branch: "${params.BRANCH}".'http://123.206.xxx.xxx/xuel/go2cloud_platform.git'
        }
    }
    stage('exec shell'){
        steps{
            echo "pylint,Unit test"
            sh ' ''# Jenkins server project workspace directory base_dir = / root /. Jenkins # / workspace/project name project = go2cloud_platform # python environment project Project_env = go2cloud_platform_pipeline switch # python environment source/data/miniconda3 / bin/activate ${project_env} $(which python) -m PIP install Mock nose coverage pylint # Update python environment echo "++++++ update Python environment +++" if [-f ${base_dir}${project}requirements/requirements.txt ]; then $(which python) -m pip install -r ${base_dir}${project}/requirements/requirements.txt && echo 0 || echo 0 fi # Echo "++ Code check/unit test/code test coverage ++" CD ${base_dir} # Generate pylint. XML $(which PyLint) -f parseable --disable=C0103,E0401,C0302 $(find ${project} / * - name *. Py) > ${base_dir} ${project} _pylint. XML | | echo echo 0 # "+ + + + unit testing" generate nosetests. XML # $(which nosetests) --with-xunit --all-modules --traverse-namespace --with-coverage --cover-package=go2cloud-api-deploy-prod --cover-inclusive || echo 0 $(which nosetests) --with-xunit --all-modules --traverse-namespace --with-coverage - py3where = ${project} - cover - package = ${project} - cover - inclusive | | echo echo 0 # "+ + + + + code coverage" generate coverage. The XML python -m coverage xml --include=${project_env}* || echo 0'' '

        }
    }
    stage("deploy") {
        steps {
            echo "send file"
            sshPublisher(publishers: [sshPublisherDesc(configName: 'go2cloud_platform_host', transfers: [sshTransfer(cleanRemote: false, excludes: ' '.execCommand: ' ''base_dir=/devops_pipeline project_src=go2cloud_platform Project_env = / data/miniconda3 / envs/go2cloud_platform_pipeline/bin/python echo "+ + + + + + + update deployment server python environment + + + + + + + + + +" if [-f ${base_dir}/requirements/requirements.txt ]; then ${project_env} -m pip install -r ${base_dir}/requirements/requirements.txt && echo 0 || echo 0 fi echo -e "\\033[32m Startup service script \\033[0m" $(which bash) ${base_dir}/run_server.sh ${project_env} ${base_dir}/apps/manage.py ${port} '' '.execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[and] +', remoteDirectory: '/devops_pipeline', remoteDirectorySDF: false, removePrefix: ' '.sourceFiles: '* * / * *')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false) [)}}}}Copy the code

Running the build


    


View the results


      

      


Six, pay attention to the main points

Be sure to sort out the process before deployment and understand which steps are performed on the target server and Jenkins server
The directory and Python virtual environment must be properly mapped to avoid environment pollution

Seven, to reflect on

In this use to Conda virtual environment management, to solve the environment on the same server inconsistent line, can also use Docker to solve
Using Pipeline project release visualization, clear stage, convenient stage view and troubleshooting, Jenkinsfile can be placed in git repository