An overview of
Code to the project quality management, in the current service/modular/fast iterative agile development if only rely on simple IDE check and man-made codereview for a lot of code is not very suitable for, not just rely on the developer’s coding standard coding and pay attention to the application robustness, also need some tools to help us advance prevention and mandatory testing specification.
Sonarqube is a code analysis and inspection tool that integrates with DevOps, such as gitLab CI/CD or Jenkins, to deploy automatic code inspection, find and handle bugs in time, and minimize bugs and irregularities during the coding phase. Sonarqube has a number of internal analysis tools. For example, PMD-CPD, CheckStyle, FindBugs, Jenkins help us manage source code quality in seven ways. This article installs the latest version of Sonarqube-7.9.1. This version does not support custom MySQL database, and the JDK requires higher version 11.
1.1 the characteristics of
-
Check that the code complies with programming standards: naming conventions, writing conventions, etc.
-
Check for potential bugs in your design: SonarQube uses tools like plug-ins Findbugs, Checkstyle, and others to detect bugs in your code.
-
Detect code duplication: SonarQube can show that there is a lot of copy-and-paste code in a project.
-
Detect the degree of comment in your code: Too much or too little comment in your source code is not a good idea and affects the legibility of your program.
-
Check the relationships between packages and classes in code: analyze whether the relationships between classes are reasonable and how complex they are.
1.2 components
- SonarQube Server: A SonarQube Server that receives scan reports from clients
- SonarQube Database: ES/ and Database engine Oracle, PostgresQL, MSSQL
- SonarQube Plugins: You can install plug-ins later on the SonarQube server
- SonarQube Scanner: Scan tool installed on the client
1.3 architecture diagram
Developers push code into SCM (such as Gitlab) -> Jenkins to build defined jobs, The scanner client in Jenkins/gitlab-CI sends the analysis report to sonarqube Server
2 Installation and Deployment
2.1 SonarQube Server installation
2.1.1 Host Test environment Simple deployment
#downloadWget HTTP: / / https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-7.9.1.zip#Unpack the
/opt/sonarqube/
unzip /opt/sonarqube/bin/[OS]/sonar.sh console
#Log in to the host http://localhost:9000 (admin/admin)
Copy the code2.1.2 Host Deployment in production Environment
#The host machine requirementes
sysctl -w vm.max_map_count=262144
sysctl -w fs.file-max=65536
ulimit -n 65536
ulimit -u 4096
cat >> /etc/sysctl.conf << EOF
vm.max_map_count=262144
fs.file-max=65536
EOF
#Sonarqube cannot be executed as root
useradd sonarqube
echo "sonarqubepwd" | passwd --stdin sonarqube
#Check the system
[root@devops-sonarqube ~]# grep SECCOMP /boot/config-$(uname -r)
CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
CONFIG_SECCOMP_FILTER=y
CONFIG_SECCOMP=y
cat > /etc/security/limits.d/99-sonarqube.conf <<EOF
sonarqube - nofile 65536
sonarqube - nproc 4096
EOF
#Sonarqube ES requires the installation of JDK11
yum -y install java-11-openjdk.x86_64
#7.9 The latest version does not support mysql, but supports MSSQL, Oracle, and PostgreSQL
#Install PostgreSQL
#Create the sonarqube user and grant the create, UPDATE, and DELETE permissions to the sonarqube user
#If you want to customize the database name instead of pulic, you need to search for path changesYum yum install - y https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm Install -y postgresql96-server postgresql96-contrib /usr/pgsql-9.6/bin/postgresql96-setup initdb systemctl start Postgresql-9.6 systemctl enable postgresql-9.6 su - postgres PSQL create user sonarqube with password 'sonarqube'; create database sonarqube owner sonarqube; grant all on database sonarqube to sonarqube; \q
#Check the PostgresQL listenerVi/var/lib/PGSQL / 9.6 / data/postgresql. Conf
#Configuring a whitelistVi/var/lib/PGSQL / 9.6 / data/pg_hba conf host all all 127.0.0.1/32 md5#Restart the serviceSystemctl restart postgresql 9.6 ss - tan | grep, 5432#Create libraries/users and authorize themPSQL -h 127.0.0.1 -p 5432 -u postgres
#Downloading software PackagesCD/opt && wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-7.9.1.zip ln - sv sonarqube - 7.9.1 sonarqube chown sonarqube.sonarqube sonarqube/* -R
#Switch to the system sonarqube user to begin installation
su - sonarqube
#Set up database access, edit$SONARQUBE-HOME/conf/sonar.properties
sonar.jdbc.username=sonarqube
sonar.jdbc.password=sonarqube
#Note to 127.0.0.1Sonar, the JDBC url = JDBC: postgresql: / / / sonarqube 127.0.0.1
#Configuration ES storage paths, edit SONARQUBE - HOME/conf/sonar. The properties
sonar.path.data=/var/sonarqube/data
sonar.path.temp=/var/sonarqube/temp
#Configure the web server, edit SONARQUBE - HOME/conf/sonar. The propertiesSonar. Web. Host = 192.0.0.1 sonar. Web. Port = 80 sonar. Web. The context = / sonarqube
#Web server performance tuning
$SONARQUBE-HOME/conf/sonar.properties
sonar.web.javaOpts=-server
$SONARQUBE-HOME/conf/wrapper.conf
wrapper.java.command=/path/to/my/jdk/bin/java
#Execute the startup script
Start:
$SONAR_HOME/bin/linux-x86-64/sonar.sh start
Graceful shutdown:
$SONAR_HOME/bin/linux-x86-64/sonar.sh stop
Hard stop:
$SONAR_HOME/bin/linux-x86-64/sonar.sh force-stop
#Plug-in installation1. Install in Marketplace mode (Administration > Marketplace) 2. Install it manually (upload the downloaded plug-in to the server directory: $SONARQUBE_HOME/ Extensions /plugins and restart the Sonarqube service)Copy the code2.1.3 Docker deployment
docker pull sonarqube
docker run -d --name sonarqube -p 9000:9000 sonarqube
#Analyze the MVN project
# On Linux:
$ mvn sonar:sonar
# With boot2docker:
$ mvn sonar:sonar -Dsonar.host.url=http://$(boot2docker ip):9000
#Docker host system requirements
sysctl -w vm.max_map_count=262144
sysctl -w fs.file-max=65536
ulimit -n 65536
ulimit -u 4096
Copy the code2.2 Gitlab integration
2.2.1 sonar – scanner is installed
Since there are many gITlab projects and gitlab-Runner is shared, sonner-scanner can be installed in Gitlab-Runner to scan the built projects in general
#Download and installCD/opt && wget HTTP: / / https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.0.0.1744-linux.zip
#Add to the PATHMv sonar-scanner-4.0.0.1744- Linux sonar-scanner cat > /etc/profile.d/sonar-scanner. Sh <<EOF export PATH=$PATH:/opt/sonar-scanner/bin EOF source /etc/profile.d/sonar-scanner.sh [root@common-runner ~]# sonar-scanner -h INFO: INFO: usage: sonar-scanner [options] INFO: INFO: Options: INFO: -D,--define <arg> Define property INFO: -h,--help Display help information INFO: -v,--version Display version information INFO: -X,--debug Produce execution debug outputCopy the code2.2.2 Sonarqube Web configuration project
Create new project->Provide a token->
2.2.3 configuration gitlab – ci
stages:
- sonarqube_scan
- deploy_src
- install_dependency
- restart_server
- check_server
variables:
RUNNER_BASE_DIR: "/home/gitlab-runner/builds/QFafxxxEq/0/devops/"
BASE_DIR: "/go2cloud_api/"
job sonarqube_scan_job:
stage: sonarqube_scan
#Note that this user executes for gitlab-runner, specifying /. For this project directoryscript: - sonar-scanner -Dsonar.projectKey=go2cloud_api_test -Dsonar.sources=/. -Dsonar.host.url=http://43.xxx.xxx.xxx:9110 -Dsonar.login=a393276xxxxxxxxxxxxxxxxxxx03004a714 tags: - 51common-runner only: - go2cloud-platform-test when: always job deploy_src_job: stage: deploy_src script: -scp -r ${RUNNER_BASE_DIR}${BASE_DIR}* [email protected]:/project${BASE_DIR} Tags: -51common-runner only: - go2cloud-platform-test when: alwaysCopy the code
Submit code tests:
Viewing a Running Job
Check out the Sonarqube project
! [image-20190810112146155](/Users/xuel/Library/Application Support/typora-user-images/image-20190810112146155.png)
Check the details
2.3 Jenkins integration
You can use plug-in integration, or you can install Sonar – Scanner on top of Jenkins service area, each time the tool is scanned.
2.3.1 sonar – scanner is installed
Sonar -scanner installation is the same as gitlab-Runner installation, see: 2.2.1 Sonar -scanner installation
It can be integrated in two ways: scan command analysis reports directly at build time, and plug-in integration.
2.3.2 integration job
2.3.2.1 Script construction integration
Integrate with the installed sonar-scanner command at build time
#Configure the PATHexport PATH=/data/apps/miniconda3/bin:/data/apps/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/data/ap ps/miniconda3/bin:/data/apps/software/sonar-scanner/bin:/root/bin#Specify Jenkins' workspace directory
BASE_DIR=/root/.jenkins/workspace/
#Specify the project name, the name of this job
PROJECT=go2cloud_api_prod_job
#Specifies the virtual environment for the project in Conda
PROJECT_ENV=go2cloud-api-prod-env
#Switching The Python environment
source activate ${PROJECT_ENV}
$(which python) -m pip install mock nose coverage
#Update the Python environmentEcho "+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + update Python environment + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +" if [-f ${BASE_DIR}${PROJECT}/requirements.txt ]; then $(which python) -m pip install -r ${BASE_DIR}${PROJECT}/requirements.txt && echo 0 || echo 0 fi
#Code review/unit test/code test coverageEcho "+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + code review + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +" CD ${BASE_DIR}#Generate pylint. XML
$(which pylint) -f parseable --disable=C0103,E0401,C0302 $(find ${PROJECT}/* -name *.py) >${BASE_DIR}${PROJECT}-pylint.xml || echo 0
#echo "+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + unit testing + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +"
#Generate nosetests. XML
#$(which nosetests) --with-xunit --all-modules --traverse-namespace --with-coverage --cover-package=go2cloud-api-deploy-prod --cover-inclusive || echo 0
#$(which nosetests) --with-xunit --all-modules --traverse-namespace --with-coverage --py3where=${PROJECT} --cover-package=${PROJECT} --cover-inclusive || echo 0
#echo "+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + code coverage + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +"
#Generate coverage. XML
# -m coverage xml --include=${PROJECT}* | |echo 0
#Sonarqube code scan
sonar-scanner \
-Dsonar.projectKey=go2cloud_api_prod \
-Dsonar.sources=${BASE_DIR}${PROJECT}/. \
-Dsonar.host.url=http://xxx.xxx.xxx.xxx:9100 \
-Dsonar.login=2194d90xxxxxxxxxxxxxxxxxxxxxxxxbec7f69
Copy the code
Run project view
Check out the Sonarqube project
2.3.2.2 Plug-in integration
- Jenkins server scanner configuration
#If the address for sending scan reports to the server is not specified during the build, configure it on the client and change it to: sonar-scanner.properties Server address in the conf directory of scanner
sonar.host.url=http://xxx.xxx.xxx.xxx:9100
Copy the code-
Jenkins Server install scanner
-
Configure the scanner in tools
- The Sonarqube Server produces tokens
The Jenkins API token needs to be configured on sonarqube Server to enable Jenkins to send reports to Sonarqube Server
- Add SonarQube Servers to global tool configuration
Jenkins uses Sonarqube’s token to create credentials
- Build the project configuration
Fill in the corresponding Analysis Properties according to the scanned program language, and fill in the project information here.
If you are using pipelines, see the declarative example
pipeline {
agent any
stages {
stage('SonarQube analysis 1') {
steps {
sh 'mvn clean package sonar:sonar'
}
}
stage("Quality Gate 1") {
steps {
waitForQualityGate abortPipeline: true
}
}
stage('SonarQube analysis 2') {
steps {
sh 'gradle sonarqube'
}
}
stage("Quality Gate 2") {
steps {
waitForQualityGate abortPipeline: true
}
}
}
}
Copy the code- Viewing scan Results
With projectName configured on Jenkins, you don’t need to configure project on Sonarqube
Three thinking
- Integration scenario Since the backend of the project is Python and is not compiled here, Java projects need to install the corresponding MVN or other compilation tools.
- The test environment was gitLab CI/CD, and the formal environment was released using Jenkins
- Compared with Jenkins script integration and plug-in integration, it can be found that the script integration needs to be configured on both sides, and for different projects, it is troublesome to apply for tokens and associate project notes each time. If the plug-in integration is configured once, it does not need to be configured on Sonarqube. After that, you only need to configure projectName for Jenkins job, which is more convenient.
Four Error handling
-
Ob’s log exceeded limit of 4194304 bytes ob’s log exceeded limit of 4194304 bytes
https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/configuration/advanced-configuration.md#the-runners-section Copy the code
5 reference links:
-
Docker mirror
-
The official documentation
-
Postgresql Common commands
-
Sonarqube upgrade