This is the sixth day of my participation in the November Gwen Challenge. See details: The Last Gwen Challenge 2021.

Hello everyone, I’m Zhang Jintao.

As the cornerstone of cloud native, Kubernetes brings us great convenience, and more and more companies are using Kubernetes in their production environments. However, while enjoying the convenience, we also need to pay attention to some of the security risks.

In this article, I will focus on container image security.

Generally, when we refer to container image security, we mainly mean the following two aspects:

  • The security of the mirror’s own content;
  • Image distribution process security;

Security of mirror’s own content

To talk about the security of a mirror’s own content, we need to know what a mirror is and what it contains.

What is a mirror image

Take the debian image as an example. Pull the latest image, save it as a tar file, and unpack it:

➜  ~ mkdir -p debian-image
➜  ~ docker pull debian
Using default tag: latest
latest: Pulling from library/debian
647acf3d48c2: Pull complete 
Digest: sha256:e8c184b56a94db0947a9d51ec68f42ef5584442f20547fa3bd8cbd00203b2e7a
Status: Downloaded newer image forDebian: latest docker. IO/library/debian: latest ➜ ~ docker image save - o debian - image/debian. Tar debian ➜ ~ ls Debian -image debian. Tar ➜ ~ tar -c debian-image -xf debian-image/debian. Tar ➜ ~ tree -i debian Debian - image ├ ─ ─ 827 e5611389abf13dad1057e92f163b771febc0bcdb19fa2d634a7eb0641e0cc. Json ├ ─ ─ B331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda │ ├ ─ ─ json │ ├ ─ ─ layer. The tar │ └ ─ ─ VERSION ├ ─ ─ Manifest. json └─ repositories 1 Directory, 6 filesCopy the code

After decompressing it, we see that it is a bunch of JSON files and layer.tar files. We unzip the layer.tar file again:

➜ debian ~ tar - C - image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda - xf Debian - image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda/layer. The tar ➜ ~ tree - I'layer.tar|json|VERSION'-L 1 debian-image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda Debian - image/b331057b5d32f835ac4b051f6a08af6e9beedb99ec9aba5c029105abe360bbda ├ ─ ─ bin ├ ─ ─ the boot ├ ─ ─ dev ├ ─ ─ etc ├ ─ ─ the home ├ ─ ─ lib ├ ─ ─ lib64 ├ ─ ─ media ├ ─ ─ MNT ├ ─ ─ opt ├ ─ ─ proc ├ ─ ─ root ├ ─ ─ the run ├ ─ ─ sbin ├ ─ ─ the SRV ├ ─ ─ sys ├ ─ ─ TMP ├ ─ ─ usr └ ─ ─ var 19 directories, 0 filesCopy the code

You should already be familiar with the unzipped directory structure. Yes, this is the directory structure of Rootfs.

If we use some of our own application images, after a few unzipping, you will also find the corresponding application files in them.

How to ensure the security of the mirror’s own content

We have seen that the container image is a combination of RootFs and applications, as well as some configuration files. Therefore, to ensure its own content security, mainly from the following aspects to consider:

rootfssecurity

In our case, rootfs are usually provided by the underlying (system) image we use, or can be thought of as the image configured in the FROM field of the Dockerfile when we build the image.

To achieve security in this aspect, we need to:

  • Use images from trusted sources, such as official Docker maintained images;

  • Continuous vulnerability scanning and upgrading of the base image;

  • You can also consider using Distroless mirroring to provide some protection from attacks;

The application

The application is actually provided by ourselves, and to be secure in this respect, we need to:

  • Continuous software vulnerability scanning;

  • Update dependencies in a timely manner;

  • Consider transitioning from SDL (Security Development Lifecycle) to DevSecOps;

The configuration file

The configuration files contained in the image are provided by the image builder, and as long as we ensure that the image builder we use has not been tampered with or left with any bugs, this is generally not a problem.

In general, we can directly use tools such as Trivy or Anchore Engine to help ensure the security of mirror content. In addition, some image warehouses, such as Harbor, have built-in image security scanning tools, or you can use the Docker Scan command for image security scanning.

Image Distribution Security

How to distribute images

Let’s start by looking at how container images are built and deployed into our Kubernetes environment.

Figure 1 shows a brief diagram of the container image from creation to release deployment

After writing the code, the developer pushes the code to the repository. This triggers the CI to build, during which the image is built and pushed to the mirror repository.

In the CD part, images from the mirror repository are used and deployed to the target Kubernetes cluster.

So how does an attacker attack during this process?

Security issues in image distribution

Figure 2, image distribution deployment security example

As shown in the figure, in the process of image distribution deployment, its upstream is the mirror warehouse and downstream is the Kubernetes cluster. For mirror warehouse, even for Intranet self-built environment, because our concept has changed from boundary-based security to zero trust security, so we uniformly take public warehouse as an example to explain.

An attacker can hijack and replace a malicious image by several means, including attacking the mirror warehouse directly.

To ensure the security source and integrity of the images deployed to Kubernetes cluster, there are two main steps:

  • Signature of the image when building the image;
  • Verify signatures during image distribution deployment. (Continue in the next article)

Let’s look at each of them.

Mirrored labels and summaries

We usually have two options when using container images:

  • Tags, such as Alpine :3.14.3

  • In this paper, such as alpine @ sha256:635 f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2

In most cases, we will use the tag directly because it is more readable. However, the contents of the image may change over time, because we may use the same tag for different images. The most common tag is “Latest”. Each time a new version is released, the latest tag will continue to be used for the image of the new version, but the version of the application has been updated to the latest.

The main drawback to using a digest is its poor readability, but each mirrored digest is unique, and the digest is the hash of the mirrored content’s SHA256. So we can guarantee the uniqueness of the mirror image by abstracting it.

You can see the label and summary information directly in the following example:

➜ ~ docker pull Alpine :3.14.3 3.14.3: Pulling from library/ Alpine Digest: sha256:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2 Status: The Image is the up to date for alpine: 3.14.3 docker. IO/library/alpine: 3.14.3 ➜ ~ docker Image inspect alpine: 3.14.3 | jq - r '[] | {RepoTags: RepoTags, RepoDigests: RepoDigests}' {" RepoTags ": [" alpine: 3.14.3"], "RepoDigests" : [ "alpine@sha256:635f0aa53d99017b38d1a0aa5b2082f7812b03e3cdb299103fe77b5c8a07f1d2" ] }Copy the code

So how to ensure the correctness/security of the mirror? This is the main problem that mirror signature solves.

Mirror signature solution

Digital signatures are a well-known method for maintaining the integrity of any data transmitted over a network. There are several common schemes for container image signatures.

Docker Content Trust (DCT)

You may have similar experiences when transferring common files. For example, the downloaded files are incomplete due to network reasons. Or files are tampered with or replaced by an attack by a middleman.

In fact, images may encounter similar problems in the distribution process, and this is the focus of our discussion here, which is mainly solved by Docker Content Trust (DCT).

Docker Content Trust uses digital signatures and allows clients or runtimes to verify the integrity and publisher of a particular image tag. For use, this is the function provided by the Docker Trust command. Note: This requires Docker CE 17.12 and above.

As mentioned earlier, a mirrored record can have some tags in the following format:

[REGISTRY_HOST[:REGISTRY_PORT]/]REPOSITORY[:TAG]
Copy the code

In the case of a label, the DCT is associated with a portion of the label. Each mirror repository has a set of keys that the mirror publisher uses to sign the image label. (Image publishers can decide which labels to sign.) A mirror repository can contain multiple images with both signed and unsigned labels.

Note that if a mirror publisher pushes the latest image with a signature first and then an unsigned latest image, the latest image does not affect the content of the previous image (different from the area covered by the label above).

Figure 4. Example of DCT image signature (in the figure, the authentication process for logging in to the image warehouse is simplified)

In production, we can enable DCT to ensure that the images used are signed. If DCT is enabled, only trusted images (signed and verifiable images) can be pulled, run, or built.

Enabling DCT is a bit like applying a “filter” to the image repository. That is, only signed image labels are seen, not unsigned image labels. If the client does not have DCT enabled, it can see all images.

Here’s a quick look at how DCT works

Its trust in the mirror label is managed through the use of a signing key. The key set is created when we first turn on the DCT and use it. A key set consists of the following types of keys:

  • An offline key, which is the root of the image label DCT (difficult to recover if the root key is lost)

  • The repository or tag key that signs the tag

  • Server-managed keys, such as timestamp keys

Figure 5. Example of a mirror signing key

We just mentioned that the client uses DCT, which is also our Docker Trust command, which is built on Noker V1. DCT is disabled in Docker clients by default. To enable this, set the DOCKER_CONTENT_TRUST=1 environment variable.

The effect is as follows:

➜ ~ DOCKER_CONTENT_TRUST=1 docker pull alpine:3.12 pull (1 of 1): Alpine: 3.12 @ sha256: c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a docker.io/library/alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a: Pulling from library/alpine 188c0c94c7c5: Already exists Digest: sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a Status: Downloaded newer image for alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a Tagging Alpine @ sha256: c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a as alpine: 3.12 Docker. IO/library/alpine: 3.12Copy the code

Notary v1

DCT is based on Notary V1, but it is not the focus of this paper. Project Address: github.com/notaryproje…

Figure 6 shows the interaction flow of the Notary client, server, and signature

Procedure 1 – Authentication, any connection without a token will be redirected to the authorization server (Docker Registry V2 authentication);

Procedure 2 – The client will log in to the authorization server through authentication over HTTPS to obtain the token;

Procedure 3 – When the client uploads new metadata files, the server checks them for conflicts based on previous versions and verifies the signature, checksum, and validity of the uploaded metadata;

Procedure 4 – Once all uploaded metadata has been validated, the server generates timestamps (and possibly snapshots) and sends them to Sign for signature;

Procedure 5-sign retrieves the encrypted private keys from its database, decrypts the keys, signs them with them, and sends them back to the server;

Procedure 6 – The server stores client uploads and server-generated metadata in the TUF library. The generated timestamp and snapshot metadata prove that the metadata uploaded by the client is the latest version of the trusted collection. After that, the server notifies the client that the upload was successful;

Procedure 7 – The client can now download the latest metadata from the server immediately. In the case of an expired timestamp, the server iterates through the sequence, generates a new timestamp, requests the sign signature, and stores the timestamp of the new signature in the database. It then sends this new timestamp along with other stored metadata to the requesting client;

This project is a security project, although the purpose is very large, but overall not active. V2 version is now under development, interested partners welcome to join.

Sigstore and Cosign

Here’s another project from the Linux Foundation called SIGStore that aims to provide some standard libraries/tools for better signing and validation. Sigstore, of course, is now a collection of open source projects including Cosign, Fulcio, and Rekor, covering aspects of mirroring, mirroring signature verification, and supply chains.

Figure 7, SigStore introduction

Cosign is one of sigStore’s tools for creating, storing, and validating container image signatures in OCI Registry. Cosign V1.0 was released in the second half of this year, and it remains to be seen whether it can be used in a stable production environment. As of now, Cosign has released V1.3.1. Please refer to ReleaseNote: github.com/sigstore/co…

So let’s see how does it sign the mirror

➜ cosign cosign generate-key-pair Enter password for private key: Enter password for private key again: Private key written to cosign.key Public key written to cosign.pub ➜ cosign cosign sign -- Key cosign.key ghcr.io/tao12345666333/argo-cd-demo/argo-cd-demo:fa5714f419b3d11dee6ac795e38356e9c3c439cb Enter password for private key: % ➜ cosign cosign verify --key cosign.pub ghcr.io/tao12345666333/argo-cd-demo/argo-cd-demo:fa5714f419b3d11dee6ac795e38356e9c3c439cb Verification for ghcr.io/tao12345666333/argo-cd-demo/argo-cd-demo:fa5714f419b3d11dee6ac795e38356e9c3c439cb -- The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key - Any certificates were verified against the Fulcio roots. [{"critical":{"identity":{"docker-reference":"ghcr.io/tao12345666333/argo-cd-demo/argo-cd-demo"},"image":{"docker-manife st-digest":"sha256:768845efa2a32bc5c5d83a6f7ec668b98f5db46585dd1918afc9695a9e653d2d"},"type":"cosign container image signature"},"optional":null}]Copy the code

It looks pretty simple.

conclusion

This is about the content security of the mirror itself and the image signature verification part of the image distribution security.

In the next article, I will show you how to verify signatures during image distribution and deployment, and how to protect the Kubernetes cluster from unsigned or untrusted source images. Stay tuned!

Please feel free to subscribe to my official account [MoeLove]