I recently had a project to consider safe containers, so I researched Kata Contaienr for a taste

Introduction of Kata container

Kata Container is a merger of two existing open source projects: Intel Clear Containers and Hyper runV.

The Goal of the Intel Clear Container project is to address security issues inside containers with Intel ® virtualization technology (VT), and to be able to start containers as lightweight virtual machines (VMS), providing an optional uptime focused on performance (<100ms startup time), Interoperable with common container environments such as Kubernetes and Docker. Intel ®Clear Container shows that security and performance do not have to be a trade-off, and that security by isolating hardware can be met with the performance of the Container.

Hyper runV supports many different CPU architectures and hypervisors prior to technology independence.

The container security

When using Docker lightweight containers, the biggest problem is the problem of security. Several different containers can attack each other. If the kernel is attacked, all other containers will crash. If virtualization technologies such as KVM are used, the security problem can be solved perfectly, but the response performance may be affected.

For Docker alone, security can be summarized into two points:

  • Hosts are not affected
  • There is no impact on other containers

So more than 90% of security problems can be attributed to isolation problems. The security problem of Docker is essentially the security problem of container technology, including the shared kernel problem and the incomplete limitation of Namespace:

  1. /proc, /sys, and so on are not completely isolated
  2. The information displayed in the Top, free, and iostat commands is not isolated
  3. The Root user is not isolated
  4. /dev device is not isolated
  5. The kernel module is not isolated
  6. SELinux, time, syslog, and other information outside the existing Namespace are not isolated. Of course, the insecure image itself may cause security problems.

Kata isolation technology

As we all know, Docker and other containers use Cgroup to limit resources, and use the Namespace mechanism of Linux to isolate containers. However, in actual operation, hosts still provide containers with network, storage, computing and other resources directly. Although performance loss is small, there are certain security problems.

Kata Containers is an open source community dedicated to building secure container runtimes with lightweight virtual machines that feel and perform like Containers, but use hardware virtualization technology as a second layer of defense to provide greater workload isolation.

The KATA infrastructure implements the best part of Intel Clear Containers merged with Hyper-.sh RunV and has been extended to include support for major architectures, including AMD64 beyond x86_64, ARM, IBM P series, and IBM Z Series.

Kata Containers also supports several VIRTUAL machine managers including QEMU, NEMU and Firecracker, and integrates with other containerization programs.

QEMU IS a vm similar to KVM, but KVM is a hardware-assisted virtualization technology. KVM is mainly responsible for CPU and memory virtualization, while QEMU is responsible for I/O virtualization

Architecture principles

Kata actually implements resource isolation between containers by creating lightweight virtual machines, and then running the container runtime in the virtual machine. This enables the container to run in a dedicated kernel, providing network, I/O, and memory isolation, and enforcing hardware isolation through virtualized VT extensions. Provides security while still providing high performance.

Kata consists of two main components:

  • Container Runtime: Creates a QEMU for each Container or POD/KVM VM: SupportedOCI runtime specification, can be embedded in Docker as a plug-in as the underlying container runtime; alsoSupports THE CRI interface of K8s*, can be integrated with Cri-O and Containerd
  • Agent: Used in the host machine to manage containers and the processes running in those containers.

Kata and kata – shim

In the Docker engine, Docker calls Containerd, which uses shim to call Runc to create the underlying container. (This process was described in more detail in Understanding Containers.) Containerd calls the kata-shim provided by KATA to create the KATA container runtime.

It is worth mentioning that Kata is now working on implementing the Shimv2 interface, which mainly affects K8s calls to CRI to create pods and containers.

With shimv2, Kubernetes can use one shimv2 per Pod instead of 2N + 1 Shim (shim and Kata Shim for each container and Pod sandbox itself)

agent

Container processes are generated by the Agent and run as daemons on VMS.

Kata-agent uses the VIRTIO serial or VSOCK interface to run the gRPC server in the host machine, where QEMU provides links to the outside world by exposing socket files. Kata-runtime uses the gRPC protocol to communicate with the Agent. This protocol allows the runtime to send container-managed commands to the Agent. The protocol is also used to transfer I/O streams (STdout, STderr, stDIN) between containers and management engines such as Docker Engine. This means that in the container, all commands and IO that interact with the container, including the init process, will be VIRTIO or VSOCK.

K8s integration kata – the container

Preparation process

  1. Hardware support

    The node must support any of the following CPU virtualization technologies
    • Intel VT-x technology
    • ARM Hyp mode
    • IBM Power Systems
    • IBM Z manframes 
> deployment in a VMware virtual machine, if the host machine open virtualization * * * * nested functions, open step see links to https://blog.51cto.com/11434894/2389180?source=draCopy the code
  1. Software depend on
    • A functioning K8s environment
    • containerd

Install the kata – the container

Execute the following script to install

source /etc/os-release sudo yum -y install yum-utils ARCH=$(arch) BRANCH="${BRANCH:-master}" sudo -E yum-config-manager --add-repo "http://download.opensuse.org/repositories/home:/katacontainers:/releases:/${ARCH}:/${BRANCH}/CentOS_${VERSION_ID}/home: katacontainers:releases:${ARCH}:${BRANCH}.repo" sudo -E yum -y install kata-runtime kata-proxy kata-shimCopy the code

Note: At the time of my installation, Kata had just released version 1.11 Alpha0, which relied on qume-Vanilla version 4.1.1+ Git.99c5874a9b, However, the repo source is only 4.1.0+ git.9e06029AA-6.1.x86_64, so before executing the script, you need to set the BRANCH environment variable: export BRANCH=’ stables -1.10′

After the installation is complete, run the kata-Runtime kata-check command to check whether the system supports the Kata Runtime

WARN[0000] modprobe insert module failed: modprobe: FATAL: Module vhost_vsock not found.
  arch=amd64 error="exit status 1" module=vhost_vsock name=kata-runtime pid=21060 source=runtime
ERRO[0000] kernel property not found                     arch=amd64 description="Host Support for Linux VM Sockets" name=vhost_vsock pid=21060 source=runtime type=module
System is capable of running Kata Containers
System can currently create Kata ContainersCopy the code

The error may be caused by my use of VMware nested virtualization.

Changed to Containerd when THE K8s connection is running

  1. First, enable Containerd
    systemctl enable containerdCopy the code

Modify containerd configuration file, / etc/containerd/config toml, website see the default configuration link: https://github. com/containerd/cri/blob/master/docs/config. The md

Note that: IO /pause-amd64:3.2 '-containerd prior to version 1.3.2, the non-secure registry was not supported. If you are using a private repository, you can upgrade to 1.3.3. And adding entries ` ` ` toml [plugins. "IO. Containerd. GRPC. V1. Cri". The registry. The mirrors. "your registry". TLS] insecure_skip_verify = true ` ` ` 2. Modify kubelet launch parameters ` ` ` bash $sudo mkdir -p/etc/systemd/system/kubelet. Service. D / $cat < < EOF | sudo tee /etc/systemd/system/kubelet.service.d/0-containerd.conf [Service] Environment="KUBELET_EXTRA_ARGS=--container-runtime=remote --runtime-request-timeout=15m --container-runtime-endpoint=unix:///run/containerd/containerd.sock" EOF ``` 3. -reload systemctl start containerd systemctl restart kubeletCopy the code

K8s configuration uses Kata

K8s uses Kata in two ways:

  • Using K8s untrustedworkloadruntime
  • Use K8s RuntimeClass, which was used as an alpha feature in version 1.12

The use of untrustedworkloadruntime

First, add an entry to containerd’s configuration file

# "plugins.cri.containerd.untrusted_workload_runtime" is a runtime to run untrusted workloads on it.
    [plugins.cri.containerd.untrusted_workload_runtime]
      # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
      runtime_type = "io.containerd.kata.v2"Copy the code

If you are using an earlier version of Kata that does not support shimV2, use the entry

    [plugins.cri.containerd.untrusted_workload_runtime]
      # runtime_type is the runtime type to use in containerd e.g. io.containerd.runtime.v1.linux
      runtime_type = "io.containerd.runtime.v1.linux"

      # runtime_engine is the name of the runtime engine used by containerd.
      runtime_engine = "/usr/bin/kata-runtime"Copy the code

Then restart containerd systemctl daemon-reload & systemctl restart containerd

K8s resources need to add annotations

annotations:
   io.kubernetes.cri.untrusted-workload: "true"Copy the code

Such as:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-untrusted
  annotations:
    io.kubernetes.cri.untrusted-workload: "true"
spec:
  nodeName: k8s3
  containers:
  - name: nginx
    image: nginxCopy the code

Using RuntimeClass

In this mode, the software version requirements are as follows:

  • Kata Container > = 1.5.0
  • Containerd > = 1.2.0
  • K8s > = 1.12.0

Configure the items in the Containerd configuration file

[plugins.cri.containerd] no_pivot = false [plugins.cri.containerd.runtimes] [plugins.cri.containerd.runtimes.runc] runtime_type = "io.containerd.runc.v1" [plugins.cri.containerd.runtimes.runc.options] NoPivotRoot = false NoNewKeyring =  false ShimCgroup = "" IoUid = 0 IoGid = 0 BinaryName = "runc" Root = "" CriuPath = "" SystemdCgroup = false [plugins.cri.containerd.runtimes.kata] runtime_type = "io.containerd.kata.v2" [plugins.cri.containerd.runtimes.katacli] runtime_type = "io.containerd.runc.v1" [plugins.cri.containerd.runtimes.katacli.options] NoPivotRoot = false NoNewKeyring = false ShimCgroup = "" IoUid = 0 IoGid = 0 BinaryName = "/usr/bin/kata-runtime" Root = "" CriuPath = "" SystemdCgroup = falseCopy the code

Create RuntimeClass in K8s

apiVersion: node.k8s.io/v1beta1 # RuntimeClass is defined in the node.k8s.io API group kind: RuntimeClass metadata: Name: kata handler: kata # here and in the configuration file is containerd [plugins. Cri. Containerd. Runtimes. {handler}] matchCopy the code

Create a pod

apiVersion: v1
kind: Pod
metadata:
  name: kata-nginx
spec:
  runtimeClassName: kata
  containers:
    - name: nginx
      image: nginx
      ports:
      - containerPort: 80Copy the code

You can view the created Container by running the kata-Runtime list

ID PID STATUS BUNDLE CREATED OWNER 5a512bfa19cd166c6cc6411dd7e1d20cc41490d0f2df064adab7871c9656a058 -1 running /run/containerd/io.containerd.runtime.v2.task/k8s.io/5a512bfa19cd166c6cc6411dd7e1d20cc41490d0f2df064adab7871c9656a058 The 2020-02-26 T09:49:43. 743162591 z # 0 432628537 c5d5606707e225869846542c68cb7406992dcf55a4b48a494f333b8-1 running /run/containerd/io.containerd.runtime.v2.task/k8s.io/432628537c5d5606707e225869846542c68cb7406992dcf55a4b48a494f333b8 The 2020-02-26 T09:49:47. 928499707 z # 0Copy the code

If you like, please pay attention to my public number, or view my blog at http://packyzbq.coding.me. I will send my own learning records from time to time, so we can learn and communicate with each other