There was a time when we were running our applications on Kubernetes, and every time a container crashed, we had to restart the container and not know what to do with it. The shell is usually included in the image of the business development build. We can also block the start of the container service by embedding a [“sleep”, “3600”] command in the command, but sometimes a distroless image will appear out of nowhere. Operations may be the first to collapse. It was the first time since the birth of the operation and maintenance profession that I felt helpless and out of control. So in the K8S environment can not debug container stem began to be widely mocked.

The first to break the spell is Kubectl-debug, which contains both agent and Debug-Tools. It is also the most complete solution in the whole network. Development seems to have stalled, however, with the last submission eight months ago and the most recent Release two years ago. What is more unacceptable is that it cannot currently be integrated into the K8S cluster that is Containerd when the container runs.

Although Kubectl-Debug was once a really good container debugging tool, today Kubernetes has a better container debugging solution, Ephemeral Containers.

Ephemeral Containers

Ephemeral Containers, as the name suggests, are temporary Containers. This is a new feature that has been introduced as alpha since Kubernetes V1.16. Although it doesn’t have GA yet, since Kubernetes V1.18, the Debug client has been integrated in Kubectl, and we can almost fully use and experience its new features.

The goal of the temporary container is to provide Kubernetes users with a troubleshooting tool that meets the following needs:

  • As an out of the box platform chemical tool
  • Does not depend on the tools already included in the container image
  • You don’t need to log in directly to the compute Node (you can access Node through Kubernetes API administration)

However, there are some things temporary containers are not planned to support, such as enabling temporary containers on Windows that are not very friendly.

Enabling temporary containers is also very simple. In versions after Kubernetes V1.16, configure the startup parameter –feature-gates=EphemeralContainers=true to kube-API and Kubelet services.

Prior to 1.20, the kubectl debug tool was placed in alpha. Note the difference in command operation between different versions. It is recommended to use the 1.20+ client for better experience

So what can we do with Ephemeral Containers?

1.POD Troubleshooting

As mentioned above, we can do container debugging directly with the kubectl debug command. The most direct and simple debugging command for a POD is as follows:

kubectl debug mypod -it --image=busybox
Copy the code

By default, kubectl automatically generates a unique ID for the debug container name if the user does not specify a temporary container name. If the user needs to specify the container name himself

kubectl debug mypod -c debugger --image=busybox
Copy the code

With the temporary container, we can expand the gameplay beyond the daily debug functionality. For example, batch run scripts for security scans in a namespace without interfering with the original container.

for pod in $(kubectl get -o name pod); 
do
    kubectl debug --image security/pod_scanner -p $pod /sanner.sh
done
Copy the code

2.POD Troubleshooting by Copy

For clusters that do not have the Ephemeral Containers feature enabled, the only way to debug Containers is through replication mode. It works by copying a new container with the specified POD and starting DEBUG as a sidecar along with the new container. In this way, we can also achieve the goal of curvilinear national salvation. Some of the parameters for this approach are interesting:

--copy-to Specifies the name of the new pod --replace=true Whether to delete the original pod --same-node=true Whether to dispatch the same node as the original pod --share-processes=true Whether to share the PID space of the containerCopy the code

For example, we can start a debug container with the same configuration as we need to debug the POD:

kubectl debug mypod -it \
--container=debug \
--image=busybox \
--copy-to=my-debugger \
--same-node=true \
--share-processes=true 
Copy the code

3.Node Troubleshooting

Right! You read that right! Worker nodes can also be debugged using Ephemeral Containers. When called against a node, Kubectl Debug creates a POD with the node name and schedules it to that node. The container also has the hostIPC, hostNetwork, and hostPID privilege modes. Surprisingly, the root file system of the Worker node is also mounted to the /host directory of the DEBUG container.

To debug the host, run this command directly.

kubectl debug node/mynode -it --image=busybox
Copy the code

The Debug image

To do a good job, he must sharpen his tools. Either way, we need a well-tooled debug image that can handle problems at our ease. There are plenty of debug images available online, but none of them are as enjoyable as building your own.

Here xiao White share a Debug image Dockerfile, you can modify according to their own conditions.

FROM golang:alpine as grpcurl RUN apk update \ && apk add --virtual build-dependencies git \ && apk add bash curl jq \ && go get -u github.com/fullstorydev/grpcurl \ && go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest FROM alpine:latest RUN sed -i 's/dl-cdn.alpinelinux.org/mirrors.tuna.tsinghua.edu.cn/g' /etc/apk/repositories && \ apk update  && \ apk add --no-cache vim bash tcpdump curl wget strace mysql-client iproute2 redis jq iftop tzdata tar nmap bind-tools htop && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime RUN wget -O /usr/bin/httpstat https://github.com/davecheney/httpstat/releases/download/v1.0.0/httpstat-linux-amd64-v1.0.0 && \ chmod + x /usr/bin/httpstat COPY --from=grpcurl /go/bin/grpcurl /usr/bin/grpcurl ENV TZ=Asia/Shanghai LC_ALL=C.UTF-8 LANG=C.UTF-8 LANGUAGE=C.UTF-8 ENTRYPOINT [ "/bin/bash" ]Copy the code

The following figure shows the tool packages supported by the DEBUG image

conclusion

This article focuses on the Ephemeral Containers feature of Kubernetes, which was put on alpha after v1.18. Through temporary Containers we can debug Containers and even hosts. It is indeed a very convenient and adequate alternative to Kubectl-Debug. However, there is no special description for user permissions in temporary containers, especially when debugging the host in privileged mode. We hope to use PSP (Pod Security Policy) to make an additional supplement.

Pay attention to the public account [Cloud native Xiaobai]