Creditease Container Cloud is a container management platform based on Kubernetes. When line of business users deploy applications on the container cloud, they often encounter situations where the container does not start or the application does not run properly. In order to facilitate users to troubleshoot problems in the cloud process of application, we have integrated a series of troubleshooting methods on the Web side, as shown in the following figure:
I. Terminal information
The terminal view is the standard output log when the container instance is running.
Kubectl logs PODNAME [-c CONTAINER]
The basic principle is as follows:
During application deployment, the kubelet of the node to which it belongs invokes the Container Runtime Interface through GRPC to request the Docker daemon to create the container runtime.
The docker daemon creates a coroutine to receive STDOUT logs from the container runtime. This coroutine eventually writes STDOUT logs to the corresponding directory on the node where the container runtime is located: /var/lib/docker/containers/container_id/{container_id-json.log}
The diagram below:
When viewing the terminal information of the corresponding instance on the Web side, Kubelet converts the received API-server request into docker Client to request the Docker daemon. Docker daemon reads the log file data of the corresponding container in the corresponding directory, and then kubelet returns the log data to apI-server, which is finally displayed on the Web side for users to view.
The lifetime of a container log is the same as the lifetime of the container. After the container is destroyed, its associated log files are also destroyed.
Second, the events
Events is used by Kubelet to record the events during container startup and execution.
Kubectl get events
Similarly, when using Kubectl Describe Pod to view pod, you can also see events related to the POD. From this information, you can clearly see the state change of the event, thus learning the various reasons for the POD startup failure. Such as:
1) There are no available nodes for scheduling, such as insufficient node resources;
2) Health status check fails;
3) Failed to pull the mirror as shown in the following figure:
The basic implementation of Events is as follows:
Events contains event-related types, causes, sources, messages, etc., which will be generated in components such as Kubelet and Controller Manager. After broadcast, it will be filtered and aggregated by a series of functions, and then sent to API-Server for storage in ETCD. When events events are viewed on the Web end, apI-server is requested to read the corresponding event in ETCD and return the display for users to view abnormal parameters and error status.
Three, web terminal
Web Terminal provides an interactive interface shell for executing various commands.
Kubectl exec it
bash
The web display is shown as follows:
The implementation is as follows:
Web Terminal is mainly realized through websocket technology, and the front interface uses the open source project Container-Terminal (github.com/kubernetes-…
When viewing the Web Terminal, the front-end Web initiates a Websocket request to the API-server. The kubelet of the owning node responds to the request of the API-server and establishes a connection with the container runtime.
The connection between Kubelet and the container runtime is possible because Kubelet defines a RuntimeServiceClient interface in the CRI specification, and the RuntimeServiceServer (Streaming Server, Streaming API) implements this interface.
After kubelet establishes a connection with the container runtime, Kubelet returns a request, Api-server establishes a stream with the target container runtime Streaming Server by upgrading the request to SPDY(SPDY allows reuse of independent STDIN/STDOUT/STDERR in a single TCP request and mapping the WS stream to the corresponding standard stream of SPDY. Api-server implements data interaction between the Web and the container runtime.
In this case, you can enter the command on the Web and see the returned result after executing the command. In this way, interaction is realized.
Web Terminal provides convenience for entering containers, and users can perform any operation. For security, necessary security measures are taken:
1) User operation commands are recorded.
After a user enters a command, the operation is recorded for security audit.
2) The production environment uses ordinary users to enter containers.
C chmod -r 777 $KUBERNETES_FILELOGS; $KUBERNETES_FILELOGS; useradd spider > /dev/null 2>&1; Su spider, where the environment variable $KUBERNETES_FILELOGS is the file directory that needs to be authorized when the container is created. The main purpose is to prevent user misoperations, such as deleting storage mounts.
4. Debug container
The Debug container uses the tool container to remove barriers from the service container.
In the process of using Web Terminal to debug applications, line of business users often need various commands to debug programs. Previous solutions were either to customize the base image for the line of business to cover as many commands as they needed, or to add commands to the Dockerfile as the line of business users built the image.
However, because there are many lines of business, the workload of customized basic mirror is too large; Adding too many commands during service mirroring construction is cumbersome and may cause security risks. None of these solutions actually conforms to the container technology practice of building the simplest container image possible, and the simplified image is severely missing the required command tools.
In view of this contradiction, we integrated and revamped Kubectl-debug (github.com/aylei/kubec…
The effect is similar to:
docker run -it --network=container:<container_ID> --pid=container:<container_ID> --ipc=container :<container_ID> -v /log/container _ID:/debugviewlogs <image>
The following figure is displayed on the Web:
The debug container works as follows:
Debug-agent was deployed to all nodes of Kubernetes cluster in the form of DaemonSet, and /var/dock/docker.sock of the host was mounted to realize communication with docker Daemons. Steps as shown above:
1) The Web end provides pod cluster, namespace, and Podname information. The Backend server initiates Websocket requests.
2) After receiving the request, the Backend server checks whether the POD exists to the API-server and sends information about the Node host where the POD resides and the POD container to determine whether debugging can be performed.
Note: If the pod status reason is CrashLoopBackOff, the Backend server will ask apI-server for Kubelet to copy a POD with the boot command (sleep) overwritten, label removed, and health check removed. Subsequent debug operations are performed on the copied POD.
3) The Backend server sends DEBUGGING POD information and sends debugging requests (upgraded SPDY requests that map the standard FLOW of WS).
4) After receiving the request, the debug-agent starts to pull the Debug tool image to create a Debug container and set each namespace of the Debug container as the namespace of the target service container APP. Then mount the directory /log/ of the host Node to the directory/debugViewLogs of the debug container to download the files generated in the debug container on the Web. The following two figures:
After the debug container is created, the debug-agent relayed SPDY requests from the Backend server to the Debug container. The Debug container attaches the SPDY standard flow to the business container. This allows the Web side to interact with the Debug container. After the debug operation is complete, the Debug-agent clears the debug container and reclaims it. Similarly, debug operations are audited.
Therefore, we simply build an image with a number of troubleshooting tools, which not only practices the principle of keeping the business image as simple as possible, but also provides the various command tools needed to debug the application.
conclusion
Terminal information, Events, Web Terminal, and Debug containers all provide a visual Web that allows users to quickly and easily troubleshoot Pods and applications.
Author: Duan Dehua
Source: Creditease Institute of Technology