In order to facilitate everyone to learn Kubernetes system, I organized a Kubernetes learning series of articles, covering the basic knowledge of Kubernetes, installation steps and the related content of the whole Kubernetes system, I believe we read this series, To have a deeper understanding of Kubernetes.

This paper introduces Pod container health check related content, configuration method and experimental test, the experimental environment is Kubernetes 1.11, the method of building reference kubeadm installation Kubernetes V1.11.1 cluster

0. What are Container Probes

Each Node has Kubelet. The Container Probe is performed periodically by Kubelet.

Kubelet performs the action of checking by calling the Pod container’s Handler, which comes in three types.

  • ExecAction, to execute a specific command in the container. Command exit returns 0 indicating success
  • TCPSocketAction: Checks TCP based on the container IP address and the specified port. If the port is opened, the port succeeds
  • HTTPGetAction, which initiates an HTTP request based on the container IP, port, and access path. Success is achieved if the return code is between 200 and 400. Each check action may have three return states.
  • Success: indicates that the health check is passed
  • Failure: failed a health check
  • Unknown: indicates that the check action fails

Liveness and Readiness are two ways to detect the operation of containers within a Pod when creating a Pod. Liveness is used to check the viability of applications in the container. Failure to check kills the container process, and the decision to restart the container depends on the Pod restart policy. Readiness Checks whether applications in the container can provide services properly. If the detection fails, the Endpoint Controller deletes the IP address of the Pod from the service.

1. Application scenarios

We all know that Kubernetes maintains the Pod state and number, so if you just want to keep the Pod container restarted after a failure, there is no need to add a health check, just configure the Pod restart policy properly. A more suitable scenario for health check is one in which containers need to be killed and restarted according to the check results. In addition, some containers need to load some data before providing services formally. In this case, Readiness can be used to check whether these actions are completed.

2. Liveness check instances

2.1 the Container Exec

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-exec
spec:
  containers:
  - name: liveness
    image: docker.io/alpine
    args:
    - /bin/sh
    - -c
    - touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600
    livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5
Copy the code

This example creates a container that checks for the existence of a file to determine whether the container is working properly. After the container runs for 30 seconds, the file is deleted, so the container fails to check LIVENESS and restarts the container.

2.2 HTTP Health Check

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
    app: httpd
  name: liveness-http
spec:
  containers:
  - name: liveness
    image: docker.io/httpd
    ports:
    - containerPort: 80
    livenessProbe:
      httpGet:
        path: /index.html
        port: 80
        httpHeaders:
        - name: X-Custom-Header
          value: Awesome
      initialDelaySeconds: 5
      periodSeconds: 5
Copy the code

This example creates an Apache server that accesses index to determine whether the service is alive. Manually deleting the file can cause the check to fail and restart the container.

[root@devops-101 ~]# kubectl exec -it liveness-http /bin/sh
# 
# ls
bin  build  cgi-bin  conf  error  htdocs  icons  include  logs	modules
# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 11:39 ?        00:00:00 httpd -DFOREGROUND
daemon       6     1  0 11:39 ?        00:00:00 httpd -DFOREGROUND
daemon       7     1  0 11:39 ?        00:00:00 httpd -DFOREGROUND
daemon       8     1  0 11:39 ?        00:00:00 httpd -DFOREGROUND
root        90     0  0 11:39 ?        00:00:00 /bin/sh
root        94    90  0 11:39 ?        00:00:00 ps -ef
#              
# cd /usr/local/apache2
# ls
bin  build  cgi-bin  conf  error  htdocs  icons  include  logs	modules
# cd htdocs
# ls
index.html
# rm index.html
# command terminated with exit code 137
[root@devops-101 ~]# kubectl describe pod liveness-http
Events:
  Type     Reason     Age               From                 Message
  ----     ------     ----              ----                 -------
  Normal   Scheduled  1m                default-scheduler    Successfully assigned default/liveness-http to devops-102
  Warning  Unhealthy  8s (x3 over 18s)  kubelet, devops-102  Liveness probe failed: HTTP probe failed with statuscode: 404
  Normal   Pulling    7s (x2 over 1m)   kubelet, devops-102  pulling image "docker.io/httpd"
  Normal   Killing    7s                kubelet, devops-102  Killing container with id docker://liveness:Container failed liveness probe.. Container will be killed and recreated.
  Normal   Pulled     1s (x2 over 1m)   kubelet, devops-102  Successfully pulled image "docker.io/httpd"
  Normal   Created    1s (x2 over 1m)   kubelet, devops-102  Created container
  Normal   Started    1s (x2 over 1m)   kubelet, devops-102  Started container
Copy the code

2.3 the TCP Socket

This way to determine whether a TCP connection is alive or not, Pod choreography example.

apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
    app: node
  name: liveness-tcp
spec:
  containers:
  - name: goproxy
    image: Docker. IO/googlecontainer/goproxy: 0.1
    ports:
    - containerPort: 8080
    readinessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10
    livenessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20
Copy the code

(3) Readiness

Another type of Readiness configuration is similar to LIVENESS, where you change livenessProbe to readinessProbe.

4. Set parameters

We can get throughkubectl explainCommand to view the specific configuration properties, here again briefly list the main properties.

  • InitialDelaySeconds: Checks the start time of execution, calculated from completion of container startup

  • PeriodSeconds: Period of the check execution. The default value is 10 seconds and the minimum value is 1 second

  • TimeoutSeconds: specifies the check timeout duration. The default value is 1 second and the minimum value is 1 second

  • SuccessThreshold: Indicates the number of consecutive successful checks since the last check failure. The default value is 1

  • FailureThreshold: indicates the threshold of check failures (consecutive failures) since the last check succeeded. The default value is 1

  • The properties of httpGet

    • Host: indicates the host name or IP address
    • Scheme: Link type, HTTP or HTTPS. The default is HTTP
    • Path: request path
    • HttpHeaders: Custom request headers
    • Port: request port

The resources

  1. Kubernetes 201
  2. Container Probes
  3. Kubernetes Task Probe
  4. Configure Liveness and Readiness Probes
  5. package handler
  6. Kubernetes Reference Probe