First, deploy the Logging Agent on the Node and forward the log files to the back-end storage for storage

As shown in figure 1.1

  • It is not difficult to see that the core here lies in the logging agent, which generally runs on the node in a DaemonSet mode, and then mounts the container log directory on the host. Finally, logging-agent forwards the logs.
  • For example, we can use the +Fluentd+ project as the +logging-agent on the host and forward the log to the remote +ElasticSearch+ for future retrieval. Kubernetes deployment will automatically enable Logrotate for you, and automatically roate log files when they reach 10MB.
  • As you can see, the biggest advantage of deploying a Logging Agent on a Node is that only one agent needs to be deployed and there is no intrusion into the application or pod. So this is the most common one in the area.
  • However, the subsidy cost of this solution is that it requires the application to output logs directly to the container’s STdout and stderr

Second, a treatment of a special case

  • When the container logs can only be exported to certain files, we can use a Sidecar container to re-export the log files to stdout and stderr of sidecar, as shown in Figure 1.2

  • Now my pod app has only one container, and it outputs logs to the /var/log/1.log and 2.log files in the container. The YAML file for this pod looks like this: 
    apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/1.log;
        echo "$(date) INFO $i" >> /var/log/2.log;
        i=$((i+1));
        sleep 1;
      done
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  volumes:
  - name: varlog
    emptyDir: {}
Copy the code
  • In this case, you will not see any application logs using kubect. And plan one, the most common one, is also impossible to use. At this point, we can add two sidecar containers to the POD to output the contents of the two log files as stout and stderr, respectively. The YAML file is written as follows: 
    apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/1.log;
        echo "$(date) INFO $i" >> /var/log/2.log;
        i=$((i+1));
        sleep 1;
      done
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: count-log-1
    image: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/1.log']
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  - name: count-log-2
    image: busybox
    args: [/bin/sh, -c, 'tail -n+1 -f /var/log/2.log']
    volumeMounts:
    - name: varlog
      mountPath: /var/log
  volumes:
  - name: varlog
    emptyDir: {}
Copy the code

    * Because sidecar shares Volume with the main container, the sidecar solution does not have a high secondary performance loss, and therefore consumes more CPU and memory. However, there are two identical log files on the host: one written by the application itself; The other is the JSON file corresponding to Sidecar’s STdout and DTderr. This is a huge waste of disk. Therefore, unless absolutely necessary or the application container cannot be modified at all, it is recommended that you go straight to Plan 1 or plan 3.

In solution 3, log files are directly sent to the remote storage after the Sidecar container

  • This is equivalent to putting the Logging Agent in Scenario 1 into the pod. As shown in figure 1.3:

  • In this scenario, your application can export logs directly to a fixed file instead of stdout, your logging-Agent can use FluentD, and the backend storage can be ElasticSearch. Instead, the input source for FluentD is the application log file. Fluentd input source configuration is stored in a ConfigMap, as follows: 
    apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config
data:
  fluentd.conf: |
    <source>
      type tail
      format none
      path /var/log/1.log
      pos_file /var/log/1.log.pos
      tag count.format1
    </source>
    
    <source>
      type tail
      format none
      path /var/log/2.log
      pos_file /var/log/2.log.pos
      tag count.format2
    </source>
    
    <match **>
      type google_cloud
    </match>
Copy the code
  • A Fluentd container is used as a sidecar to forward 1. Log and 2. Log generated by the application to ElasticSearch. 
    apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args:
    - /bin/sh
    - -c
    - >
      i=0;
      while true;
      do
        echo "$i: $(date)" >> /var/log/1.log;
        echo "$(date) INFO $i" >> /var/log/2.log;
        i=$((i+1));
        sleep 1;
      done
    volumeMounts:
    - name: varlog
      mountPath: /var/logIO /fluentd-gcp:1.30 env: -name: FLUENTD_ARGS value: -c /etc/fluentd-config/fluentd.conf volumeMounts: - name: varlog mountPath: /var/log
    - name: config-volume
      mountPath: /etc/fluentd-config
  volumes:
  - name: varlog
    emptyDir: {}
  - name: config-volume
    configMap:
      name: fluentd-config
Copy the code
  • As shown above, the input source used by the Fluentd container is specified by referring to the ConfigMap we wrote earlier. A Projected Volume is used to hang ConfigMap into the Pod.
  • This solution is the easiest to deploy and the most host-friendly, but the Sidecar container is likely to consume a lot of resources and even drag down the application container. Since the logs are not output to STdout, you cannot see the log information through kubectl logs.

conclusion

If you want to log your application to stDout and STderr, then you can deploy logging-agent in Suzhushu.com to process your application logs in a centralized way. This solution is not only simple, but also easy to use. Kubectl logs are still available and are officially recommended.