1. The overall solution of unified log management
Application and system logs provide insight into what is happening within the Kubernetes cluster and are useful for debugging problems and monitoring cluster activity. For most applications, there will be some kind of logging mechanism. Therefore, most container engines are also designed to support some kind of logging mechanism. The simplest and most acceptable method of logging for containerized applications is to write log content to standard output and standard error stream.
However, the native functionality provided by the container engine or runtime is usually insufficient to support a complete logging solution. For example, if a container crashes, a Pod is expelled, or a Node dies, application stakeholders may still need access to the application’s logs. Therefore, logs should have separate storage and lifecycle independent of Node, Pod, or container, a concept known as cluster-level logging. Cluster-level logging requires a separate back end to store, analyze, and query logs. Kubernetes itself does not provide a native storage solution for log data, but many existing logging solutions can be integrated into the Kubernetes cluster. In Kubernetes, there are three levels of logging:
-
Based on the log
-
Node level logs
-
Cluster level logging architecture
1.1 Basic Logs
Kubernetes basic logging is to output log data to the standard output stream. You can use the kubectl logs command to obtain container log information. If there are multiple containers in the Pod, you can specify which container’s logs to access by appending the container name to the command. For example, if you have a Pod named Nexus3-F5b7fc55C-hq5v7 under the devops namespace in the Kubernetes cluster, you can retrieve logs by using the following command:
$ kubectl logs nexus3-f5b7fc55c-hq5v7 --namespace=devops
Copy the code1.2 Node level Logs
Everything that container applications write to stdout and Stderr is handled and redirected by the container engine. For example, the Docker container engine redirects these two streams to the logging driver, which in Kubernetes is configured to write files in JSON format. The Docker JSON logging driver treats each line as a separate message. When using the Docker logging driver, multi-line messages are not supported and therefore need to be processed at the logging agent level or higher.
By default, kubectl keeps a terminated container and its logs if the container is restarted. If a Pod is expelled from Node, all corresponding containers in the Pod are expelled, along with their logs. An important consideration in logging at the Node level is to implement log rotation so that logs do not consume all available storage on Node. Kubernetes is not currently responsible for rotation logging and the deployment tool should build a solution to solve this problem.
There are two types of system components in Kubernetes: components running in containers and components not running in containers. Such as:
-
The Kubernetes scheduler and kube-Proxy run in the container.
-
Kubelet and container runtimes, such as Docker, do not run in containers.
On machines with Systemd, kubelet and container runtime write journaId. If systemd does not exist, they write.log files to the /var/log directory. The system components in the container always bypass the default logging mechanism and write to the /var/log directory using the GOLg log library. You can find conventions in logging for which components record severity in the development document.
Similar to container logs, system component logs in the /var/log directory should be rotated. These logs are configured to be rotated daily by Logrotate or when the size exceeds 100MB.
1.3 Cluster-level log Architecture
Kubernetes itself does not provide a native solution for cluster level logging, but there are several common approaches that can be taken:
-
Use node-level logging agents that run on each Node;
-
Include a Sidecar for logging in the application Pod.
-
Push logs directly from within the application to the back end.
All things considered, this article uses cluster level logging by including Node level logging agents on each Node. A logging agent exposes logs or pushes logs to a specialized tool on the back end. Typically, logging-Agent is a container that has access to the log files of all application containers on the Node.
Because logging must be run on every Node, it is often used as a copy of DaemonSet, or as a dedicated native process on a manifest Pod or Node. However, the latter two methods will be abandoned in the future. Using Node level logging agents is the most common and popular approach for Kubernetes clustering, as it creates only one agent per Node and does not require any changes to the applications running on the nodes. However, Node level logging only applies to application standard output and standard error.
Kubernetes itself does not specify a logging agent, but two alternative logging agents are available packaged with Kubernetes versions: Stackdriver and Elasticsearch with Google Cloud platform, both of which use custom configured FluentD as agent on Node. In this solution, Fluentd is used for logging-agent, Elasticsearch is used for Logging Backend, and Grafana is used for front-end display. Fluentd is used as logging-agent to collect logs and push them to the backend Elasticsearch. Grafana takes the logs from Elasticsearch and displays them uniformly.
2. Install the unified log management component
In this paper, the use of Node logging agent aspects of Kubernetes unified log management, related tools are used:
-
Logging-agent: The logging agent is used to fetch log information from the container, using Fluentd.
-
Logging-backend: The log Backend processes logs pushed by the log Logging agent. The log Backend uses Elasticsearch.
-
Log display: Log display is used to display unified log information to users, using Kibana.
Elasticsearch add-on is available in Kubernetes. This component includes Elasticsearch, Fluentd, and Kibana. Elasticsearch is a search engine that stores logs and allows queries. Fluentd gets the log message from Kubernetes and sends it to Elasticsearch. Kibana is a graphical interface for viewing and querying logs stored in Elasticsearch. Requirements for the environment before installation and deployment are as follows:
2.1 Installing and Deploying Elasticsearch
Elasticsearch is an open source search and data analysis engine based on Apache Lucene(TM). Elasticsearch is developed in Java and uses Lucene as its core for all indexing and searching functions. It aims to make full-text search easy by hiding Lucene’s complexity with a simple RESTful API. Elasticsearch is more than just Lucene and full text search, it also provides the following capabilities:
-
Distributed real-time file storage where every field is indexed and searchable;
-
Distributed real-time analytical search engine;
-
Scalable to hundreds of servers, processing petabytes of structured or unstructured data.
Elasticsearch contains multiple indexes, and each Index can contain multiple types. Each of these different types can store multiple documents, and each Document has multiple properties. Like a database in a traditional relational database, an index is a place to store relational documents. Elasticsearch uses a standard RESTful API and JSON. In addition, it has built and maintained clients in many other languages, such as Java, Python,.NET, and PHP.
The YAML configuration file for Elasticsearch defines a ServiceAccount named ElasticSearch-Logging and grants it access to namespaces, services, and endpoints. Deploy Elasticsearch as StatefulSet.
# RBAC authn and authz
apiVersion: v1
kind: ServiceAccount
metadata:
name: elasticsearch-logging
namespace: kube-system
labels:
k8s-app: elasticsearch-logging
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: elasticsearch-logging
labels:
k8s-app: elasticsearch-logging
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
- ""
resources:
- "services"
- "namespaces"
- "endpoints"
verbs:
- "get"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: kube-system
name: elasticsearch-logging
labels:
k8s-app: elasticsearch-logging
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
name: elasticsearch-logging
namespace: kube-system
apiGroup: ""
roleRef:
kind: ClusterRole
name: elasticsearch-logging
apiGroup: ""
---
# Elasticsearch deployment itself
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: elasticsearch-logging
namespace: kube-system
labels:
k8s-app: elasticsearch-logging
Version: v6.2.5
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
serviceName: elasticsearch-logging
replicas: 2
selector:
matchLabels:
k8s-app: elasticsearch-logging
Version: v6.2.5
template:
metadata:
labels:
k8s-app: elasticsearch-logging
Version: v6.2.5
kubernetes.io/cluster-service: "true"
spec:
serviceAccountName: elasticsearch-logging
containers:
- image: k8s. GCR. IO/elasticsearch: v6.2.5
name: elasticsearch-logging
resources:
# need more cpu upon initialization, therefore burstable class
limits:
cpu: 1000m
requests:
cpu: 100m
ports:
- containerPort: 9200
name: db
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
volumeMounts:
- name: elasticsearch-logging
mountPath: /data
env:
- name: "NAMESPACE"
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumes:
- name: elasticsearch-logging
emptyDir: {}
# Elasticsearch requires vm.max_map_count to be at least 262144.
# If your OS already sets up this number to a higher value, feel free
# to remove this init container.
initContainers:
- image: alpine: 3.6
command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"]
name: elasticsearch-logging-init
securityContext:
privileged: true
Copy the codeTo deploy Elasticsearch, run the following command:
$ kubectl create -f {path}/es-statefulset.yaml
Copy the codePort 9200 is exposed for Elasticsearch proxy service.
apiVersion: v1
kind: Service
metadata:
name: elasticsearch-logging
namespace: kube-system
labels:
k8s-app: elasticsearch-logging
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "Elasticsearch"
spec:
ports:
- port: 9200
protocol: TCP
targetPort: db
selector:
k8s-app: elasticsearch-logging
Copy the codeRun the following command to deploy the Elasticsearch proxy service:
$ kubectl create -f {path}/es-service.yaml
Copy the code2.2 Installing and Deploying Fluentd
Fluentd is an open source data collector that can uniformly collect and consume data to better use and understand data. Fluentd structures the data as JSON to handle log data uniformly, including collection, filtering, caching, and output. Fluentd is a plug-in architecture, including input plug-ins, output plug-ins, filtering plug-ins, parsing plug-ins, formatting plug-ins, caching plug-ins, and storage plug-ins. You can expand and better use Fluentd through plug-ins.
The overall processing process of Fluentd is as follows: Obtain data through Input plug-in, filter, parse, format, and cache data through Engine, and Output data to a specific terminal through Output plug-in.
In this article, Fluentd acts as a logging-agent to collect logs and push the collected days to the backend Elasticsearch. For Kubernetes, DaemonSet ensures that all (or some) nodes will run a Pod copy. So Fluentd was deployed as DaemonSet, which will generate a POD on each node to read logs generated by kubelet, container runtime, and container and send them to Elasticsearch. To make Fluentd can work, each Node must mark beta. Kubernetes. IO/Fluentd – ds – ready = true.
Here is Fluentd’s ConfigMap configuration file, which defines the log data sources Fluentd gets and outputs those log data to Elasticsearch.
kind: ConfigMap
apiVersion: v1
metadata:
Name: fluentd - es - config - v0.1.4
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: Reconcile
data:
system.conf: |-
<system>
root_dir /tmp/fluentd-buffers/
</system>
containers.input.conf: |-
<source>
@id fluentd-containers.log
@type tail
path /var/log/containers/*.log
pos_file /var/log/es-containers.log.pos
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag raw.kubernetes.*
read_from_head true
<parse>
@type multi_format
<pattern>
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%NZ
</pattern>
<pattern>
format /^(? <time>.+) (? <stream>stdout|stderr) [^ ]* (? <log>.*)$/
time_format %Y-%m-%dT%H:%M:%S.%N%:z
</pattern>
</parse>
</source>
# Detect exceptions in the log output and forward them as one log entry.
<match raw.kubernetes.**>
@id raw.kubernetes
@type detect_exceptions
remove_tag_prefix raw
message log
stream stream
multiline_flush_interval 5
max_bytes 500000
max_lines 1000
</match>
output.conf: |-
# Enriches records with Kubernetes metadata
<filter kubernetes.**>
@type kubernetes_metadata
</filter>
<match **>
@id elasticsearch
@type elasticsearch
@log_level info
include_tag_key true
host elasticsearch-logging
port 9200
logstash_format true
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.system.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 2M
queue_limit_length 8
overflow_action block
</buffer>
</match>
Copy the codeRun the following command to create Fluentd ConfigMap:
$ kubectl create -f {path}/fluentd-es-configmap.yaml
Copy the codeThe YAML configuration file for Fluentd itself is as follows:
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd-es
namespace: kube-system
labels:
k8s-app: fluentd-es
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: fluentd-es
labels:
k8s-app: fluentd-es
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
- ""
resources:
- "namespaces"
- "pods"
verbs:
- "get"
- "watch"
- "list"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: fluentd-es
labels:
k8s-app: fluentd-es
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
name: fluentd-es
namespace: kube-system
apiGroup: ""
roleRef:
kind: ClusterRole
name: fluentd-es
apiGroup: ""
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
Name: fluentd - es - v2.2.0
namespace: kube-system
labels:
k8s-app: fluentd-es
Version: v2.2.0
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
k8s-app: fluentd-es
Version: v2.2.0
template:
metadata:
labels:
k8s-app: fluentd-es
kubernetes.io/cluster-service: "true"
Version: v2.2.0
# This annotation ensures that fluentd does not get evicted if the node
# supports critical pod annotation based priority scheme.
# Note that this does not guarantee admission on the nodes (#40573).
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
spec:
priorityClassName: system-node-critical
serviceAccountName: fluentd-es
containers:
- name: fluentd-es
Image: k8s. GCR. IO/fluentd - elasticsearch: v2.2.0
env:
- name: FLUENTD_ARGS
value: --no-supervisor -q
resources:
limits:
memory: 500Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: config-volume
mountPath: /etc/fluent/config.d
nodeSelector:
beta.kubernetes.io/fluentd-ds-ready: "true"
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: config-volume
configMap:
Name: fluentd - es - config - v0.1.4
Copy the codeRun the following command to deploy Fluentd:
$ kubectl create -f {path}/fluentd-es-ds.yaml
Copy the code2.3 Installing and Deploying Kibana
Kibana is an open source analytics and visualization platform designed to work with Elasticsearch. With Kibana, you can search, view, and interact with Elasticsearch data. You can analyze and visualize data using various charts, tables, and maps. The YAML deployed for Kibana is as follows. Specify the Elasticsearch service to obtain log data by using the environment variable ELASTICSEARCH_URL. http://elasticsearch-logging:9200, ElasticSearch. Cattle-logging is the name of elasticSearch’s proxy service in Kubernetes. In the Fluented configuration file, there are some key instructions:
-
The source directive determines the input source.
-
The match directive determines the output target.
-
The filter directive identifies the event processing pipeline.
-
The system directive sets system-wide configurations.
-
The label directive groups output and filters for internal routing
-
The @include directive contains other files.
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana-logging
namespace: kube-system
labels:
k8s-app: kibana-logging
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
replicas: 1
selector:
matchLabels:
k8s-app: kibana-logging
template:
metadata:
labels:
k8s-app: kibana-logging
annotations:
seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
spec:
containers:
- name: kibana-logging
Image: docker. Elastic. Co/kibana/kibana - oss: 6.2.4
resources:
# need more cpu upon initialization, therefore burstable class
limits:
cpu: 1000m
requests:
cpu: 100m
env:
- name: ELASTICSEARCH_URL
value: http://elasticsearch-logging:9200
ports:
- containerPort: 5601
name: ui
protocol: TCP
Copy the codeRun the following command to deploy the Kibana proxy service:
$ kubectl create -f {path}/kibana-deployment.yaml
Copy the codeThe following proxy service YAML configuration file for Kibana. The proxy service type is NodePort.
apiVersion: v1
kind: Service
metadata:
name: kibana-logging
namespace: kube-system
labels:
k8s-app: kibana-logging
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "Kibana"
spec:
type: NodePort
ports:
- port: 5601
protocol: TCP
targetPort: ui
selector:
k8s-app: kibana-logging
Copy the codeRun the following command to deploy the Kibana proxy service:
$ kubectl create -f {path}/kibana-service.yaml
Copy the code3. Log data display
Run the following command to obtain the exposed ports of Kibana:
$ kubectl get svc --namespace=kube-system
Copy the code
As you can see from the output, the exposed port of Kibana is 30471, so you can access Kibana outside the Kubernetes cluster by going to http://{NodeIP}:30471.
Click “Discover” to see the log information retrieved from the container in real time:
About the author: Ji Xiangyuan, product manager of Beijing Shenzhou Aerospace Software Technology Co., LTD. The copyright of this article belongs to the author.
The resources
1. Logging Architecture
https://kubernetes.io/docs/concepts/cluster-administration/logging/
Kubernetes Logging with Fluentd
https://docs.fluentd.org/v0.12/articles/kubernetes-fluentd
3. Quickstart Guide
https://docs.fluentd.org/v0.12/articles/quickstart
Fluentd-elasticsearch (fluentd-ElasticSearch)
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch
5. Elasticsearch address:
https://www.elastic.co/products/elasticsearch
6. What is Fluentd? Address:
https://www.fluentd.org/architecture
7. Configuration File Syntax
https://docs.fluentd.org/v1.0/articles/config-file
8. A Practical Introduction to Elasticsearch
https://www.elastic.co/blog/a-practical-introduction-to-elasticsearch