Introduction to the

This article mainly introduces the log collection application in k8s scheme, in the operation of the application log, generally need to collect to store a centralized log management system, can be convenient to analyze the log statistics, monitoring, even for machine learning, intelligent analysis application system problems, timely repair existing problems in the application.

Generally, the following log output modes are used in the K8S cluster

  • Directly follow docker’s official advice to output logs to standard output or standard error output
  • Outputs logs to the specified directory in the container
  • The application directly sends logs to the log collection system

This article deploys a combination of the above log collection solutions.

This section describes log collection components

  • Elastisearch stores collected logs
  • Kibana visual collection of logs
  • Logstash summary processing logs are sent to elastiSearch storage
  • Filebeat reads containers or applies log file processing to ElastiSearch or LogStash and can also be used to aggregate logs
  • Fluentd reads containers or applies log file processing to ElastiSearch and can also be used to summarize logs
  • Fluent-bit reads container or applies log file processing to ElastiSearch or FluentD

The deployment of

In this experiment, 3 virtual machines were used as K8S cluster, and each virtual machine had 3G memory

Preparations before deployment

# pull file
git clone https://github.com/mgxian/k8s-log.git
cd k8s-log
git checkout v1

Create a logging namespace
kubectl apply -f logging-namespace.yaml
Copy the code

Deploy elastisearch

# This deployment uses StatefulSet but does not use PV for persistent data stores
Data will be lost after pod restart. Production environment must use PV persistence to store data

# deployment
kubectl apply -f elasticsearch.yaml

# check status
kubectl get pods,svc -n logging -o wide

Wait for all pods to become running
# Access test
If all tests return data, the deployment is successful
kubectl run curl -n logging --image=radial/busyboxplus:curl -i --tty
nslookup elasticsearch-logging
curl 'http://elasticsearch-logging:9200/_cluster/health? pretty'
curl 'http://elasticsearch-logging:9200/_cat/nodes'
exit

# Cleanup test
kubectl delete deploy curl -n logging
Copy the code

Deploy kibana

# deployment
kubectl apply -f kibana.yaml

# check status
kubectl get pods,svc -n logging -o wide

# Access test
# Browser visit the following output address to see the Kibana interface represents normal
# 11.11.11.112 is the IP address of a node in the cluster
KIBANA_NODEPORT=$(kubectl get svc -n logging | grep kibana-logging | awk '{print $(NF-1)}' | awk -F[:/] '{print $2}')
echo "http://11.11.11.112:$KIBANA_NODEPORT/"
Copy the code

Deploy FluentD to collect logs

Fluentd is deployed as daemoset
Start fluentd container on each node, collect k8S component, Docker and container logs

Label each node that needs to start FluentD
# kubectl label node lab1 beta.kubernetes.io/fluentd-ds-ready=true
kubectl label nodes --all beta.kubernetes.io/fluentd-ds-ready=true

# deployment
kubectl apply -f fluentd-es-configmap.yaml
kubectl apply -f fluentd-es-ds.yaml

# check status
kubectl get pods,svc -n logging -o wide
Copy the code

Kibana reviews logs

Create index fluentd -k8S -*, you may need to wait a few minutes to see the index and data due to the need to pull the image to start the container

See the log

Apply log collection tests

Apply log output to standard output tests

Start test log output
kubectl run echo-test --image=radial/busyboxplus:curl -- sh -c 'count=1; while true; do echo log to stdout $count; sleep 1; count=$(($count+1)); done'

# check status
kubectl get pods -o wide

Command line view log
ECHO_TEST_POD=$(kubectl get pods | grep echo-test | awk '{print $1}')
kubectl logs -f $ECHO_TEST_POD

# refresh Kibana to see if any new logs are entered
Copy the code

Export application logs to the specified directory of the container (collected by FileBeat)

# deployment
kubectl apply -f log-contanier-file-filebeat.yaml

# check
kubectl get pods -o wide
Copy the code

Add index filebeat-k8s-* to view logs

Output application logs to the specified directory of the container (Fluent-bit collection)

# deployment
kubectl apply -f log-contanier-file-fluentbit.yaml

# check
kubectl get pods -o wide
Copy the code

Add index fluentbit-k8s-* to view logs

The application directly sends logs to the log system

Export logs to ElasticSearch

# deployment
kubectl apply -f log-contanier-es.yaml

# check
kubectl get pods -o wide
Copy the code

Add index k8S-app -* to view logs

Clean up the

kubectl delete -f log-contanier-es.yaml
kubectl delete -f log-contanier-file-fluentbit.yaml
kubectl delete -f log-contanier-file-filebeat.yaml
kubectl delete deploy echo-test
Copy the code

Summary of log collection system

The chart in this section uses ELK technology stack to show the illustration. In practice, EFK technology stack can be used, fluentd can be used instead of Logstash, and Fluent-bit can be used instead of FileBeat. Fluentd is recommended to replace Logstash because fluentd has better memory footprint and performance. Fluent-bit and FileBeat have similar performance and memory footprint

K8s Universal solution for collecting cluster logs

  • Log usage of related components in the clusterfluentd/filebeatcollect
  • Log usage that applies output to standard output or standard error outputfluentd/filebeatcollect
  • Apply output to the specified file log in the containerfluent-bit/filebeatcollect

General purpose log collection system

Common log collection system architecture

Architecture description

  • Log collection and location are decoupled
  • Due to the queuing between the collection and processing processes, the analysis and processing nodes are allowed enough time to digest the log data to avoid being overwhelmed when the number of logs surges
  • Log analysis and processing nodes can scale dynamically

Heavy traffic log collection system

Architecture diagram of the heavy-traffic log collection system

Architecture description

  • When the log traffic is heavy, if each log collection node writes data in a queue, the queue will be stressed due to many scattered connections and write requests. If all logs are sent to the Logstash collection node and then written to the queue centrally, the queue pressure will be reduced.

Application Log Collection Experiment (ELK Technology Stack)

This section uses nginx log collection as an example to perform log collection and analysis experiments. Elasticsearch and Kibana applications created in previous experiments are reused. The experiment adopts the architecture of high traffic log collection

Deploy the Redis queue
# deployment
kubectl apply -f redis.yaml

# check
kubectl get pods -n logging
Copy the code
Deploy indexer to analyze logs
# deployment
kubectl apply -f logstash-indexer.yaml

# check
kubectl get pods -n logging
Copy the code
Deploy the shipper to centralize logs
# deployment
kubectl apply -f logstash-shipper.yaml

# check
kubectl get pods -n logging
Copy the code
Deploy nginx test log collection
# deployment
kubectl apply -f nginx-log-filebeat.yaml

# check
kubectl get pods
Copy the code
Continuous access to nginx generates logs
# deployment
kubectl run curl-test --image=radial/busyboxplus:curl -- sh -c 'count=1; while true; Do curl -s-h "User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, Like Gecko) Chrome/44.0.2403.89 Safari/537.36 $count" http://nginx-log-filebeat/ >/dev/null; sleep 1; count=$(($count+1)); done'

# check
kubectl get pods
Copy the code
Visit Kibana to view the logs

Add index k8S-logging-elk -* Due to the slow start of logstash, it may take several minutes to see the data

Clean up the
kubectl delete -f redis.yaml
kubectl delete -f logstash-indexer.yaml
kubectl delete -f logstash-shipper.yaml
kubectl delete -f nginx-log-filebeat.yaml
kubectl delete deploy curl-test
Copy the code

Application Log Collection Experiment (EFK Technology Stack)

Because Fluentd does not officially provide support for REDis queues, redis queues are removed in this experiment.

Deploy indexer to analyze logs
# deployment
kubectl apply -f fluentd-indexer.yaml

# check
kubectl get pods -n logging
Copy the code
Deploy the shipper to centralize logs
# deployment
kubectl apply -f fluentd-shipper.yaml

# check
kubectl get pods -n logging
Copy the code
Deploy nginx test log collection
# deployment
kubectl apply -f nginx-log-fluentbit.yaml

# check
kubectl get pods
Copy the code
Continuous access to nginx generates logs
# deployment
kubectl run curl-test --image=radial/busyboxplus:curl -- sh -c 'count=1; while true; Do curl -s-h "User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, Like Gecko) Chrome/44.0.2403.89 Safari/537.36 $count" http://nginx-log-fluentbit/ >/dev/null; sleep 1; count=$(($count+1)); done'

# check
kubectl get pod
Copy the code
Visit Kibana to view the logs

Add the index k8s – logging – efk – *

Clean up the
kubectl delete -f fluentd-indexer.yaml
kubectl delete -f fluentd-shipper.yaml
kubectl delete -f nginx-log-fluentbit.yaml
kubectl delete deploy curl-test
Copy the code

Application Log Visualization

Deploy the components required for log collection

Deploy indexer Shipper Fluentbit
kubectl apply -f fluentd-indexer.yaml
kubectl apply -f fluentd-shipper.yaml
kubectl apply -f nginx-log-fluentbit.yaml

# check
kubectl get pods
kubectl get pods -n logging
Copy the code

Simulated user access

# deployment
kubectl apply -f web-load-gen.yaml

# check
kubectl get pods
Copy the code

Visit Kibana to view the logs

Add the index k8s – logging – efk – *

Create a chart

Create a Search

Need to be used before production Visualize

Searches logs by the specified criteria

Save the Search

Create the Visualize

A pre-created Visualize can be added to the Dashboard

Select Visualize

Select Visualize type

Select the Search saved using the steps above

Select the specified bucket

Select the code field for statistics

Save the Visualize

Create multiple Visualize using the previous steps

Create a Dashboard

Select Create Dashboard

Add Visualize to Dashboard

Save the Dashboard

Edit to adjust position and size

Final chart presentation

If you can use the import function directly in the Saved Ojects TAB of Managerment for a quick experience, import the k8S-Kibana-all. json file in the k8S-log download directory for this experiment

Reference documentation

  • Kubernetes. IO/docs/concep…
  • Banzaicloud.com/blog/k8s-lo…
  • Docs.fluentd.org/v0.12/artic…
  • Jimmysong. IO/kubernetes -…
  • Github.com/kubernetes/…
  • www.elastic.co/blog/shippi…
  • Github.com/elastic/bea…
  • www.elastic.co/guide/en/be…
  • www.elastic.co/guide/en/be…
  • Github.com/fluent/flue…
  • Github.com/fluent/flue…
  • Github.com/fluent/flue…
  • www.docker.elastic.co/
  • Fluentbit. IO/documentati…
  • Docs.fluentd.org/v1.0/articl…
  • www.elastic.co/guide/en/lo…