Wechat tooling good: operation and development story, author: Double_Dong

The main content

  • 1 ELK concept

  • 2 K8S Indicates the logs to be collected

  • 3 ELK Stack log scheme

  • 4 How do I collect logs in a container

  • 5 Deployment Procedure

Prepare the environment

For a properly running K8S cluster, kubeadm can be installed or deployed in binary mode

1 ELK concept

ELK is the initials of Elasticsearch, Logstash, and Kibana. It’s also known as the Elastic Stack. Elasticsearch is a distributed near-real-time search platform framework based on Lucene and interactive in a Restful way. Big data full-text search engines such as Baidu and Google can use Elasticsearch as the underlying support framework, which shows that Elasticsearch provides strong search capabilities. In the market, we call Elasticsearch ES for short in many cases. Logstash is the central data flow engine, ELK from different target (documents/data storage/MQ) to collect data of different formats, support after filtered output to different destination (file/MQ/redis/elasticsearch, kafka, etc.). Kibana allows elasticSearch data to be displayed in a user-friendly page, providing real-time analysis. With the brief introduction to ELK above, we have seen the capabilities of each open source framework that ELK literally encompasses. Many developers on the market can consistently refer to ELK as a log analysis architecture stack, but in fact ELK is not only applicable to log analysis, it can support any other data analysis and collection scenarios, log analysis and collection is more representative. It’s not unique. This tutorial focuses on building a production-level log analysis platform using ELK. Official Website:www.elastic.co/cn/products…

2 Log management platform

In the past, when all of our components were deployed to a single server, the need for a log management platform might not have been so strong. We could simply log into a server and use shell commands to view system logs and quickly locate problems. With the development of the Internet, the Internet has penetrated into all areas of life, and the number of users using the Internet is also increasing. Individual applications can no longer support the concurrent amount of huge users, especially in a populous country like China. Take apart then apply monomer, through the use of horizontal scaling to support large users is imminent, micro service concept was born in such a stage, in the era of micro service in Internet technology, a single application be broken up into multiple application, each application cluster deployment for load balancing, if a business system error, Development or operation and maintenance personnel still log in to each login server as a single application in the past to view logs and locate problems. The efficiency of solving online problems is predictable. The construction of log management platform is extremely important. Logstash collects each server log file and filters it to Kafka or Redis according to the defined regular template. Then another Logstash reads the log from Kafka or Redis and stores it in the index of ElasticSearch. Finally, it is presented to developers or operations staff through Kibana for analysis. This greatly improves the efficiency of o&M line problems. In addition, the collected logs can be used for big data analysis to obtain more valuable data for high-level decision-making.

3 ELK Stack log collection scheme in K8S

  • Solution 1: Deploy a log collector on Node

Using DaemonSet way to give each node deployment log collection application logging – agent and then use this agent on this node node/var/log and/var/lib/docker/containers/two directory to capture the log Alternatively, mount the container log directory in Pod to the host unified directory for collection

Because of the way to use stdout, only need to collect each container on the host log in/var/log and/var/lib/docker/containers (directory according to the docker info of dir, modified container will log into JSON format, Is a function of configuration in Docker.)

  • Option 2: Add a dedicated log collection container to Pod

Add a log collection container to each Pod running the application and share the log directory using emtyDir for the log collector to read.

  • Scenario 3: The application pushes the log directly

This solution needs to be developed by modifying the code to push the application directly to the remote storage instead of typing out of the console or local files, not much use, and out of Kubernetes scope

way advantages disadvantages
Solution 1: Deploy a log collector on Node Only one log collector needs to be deployed on each Node, which consumes few resources and has no intrusion on applications Application logs need to be written to standard output and standard error output, and multi-line logging is not supported
Option 2: Add a dedicated log collection container to Pod Low coupling Each Pod starts a log collection agent, increasing resource consumption and increasing o&M costs
Scenario 3: The application pushes the log directly No additional collection tools are required Immersion application, increase application complexity

4 Problems that should be paid attention to in K8S log collection

Question 1: What logs do we need to collect for a K8S cluster?

Here is an example of collecting logs:

  • Component logs of the K8S

  • Application logs deployed in K8S Cluster

– Standard output – Log file

Question 2: Where are the logs we need to collect, and how do we collect the common runtime?

Docker and Containerd container logs and related parameters

Compare the item docker containerd
The store path When Docker is running as a K8S container, the disk dumping of container logs is completed by Docker, which is saved in by default/var/lib/docker/containers/$CONTAINERIDDirectory. Kubelet in/var/log/podsand/var/log/containersLet’s create a soft link to point to/var/lib/docker/containers/$CONTAINERIDContainer log files in the directory When containerd is running as a K8S container, kubelet drives container logs and saves them to/var/log/pods/$CONTAINER_NAMEDirectory, at the same time/var/log/containersCreate a soft link in the directory to point to the log file
Configuration parameters “Log-driver “: “json-file”, “log-opts”: {“max-size”: “100m”,”max-file”: “5”} –container-log-max-files=5 –container-log-max-size=”100Mi” –container-log-max-size=”100Mi” “containerLogMaxSize”: “100Mi”, “containerLogMaxFiles”: 5,
Save the container logs to the data disk Mount the data disk to “data-root”(the default is /data/var/lib/docker) Create a soft link /var/log/pods that points to a directory under the mount point of the data disk (ln -s /data /var/log/Pods /var/log/)

Question 3: Do you need to standardize logs

The basic format

The output format is JSON. To facilitate collection, the log should be output in one line

define

Logs generated by all service applications running in the K8S cluster.

The necessary fields
  • level

Log level field. The field value is in lower case.

  • Debug: Indicates detailed event information, most useful for debugging applications.
  • Info: describes the running process of an application in coarse granularity
  • Warn: Indicates potentially harmful conditions.
  • Error: Indicates an error event, but the application may continue to run
  • Fatal: Indicates a critical error event that may result in application termination.

The log level is used as the basis for log collection and alarm. In the production system, the log collector collects only logs of higher level than INFO, and alarms of higher level than Error are generated.

  • msg

Log content.

  • remote_ip

Source IP address of the request

  • project

Service name and version number, for example, [email protected]

  • time

Logs are printed in the UTC time format.

  • func

The directory and the number of lines in the log code

Optional fields (The optional fields are used as required. The following fields will be resolved after log collection.)
  • request_url

The URL of the request

  • status

This request returns an HTTP status code

  • cost

This request takes time, in ms

  • method

The HTTP method of the request

  • _id

Log id

Ingress log

Nginx-ingress is used to expose services. Therefore, after cluster initialization, the deployed Nginx-ingress needs to specify the following fields and output logs in JSON format. To facilitate collection, the logs should be output in one line.

  • Field is required
log-format-upstream: '{"@timestamp":"$time_iso8601","host":"$server_addr", "clientip" : "$remote_addr", "size" : "$body_bytes_sent" ,"requesttime":"$request_time","upstremtime":"$upstream_response_time","upstremhost":"$upstream_addr","httphost":"$host" ,"referer":"$http_referer","xff":"$http_x_forwarded_for","agent":"$http_user_agent","clientip":"$remote_addr","request": "$request","uri":"$uri","status":"$status"}'Copy the code

5 Deployment Procedure

Solution 1: Deploy a log collection program on nodes

  • Supported CPU architectures
  • amd64
  • arm64
  • Support for the K8S runtime category
  • docker
  • containerd
  • ELK Stack Version of each component
  • Elasticsearch: 7.9.3
  • Filebeat: 7.9.3
  • Kibana: 7.9.3
  • Logstash: 7.9.3
  • K8s version supported
  • Version v1.15.0+ or later

For details about YAML in this deployment, see github.com/sunsharing-…

5.1 Single-Node Deployment of ES

The method of single-node deployment of ELK is relatively simple. You can refer to the yamL choreography file below. The whole idea is to create an ES, then create a visual display of Kibana, create an ES service, and then access the domain through ingress

  • In the K8S cluster, when the log quantity exceeds 20G per day, it is still recommended to deploy it outside the K8S cluster to support the architecture of distributed cluster. Here, the statically deployed mode is used, and hostPATH is used for persistence. Therefore, you need to label the node with es to run the YAML
Kubectl label node XXXX es=dataCopy the code
[root@k8s-master fek]# cat es.yaml --- apiVersion: v1 kind: Service metadata: name: elasticsearch-logging namespace: kube-system labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "Elasticsearch" spec: ports: - port: 9200 protocol: TCP targetPort: db selector: k8s-app: elasticsearch-logging --- # RBAC authn and authz apiVersion: v1 kind: ServiceAccount metadata: name: elasticsearch-logging namespace: kube-system labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: elasticsearch-logging labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile rules: - apiGroups: - "" resources: - "services" - "namespaces" - "endpoints" verbs: - "get" --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: kube-system name: elasticsearch-logging labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile subjects: - kind: ServiceAccount name: elasticsearch-logging namespace: kube-system apiGroup: "" roleRef: kind: ClusterRole name: elasticsearch-logging apiGroup: "" --- # Elasticsearch deployment itself apiVersion: apps/v1 kind: Create Pod metadata: name: elasticsearch-logging # Create Pod metadata: name: elasticsearch-logging # Create Pod metadata Kube-system # namespaces labels: k8s-app: elasticsearch-logging kubernetes. IO /cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile srv: srv-elasticsearch spec: serviceName: Elasticsearch-logging # associated with SVC This can be sure to use each of the following DNS address access Statefulset pod (es - cluster - [0]. Elasticsearch. Elk. SVC. Cluster. The local) replicas: Template: template: template: template: template: template: template: template: template: template: template: template: template: template: labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" spec: serviceAccountName: Elasticsearch - logging containers: - image: docker. IO/library/elasticsearch: 7.9.3 name: elasticsearch - logging resources: # need more cpu upon initialization, therefore burstable class limits: cpu: 1000m memory: 2Gi requests: cpu: 100m memory: 500Mi ports: - containerPort: 9200 name: db protocol: TCP - containerPort: 9300 name: transport protocol: TCP volumeMounts: - name: elasticsearch - logging mountPath: / usr/share/elasticsearch/data / # hardpoints env: - name: Namespace-name: "discovery. Type "# Define a single node type value: Value: "-xms512m -xmx2g" volumes: - name: ES_JAVA_OPTS "volumes: -name: ES_JAVA_OPTS" volumes: -name: ES_JAVA_OPTS "volumes: -name: ES_JAVA_OPTS # nodeSelect ES: Data Tolerations: - effect: nodeSelect es: Data Tolerations: - Effect: NoSchedule operator: Exists # Elasticsearch requires vm.max_map_count to be at least 262144. # If your OS already sets up this number to a Higher value, feel free # to remove this init container. initContainers: Elasticsearch - logging - init image: alpine: 3.6 the command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"] * * * * * * * * * * * * * * * * * * * IfNotPresent Command: ["sh", "-c", "ulimit -n 65536"] securityContext: privileged: true-name: privileged Image: alpine:3.6 command: elasticsearch-volume-init #es image: alpine:3.6 command: elasticsearch-volume-init #es - chmod - -R - "777" - /usr/share/elasticsearch/data/ volumeMounts: - name: elasticsearch-logging mountPath: /usr/share/elasticsearch/data/Copy the code
  • Create Elasticsearch using the yamL file you just wrote and check to see if it is started. You can see that a pod copy of Elasticsearch -0 has been created and is running properly. If it doesn’t start properly, use Kubectl Describe to see the details and troubleshoot the problem
[root@k8s-master fek]# kubectl get pod -n kube-system
NAME                        READY   STATUS             RESTARTS   AGE
coredns-5bd5f9dbd9-95flw    1/1     Running            0          17h
elasticsearch-0             1/1     Running            1          16m
Copy the code
  • Then, it is necessary to deploy a Kibana to make a visual display of the collected logs, and write a YAML in Deployment mode. In Seivice, nodeport 25601 is used for external exposure access, and ES is directly referenced. Ingress can also be used for exposure
[root@k8s-master fek]# cat kibana.yaml --- apiVersion: v1 kind: Service metadata: name: kibana namespace: kube-system labels: k8s-app: kibana kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: IO /name: "Kibana" SRV: srV-Kibana spec: type: NodePort # Reconcile kubernetes. Ports: -port: 5601 nodePort: 25601 protocol: TCP targetPort: ui selector: k8s-app: kibana --- apiVersion: apps/v1 kind: Deployment metadata: name: kibana namespace: kube-system labels: k8s-app: kibana kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile srv: srv-kibana spec: replicas: 1 selector: matchLabels: k8s-app: kibana template: metadata: labels: k8s-app: kibana annotations: seccomp.security.alpha.kubernetes.io/pod: 'docker/default' spec: containers: - name: kibana image: Docker. IO/kubeimages kibana: 7.9.3 # the mirror support arm64 and amd64 architecture resources: # need more cpu upon initialization, therefore burstable class limits: cpu: 1000m requests: cpu: 100m env: - name: ELASTICSEARCH_HOSTS value: http://elasticsearch-logging:9200 ports: - containerPort: 5601 name: ui protocol: TCP --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: kibana namespace: kube-system spec: rules: - host: kibana.ctnrs.com http: paths: - path: / backend: serviceName: kibana servicePort: 5601Copy the code
  • Using the YAML I just wrote to create Kibana, you can see that a pod for Kibana-B7d98644-LSHsz is generated and it works
[root@k8s-master fek]# kubectl apply -f kibana.yaml
deployment.apps/kibana created
service/kibana created
[root@k8s-master fek]# kubectl get pod -n kube-system
NAME                        READY   STATUS             RESTARTS   AGE
coredns-5bd5f9dbd9-95flw    1/1     Running            0          17h
elasticsearch-0             1/1     Running            1          16m
kibana-b7d98644-48gtm       1/1     Running            1          17h
Copy the code
  • Finally, enter http://(IP address of any node in the browser):25601, you will enter the Web interface of Kibana, has been set do not need to log in, the current page is all English mode, you can modify the Internet search to modify the location of the configuration file, it is recommended to use the English version

5.2 Deploy a FileBeat Collector on the Node to collect K8S component logs

  • After es and Kibana are deployed, how do we collect pod logs? We adopt solution 1, which is to deploy a FileBeat collector on each node, using version 7.9.3. In addition, I have done the standard logging of the Docker or Containerd runtime as shown in Question 2 in Section 4
[root@k8s-master fek]# cat filebeat.yaml --- apiVersion: v1 kind: ConfigMap metadata: name: filebeat-config namespace: kube-system labels: k8s-app: filebeat data: filebeat.yml: |- filebeat.inputs: - type: container paths: -add_kubernetes_metadata: # Add k8s field for later data cleaning host: / /var/log/containers/*.log # Add k8s field for later data cleaning host: ${NODE_NAME} matchers: - logs_path: logs_path: "/var/log/containers/" #output.kafka: If there is a large amount of log and the log in es is delayed, you can choose to add kafka # hosts between fileBeat and logstash: ["kafka-log-01:9092", "kafka-log-02:9092", "kafka-log-03:9092"] # topic: 'topic-test-log' # version: 2.0.0 output.logstash: # Since logStash is deployed to clean the data, fileBeat pushes the data into the logStash. Hosts: ["logstash:5044"] Enabled: true --- # Source: filebeat/templates/filebeat-service-account.yaml apiVersion: v1 kind: ServiceAccount metadata: name: filebeat namespace: kube-system labels: k8s-app: filebeat --- # Source: filebeat/templates/filebeat-role.yaml apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: filebeat labels: k8s-app: filebeat rules: - apiGroups: [""] # "" indicates the core API group resources: - namespaces - pods verbs: - get - watch - list --- # Source: filebeat/templates/filebeat-role-binding.yaml apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: filebeat subjects: - kind: ServiceAccount name: filebeat namespace: kube-system roleRef: kind: ClusterRole name: filebeat apiGroup: rbac.authorization.k8s.io --- # Source: filebeat/templates/filebeat-daemonset.yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: filebeat namespace: kube-system labels: k8s-app: filebeat spec: selector: matchLabels: k8s-app: filebeat template: metadata: labels: k8s-app: filebeat spec: serviceAccountName: filebeat terminationGracePeriodSeconds: 30 containers: - name: filebeat image: Docker. IO/kubeimages filebeat: 7.9.3 # the mirror support arm64 and amd64 architecture args: ["-c", "/etc/filebeat.yml", "-e","-httpprof","0.0.0.0:6060"] #ports: # -containerPort :6060 # hostPort: 6068 env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: ELASTICSEARCH_HOST value: elasticsearch-logging - name: ELASTICSEARCH_PORT value: "9200" securityContext: runAsUser: 0 # If using Red Hat OpenShift uncomment this: #privileged: true resources: limits: memory: 1000Mi cpu: 1000m requests: Memory: 100Mi CPU: 100m volumeMounts: -name: config # mountPath: /etc/filebeat.yml readOnly: MountPath: /usr/share/filebeat/data-name: /usr/share/ filebeat/data-name: /usr/share/ filebeat/data-name: /usr/share/ filebeat/data-name: /usr/share/ filebeat/data-name: Varlibdockercontainers # Mount the source log directory on the host to the FileBeat container. If you don't change the Docker or Containerd runtime to the standard log directory, Change mountPath to /var/lib mountPath: /data/ var/readonly: true-name: MountPath: /var/log/readonly: true - name: mountPath: /var/log/containers timezone mountPath: /etc/localtime volumes: - name: config configMap: defaultMode: 0600 name: filebeat-config - name: Varlibdockercontainers hostPath: if you don't change the docker or Containerd runtime to the standard log path, you can change the path to /var/lib: /data/var/ - name: varlog hostPath: path: /var/log/ # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart - name: inputs configMap: defaultMode: 0600 name: filebeat-inputs - name: data hostPath: path: /data/filebeat-data type: DirectoryOrCreate - name: timezone hostPath: Path: /etc/localtime tolerations: # Add tolerations to schedule to each node - Effect: NoExecute Key: Dedicated operator: Equal value: gpu - effect: NoSchedule operator: ExistsCopy the code
  • After deployment, check whether it was created successfully and you can see that two copies of pod named FileBeat-xx are created on two nodes
[root@k8s-master elk]# kubectl apply -f filebeat.yaml
[root@k8s-master elk]# kubectl get pod -n kube-system
NAME                        READY   STATUS    RESTARTS   AGE
coredns-5bd5f9dbd9-8zdn5    1/1     Running   0          10h
elasticsearch-0             1/1     Running   1          13h
filebeat-2q5tz              1/1     Running   0          13h
filebeat-k6m27              1/1     Running   2          13h
kibana-b7d98644-tllmm       1/1     Running   0          10h
Copy the code

5.3 Adding logStash to clean the collected original logs

  • This is a combination of business needs and secondary utilization of the log, as shown by Grafana, so logStash is added to clean the log. It needs to be converted to inGRSS field types, and the business service log has field changes and type conversions that you can adjust to your business needs
[root@k8s-master fek]# cat logstash.yaml --- apiVersion: v1 kind: Service metadata: name: logstash namespace: kube-system spec: ports: - port: 5044 targetPort: beats selector: type: logstash clusterIP: None --- apiVersion: apps/v1 kind: Deployment metadata: name: logstash namespace: kube-system spec: selector: matchLabels: type: logstash template: metadata: labels: type: logstash srv: srv-logstash spec: containers: - image: Docker. IO/kubeimages logstash: 7.9.3 # the mirror support arm64 and amd64 architecture name: logstash ports: - containerPort: 5044 name: beats command: - logstash - '-f' - '/etc/logstash_c/logstash.conf' env: - name: "XPACK_MONITORING_ELASTICSEARCH_HOSTS" value: "http://elasticsearch-logging:9200" volumeMounts: - name: config-volume mountPath: /etc/logstash_c/ - name: config-yml-volume mountPath: /usr/share/logstash/config/ - name: Timezone mountPath: /etc/localtime resources: #logstash Limits must be added to prevent resource preemption on other services. Limits: CPU: 1000M memory: 2048Mi requests: cpu: 512m memory: 512Mi volumes: - name: config-volume configMap: name: logstash-conf items: - key: logstash.conf path: logstash.conf - name: timezone hostPath: path: /etc/localtime - name: config-yml-volume configMap: name: logstash-yml items: - key: logstash.yml path: logstash.yml --- apiVersion: v1 kind: ConfigMap metadata: name: logstash-conf namespace: kube-system labels: type: logstash data: logstash.conf: | - input {beats {port = > 5044}} filter {# handling ingress log if [kubernetes] [container] [name] = = "nginx - ingress - controller"  { json { source => "message" target => "ingress_log" } if [ingress_log][requesttime] { mutate { convert => ["[ingress_log][requesttime]", "float"] } } if [ingress_log][upstremtime] { mutate { convert => ["[ingress_log][upstremtime]", "float"] } } if [ingress_log][status] { mutate { convert => ["[ingress_log][status]", "float"] } } if [ingress_log][httphost] and [ingress_log][uri] { mutate { add_field => {"[ingress_log][entry]" => "%{[ingress_log][httphost]}%{[ingress_log][uri]}"} } mutate{ split => ["[ingress_log][entry]","/"] } if [ingress_log][entry][1] { mutate{ add_field => {"[ingress_log][entrypoint]" => "%{[ingress_log][entry][0]}/%{[ingress_log][entry][1]}"} remove_field => "[ingress_log][entry]" } } else{ mutate{ add_field => {"[ingress_log][entrypoint]" => "%{[ingress_log][entry][0]}/"} remove_field => "[ingress_log][entry]" } } } If [kubernetes][container][name] =~ /^ SRV */ {json {source => "message" target => "TMP"} if [container][name] =~ /^ SRV */ {json {source => "message" target => "TMP"} if [kubernetes][namespace] == "kube-system" { drop{} } if [tmp][level] { mutate{ add_field => {"[applog][level]" => "%{[tmp][level]}"} } if [applog][level] == "debug"{ drop{} } } if [tmp][msg]{ mutate{ add_field => {"[applog][msg]" => "%{[tmp][msg]}"} } } if [tmp][func]{ mutate{ add_field => {"[applog][func]" => "%{[tmp][func]}"} } } if [tmp][cost]{ if "ms" in [tmp][cost]{ mutate{ split => ["[tmp][cost]","m"] add_field => {"[applog][cost]" => "%{[tmp][cost][0]}"} convert  => ["[applog][cost]", "float"] } } else{ mutate{ add_field => {"[applog][cost]" => "%{[tmp][cost]}"} } } } if [tmp][method]{ mutate{ add_field  => {"[applog][method]" => "%{[tmp][method]}"} } } if [tmp][request_url]{ mutate{ add_field => {"[applog][request_url]" => "%{[tmp][request_url]}"} } } if [tmp][meta._id]{ mutate{ add_field => {"[applog][traceId]" => "%{[tmp][meta._id]}"} }  } if [tmp][project] { mutate{ add_field => {"[applog][project]" => "%{[tmp][project]}"} } } if [tmp][time] { mutate{ add_field => {"[applog][time]" => "%{[tmp][time]}"} } } if [tmp][status] { mutate{ add_field => {"[applog][status]" => "%{[tmp][status]}"} convert => ["[applog][status]", "float"] } } } mutate{ rename => ["kubernetes", "k8s"] remove_field => "beat" remove_field => "tmp" remove_field => "[k8s][labels][app]" } } output{ elasticsearch { Hosts => ["http://elasticsearch-logging:9200"] codec => JSON index => "logstash-%{+ YYYY.mm.dd}" # The index name is created with the logstash+ log every day } } --- apiVersion: v1 kind: ConfigMap metadata: name: logstash-yml namespace: kube-system labels: type: logstash data: Logstash. Yml: | - HTTP. Host: "0.0.0.0" xpack. Monitoring. Elasticsearch. Hosts: http://elasticsearch-logging:9200Copy the code

5.4 Visualizing Configuration Logs on the Kibana Webui

  • First, log in to the Kibana interface and open the Stack Management module in the menu

  • Click Index Management. You can find that log indexes have been collected

  • To prevent ES logs from taking up more and more disk space, we can add an index Lifecycle Policites based on business needs by clicking Index Lifecycle Policites

  • Write the Policy name as logstash-history-ilm-policy. The Policy name cannot be changed arbitrarily. It will be referenced in subsequent templates

  • In order to be able to view logs in Discover in Kibana, you need to set up an index match, select Index Patterns, and create

  • Since we are deployed as a single node, the indexes created using the default index template will produce a copy of 1, so we will find that the indexes are yellow. The solution is as follows

Open dev Tools in the menuThe API is then called to change the number of index copies to 0

PUT _all/_settings
{
    "number_of_replicas": 0
}
Copy the code
  • To address and link the index lifecycle policy fundamentally and standardize the map type in the log fields, we need to modify the default template
PUT _template/logstash
{
    "order": 1,
    "index_patterns": [
      "logstash-*"
    ],
    "settings": {
      "index": {
      "lifecycle" : {
          "name" : "logstash-history-ilm-policy"
        },
        "number_of_shards": "2",
        "refresh_interval": "5s",
        "number_of_replicas" : "0"
      }
    },
    "mappings": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "applog": {
            "dynamic": true,
            "properties": {
              "cost": {
                "type": "float"
              },
              "func": {
                "type": "keyword"
              },
              "method": {
                "type": "keyword"
              }
            }
          },
          "k8s": {
            "dynamic": true,
            "properties": {
              "namespace": {
                "type": "keyword"
              },
              "container": {
                "dynamic": true,
                "properties": {
                  "name": {
                    "type": "keyword"
                  }
                }
              },
              "labels": {
                "dynamic": true,
                "properties": {
                  "srv": {
                    "type": "keyword"
                  }
                }
              }
            }
          },
          "geoip": {
            "dynamic": true,
            "properties": {
              "ip": {
                "type": "ip"
              },
              "latitude": {
                "type": "float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "float"
              }
            }
          }
      }
    },
    "aliases": {}
  }
Copy the code

  • Finally, validate the index and Discover

Write in the last

Log collection is only part of the observable business, and there are not only Elastic Stack for logs, but also Loki, Splunk, or log collection solutions on the managed cloud. There are many ways to get there, no matter how you do it. The best way to reflect the business problems is to quickly investigate the business problems