Abstract: ES cluster is a powerful tool for big data storage and analysis, and fast retrieval. This paper briefly describes the cluster architecture of ES, and provides a sample of rapid deployment of ES cluster in Kubernetes. This paper introduces the monitoring operation and maintenance tools of ES cluster, and provides some experience in locating problems. Finally, it summarizes the common API call methods of ES cluster.

This article is shared by Huawei cloud community “Deploying ES Cluster and O&M in Kubernetes”. The original article is written by Minucas.

ES cluster architecture:

ES clusters are classified into single point mode and cluster mode. The single point mode is not recommended in the production environment. Cluster mode is recommended. The cluster mode is divided into the deployment mode in which the Master node and Data node are assumed by the same node and the deployment mode in which the Master node and Data node are assumed by different nodes. The reliability is higher when the Master node and Data node are deployed separately. The following figure shows the deployment architecture of ES cluster:

ES cluster deployment using K8s:

1. The K8S statefulset deployment allows rapid expansion and contraction of ES nodes. In this example, 3 Master nodes + 12 Data nodes are deployed

2. The corresponding domain name and service discovery are configured through K8S Service to ensure automatic cluster connectivity and monitoring

kubectl -s http://ip:port create -f es-master.yaml
kubectl -s http://ip:port create -f es-data.yaml
kubectl -s http://ip:port create -f es-service.yaml
Copy the code

es-master.yaml:

apiVersion: apps/v1 kind: StatefulSet metadata: labels: addonmanager.kubernetes.io/mode: Reconcile k8s-app: Es kubernetes. IO /cluster-service: "true" version: v6.2.5 Name: es-master Namespace: default spec: podManagementPolicy: OrderedReady Replicas: 3 revisionHistoryLimit: 10 Selector: matchLabels: K8S-app: es Version: V6.2.5 serviceName: Es template: metadata: labels: K8S-app: camp-es kubernetes. IO /cluster-service: "true" version: v6.2.5 spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: ELASTICSEARCH_SERVICE_NAME value: es - name: NODE_MASTER value: "true" - name: NODE_DATA value: "false" - name: ES_HEAP_SIZE value: 4g - name: ES_JAVA_OPTS value: -Xmx4g -Xms4g - name: cluster.name value: es image: Elasticsearch :v6.2.5 imagePullPolicy: Always name: es ports: -containerPort: 9200 hostPort: 9200 Name: DB protocol: TCP - containerPort: 9300 hostPort: 9300 name: transport protocol: TCP resources: limits: cpu: "6" memory: 12Gi requests: cpu: "4" memory: 8Gi securityContext: capabilities: add: - IPC_LOCK - SYS_RESOURCE volumeMounts: - mountPath: /data name: es - command: - /bin/elasticsearch_exporter - -es.uri=http://localhost:9200 - -es.all=true image: Elasticsearch_exporter :1.0.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /health port: 9108 scheme: HTTP initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 name: es-exporter ports: - containerPort: 9108 hostPort: 9108 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /health port: 9108 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 resources: limits: cpu: 100m memory: 128Mi requests: cpu: 25m memory: 64Mi securityContext: capabilities: drop: - SETPCAP - MKNOD - AUDIT_WRITE - CHOWN - NET_RAW - DAC_OVERRIDE - FOWNER - FSETID - KILL - SETGID - SETUID - NET_BIND_SERVICE - SYS_CHROOT - SETFCAP readOnlyRootFilesystem: true dnsPolicy: ClusterFirst initContainers: - command: - /sbin/sysctl -- -w-vm. Max_map_count =262144 image: alpine:3.6 imagePullPolicy: IfNotPresent name: elasticsearch-logging-init resources: {} securityContext: privileged: true restartPolicy: Always schedulerName: default-scheduler securityContext: {} volumes: - hostPath: path: /Data/es type: DirectoryOrCreate name: esCopy the code

es-data.yaml

apiVersion: apps/v1 kind: StatefulSet metadata: labels: addonmanager.kubernetes.io/mode: Reconcile k8s-app: Es kubernetes. IO /cluster-service: "true" version: v6.2.5 Name: es-data Namespace: default spec: podManagementPolicy: OrderedReady Replicas: 12 revisionHistoryLimit: 10 Selector: matchLabels: K8S-app: es Version: V6.2.5 serviceName: Es template: metadata: labels: K8S-app: es kubernetes. IO /cluster-service: "true" version: v6.2.5spec: containers: - env: - name: NAMESPACE valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.namespace - name: ELASTICSEARCH_SERVICE_NAME value: es - name: NODE_MASTER value: "false" - name: NODE_DATA value: "true" - name: ES_HEAP_SIZE value: 16g - name: ES_JAVA_OPTS value: -Xmx16g -Xms16g - name: cluster.name value: es image: Elasticsearch :v6.2.5 imagePullPolicy: Always name: es ports: -containerPort: 9200 hostPort: 9200 Name: DB protocol: TCP - containerPort: 9300 hostPort: 9300 name: transport protocol: TCP resources: limits: cpu: "8" memory: 32Gi requests: cpu: "7" memory: 30Gi securityContext: capabilities: add: - IPC_LOCK - SYS_RESOURCE volumeMounts: - mountPath: /data name: es - command: - /bin/elasticsearch_exporter - -es.uri=http://localhost:9200 - -es.all=true image: Elasticsearch_exporter :1.0.2 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /health port: 9108 scheme: HTTP initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 name: es-exporter ports: - containerPort: 9108 hostPort: 9108 protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /health port: 9108 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 resources: limits: cpu: 100m memory: 128Mi requests: cpu: 25m memory: 64Mi securityContext: capabilities: drop: - SETPCAP - MKNOD - AUDIT_WRITE - CHOWN - NET_RAW - DAC_OVERRIDE - FOWNER - FSETID - KILL - SETGID - SETUID - NET_BIND_SERVICE - SYS_CHROOT - SETFCAP readOnlyRootFilesystem: true dnsPolicy: ClusterFirst initContainers: - command: - /sbin/sysctl -- -w-vm. Max_map_count =262144 image: alpine:3.6 imagePullPolicy: IfNotPresent name: elasticsearch-logging-init resources: {} securityContext: privileged: true restartPolicy: Always schedulerName: default-scheduler securityContext: {} volumes: - hostPath: path: /Data/es type: DirectoryOrCreate name: esCopy the code

es-service.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: es
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: Elasticsearch
  name: es
  namespace: default
spec:
  clusterIP: None
  ports:
  - name: es
    port: 9200
    protocol: TCP
    targetPort: 9200
  - name: exporter
    port: 9108
    protocol: TCP
    targetPort: 9108
  selector:
    k8s-app: es
  sessionAffinity: None
  type: ClusterIP
Copy the code

ES Cluster Monitoring

The operation and maintenance of middleware should first have sufficient monitoring means. ES cluster monitoring is commonly referred to as “exporter”, “EShead” and “KOPF”. As ES cluster is deployed with K8S architecture, many features will be carried out in combination with K8S

Grafana monitoring

Es-exporter exports monitoring metrics through k8S deployment, Prometheus collects monitoring data, and Grafana custom dashboard displays

ES – head components

Github address: github.com/mobz/elasti…

The ES-Head component can be searched and installed through the Google Browser App Store, and the ES cluster can be viewed using the Chrome plug-in

Cerebro (KOPF) components

Github address: github.com/lmenezes/ce…

ES cluster troubleshooting

ES configuration

Resource configuration: Pay attention to ES CPU, Memory, Heap Size, and Xms Xmx configuration. It is recommended that if the machine has 8u32GB Memory, the Heap Memory and Xms Xmx configuration should be 50%. The official website recommends that the Memory of a single node should not exceed 64GB

Index configuration: Because ES is located by index, ES will load relevant index data into memory to speed up the retrieval. Therefore, reasonable index setting has a great impact on THE performance of ES. Currently, we create indexes by date (unsplit indexes with small amount of individual data).

ES load

Relocate the shard or CPU nodes or relocate the SHard or CPU nodes whose CPU or Load is high. The shard or CPU nodes relocate the SHard or CPU nodes whose CPU or Load is high

Shard configuration

The shard configuration should be an integer multiple of the number of data nodes. The more shards the better, the more shards the better. The shard should be properly fragmented according to the data volume of the index, and ensure that each shard does not exceed the heap memory size allocated by a single data node. For example, the index with the largest data volume is about 150GB in a day, which is divided into 24 shards. After calculation, the size of a single shard is about 6-7GB

The recommended number of duplicates is 1. If the number of duplicates is too large, data will be frequently relocate, increasing the cluster load

Delete exception index

The curl -x DELETE "10.64. XXX. Xx: 9200 / SZV - prod - ingress - nginx - 2021.05.01"Copy the code

Index names can be deleted in batches using regular matching, for example, -2021.05.*

Another reason for high node load

When locating the problem, I found that the node data shard had been removed, but the node load could not go down. After logging in to the node, I used the top command to find that the CPU usage of node Kubelet was very high, and it was useless to restart kubelet. The load could not be relieved until the node was restarted

Summary of ES Cluster routine Operation and maintenance experience (refer to official website)

Check the cluster health status

ES Cluster health status can be Green, Yellow, or Red.

Green: Cluster health.

Yellow(Yellow) : The cluster is unhealthy, but can rebalance automatically when the load allows.

Red(Red) : The cluster has a problem. Some data is not ready and at least one primary shard has not been allocated successfully.

You can use the API to query the health status of clusters and unallocated fragments:

GET _cluster/health
{
  "cluster_name": "camp-es",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 15,
  "number_of_data_nodes": 12,
  "active_primary_shards": 2176,
  "active_shards": 4347,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}
Copy the code

Query pending Tasks:

GET /_cat/pending_tasks
Copy the code

Priority indicates the priority of the task

Check the reason why fragments are not allocated

GET _cluster/allocation/explain
Copy the code

The Reason field indicates the reason for which fragments are not allocated, and the detail field indicates the reason for which fragments are not allocated

View all unallocated indexes and master shards:

GET /_cat/indices? v&health=redCopy the code

Check which fragments are abnormal

curl -s http://ip:port/_cat/shards | grep UNASSIGNED
Copy the code

Reassign a primary shard:

POST _cluster/reroute? pretty" -d '{ "commands" : [ { "allocate_stale_primary" : { "index" : "xxx", "shard" : 1, "node" : "12345...", "accept_data_loss": true } } ] }Copy the code

Curl ‘IP :port/_node/process? Pretty s query

Reduce the number of copies of the index

PUT /szv_ingress_*/settings
{
  "index": {
    "number_of_replicas": 1
  }
}
Copy the code

Click to follow, the first time to learn about Huawei cloud fresh technology ~