How to monitor ELK components? Try Metricbeat

One, foreword

ELK is an open-source real-time log processing and analysis solution from Elastic, which has become the mainstream log processing solution of choice.

In the production environment, how to monitor the ELK to ensure the normal operation of each component? How do you know if your current resources can withstand the pressure online? This article uses Elastic Stack 7.x as an example to show how to monitor the components of ELK.

 

Ii. Overall structure

The common Elastic Stack log architecture is as follows

The Metricbeat component can be used as a lightweight monitoring agent to collect monitoring information of each component through HTTP endpoints and drop monitoring data into Elasticsearch. Finally, Kibana can display various monitoring data in a graphical manner.

 

Deploy Metricbeat

It is recommended that Metricbeat collect metrics on each server, and metrics from multiple Metricbeat instances will be merged on the Elasticsearch server.

To download the corresponding version of Metricbeat:

www.elastic.co/cn/download…

3.1. Collecting Elasticsearch Information

Enable and configure the Elasticsearch X-Pack module in Metricbeat from the installation directory, run:

./metricbeat modules enable elasticsearch-xpack
Copy the code

By default, the module collects Elasticsearch metrics from http://localhost:9200. If the local server has a different address, add it to the host Settings in modules.d/ ElasticSearch -xpack.yml.

 

3.2. Collect Kibana information

Enable and configure the Kibana X-Pack module in Metricbeat

./metricbeat modules enable kibana-xpack
Copy the code

This module will collect Kibana monitoring metrics from http://localhost:5601 by default. If the local Kibana instance has a different address, you must specify this through the hosts Settings in modules.d/kibana-xpack.yml.

 

3.3. Collect Logstash information

Enable and configure the Logstash X-pack module in Metricbeat

./metricbeat modules enable logstash-xpack
Copy the code

This module will collect the Logstash monitor metrics from http://localhost:9600 by default. If the local Logstash instance has a different address, it must be specified through the hosts setting in modules.d/ Logstash -xpack.yml.

 

3.4. Collect Beats information

The configuration of Beats is the same for all types

3.4.1. Enable the HTTP endpoint

You need to enable Beats’ own HTTP endpoint to output monitoring data, for example, Filebeat. Modify the filebeat.yml file and add the following configuration at the end

http:
  enabled: true
  host: 0.0. 0. 0
  port: 5066
Copy the code

3.4.2. Enable the Beat module

Enable and configure the Beat X-Pack module in Metricbeat

./metricbeat modules enable beat-xpack
Copy the code

This module will collect beat monitoring metrics from http://localhost:5066 by default. If the beat instance you are monitoring has a different address, you must specify this through the hosts Settings in modules.d/beat-xpack.yml.

 

3.5. Configure data output

To configure Metricbeat to be sent to the monitoring cluster, modify the following in the metricBeat.yml file

output.elasticsearch:
  hosts: ["http://localhost:9200"] ## Monitoring cluster

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"
Copy the code

PS: Change the IP address, user name, and password as required

 

3.6. Start Metricbeat

./metricbeat -e
Copy the code

 

Collect Elasticsearch logs

Collect Elasticsearch’s own log data using Filebeat.

Install the Filebeat component on the server where Elasticsearch resides.

4.1. Enable the ES module

To enable and configure the Elasticsearch module in Filebeat, run the following command

./filebeat modules enable elasticsearch
Copy the code

 

4.2. Configure the ES module

Modify es module configuration information to specify the log path

vim modules.d/elasticsearch.yml
Copy the code

Change the content to the following

- module: elasticsearch
  server:
    enabled: true
    var.paths:
      - /app/elk/elasticsearch/logs/*_server.json

  gc:
    enabled: true
    var.paths:
      - /app/elk/elasticsearch/logs/gc.log.[0-9]*
      - /app/elk/elasticsearch/logs/gc.log

  audit:
    enabled: true
    var.paths:
      - /app/elk/elasticsearch/logs/*_audit.json

  slowlog:
    enabled: true
    var.paths:
      - /app/elk/elasticsearch/logs/*_index_search_slowlog.json
      - /app/elk/elasticsearch/logs/*_index_indexing_slowlog.json

  deprecation:
    enabled: true
    var.paths:
      - /app/elk/elasticsearch/logs/*_deprecation.json
Copy the code

PS: Change the log path based on the actual situation

 

4.3. Configure output

Modify the filebeat.yml file and configure es information

output.elasticsearch:
  hosts: ["localhost:9200"]
  
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"
Copy the code

PS: Change the IP address, user name, and password as required

 

4.4. Start Filebeat

./filebeat -c filebeat.yml -e
Copy the code

 

5. View the monitoring interface

Enter theKibanaEnter the console interfaceStack monitoringThe menu

 

You can view monitoring information about each component

 

Scan code attention has surprise!