One, foreword

Elk is an open source real-time log processing and analysis solution from Elastic. Elk is an open source real-time log processing and analysis solution from Elastic. Elk is an open source real-time log processing and analysis solution from Elastic.

In a production environment, how to monitor ELK to ensure the normal operation of each component? How do you know if your current resources can handle the pressure online? This article uses the Elastic Stack 7.x version as an example of how to monitor ELK’s own components.

 

2. Overall structure

The common Elastic Stack logging system architecture is as follows

The MetricBeat component can be used as a lightweight monitoring agent. The monitoring information of each component can be collected through HTTP endpoints, and the monitoring data can be driven to Elasticsearch. Finally, the monitoring data can be graphically displayed through Kibana.

 

Third, deploy MetricBeat

It is recommended to run MetricBeat collection metrics on every server and metrics for multiple MetricBeat instances will be combined on the Elasticsearch server.

Download the corresponding version of MetricBeat from the following address:

https://www.elastic.co/cn/dow…

3.1. Collect ElasticSearch information

Enable and configure the Elasticsearch X-Pack module in MetricBeat from the installation directory and run:

./metricbeat modules enable elasticsearch-xpack

By default, the module collects ElasticSearch metrics from http://localhost:9200. If the local server has a different address, add it to the host Settings in modules.d/elasticsearch-xpack.yml.

 

3.2. Collect Kibana information

Enable and configure the Kibana X-Pack module in MetricBeat

./metricbeat modules enable kibana-xpack

This module will collect Kibana monitoring metrics by default from http://localhost:5601. If the local Kibana instance has a different address, it must be specified through the hosts setting in the modules.d/ Kibana-xpack.yml file.

 

3.3. Collect Logstash information

Enable and configure the Logstash X-Pack module in MetricBeat

./metricbeat modules enable logstash-xpack

This module will collect Logstash monitoring metrics from http://localhost:9600 by default. If the local Logstash instance has a different address, it must be specified through the hosts setting in the modules.d/logstash-xpack.yml file.

 

3.4. Collect BEATS information

All types
BeatsThe configuration is the same

3.4.1. Enable HTTP Endpoints

You need to turn on BEATS ‘own HTTP endpoint to output monitoring data, such as FileBeat to modify the fileBeat. YML file and add the following configuration at the end

HTTP: Enabled: True Host: 0.0.0.0 Port: 5066

3.4.2. Enable the BEAT module

Enable and configure the Beat X-Pack module in MetricBeat

./metricbeat modules enable beat-xpack

This module will collect BEAT monitoring metrics from http://localhost:5066 by default. If the Beat instance being monitored has a different address, it must be specified through the hosts setting in the modules.d/beat-xpack.yml file.

 

3.5. Data output configuration

Configure metricBeat to send to the monitor cluster, and modify the following in the metricBeat. YML file

output.elasticsearch:
  hosts: ["http://localhost:9200"] ## Monitoring cluster

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

PS: Address, user name and password are changed according to the actual situation

 

3.6. Start Metricbeat

./metricbeat -e

 

Collect Elasticsearch logs

Use FileBeat to collect Elasticsearch’s own log data.

First you need to install the FileBeat component on the same server as Elasticsearch.

4.1. Enable the ES module

Enable and configure the Elasticsearch module in FileBeat and execute the following commands

./filebeat modules enable elasticsearch

 

4.2. Configure the ES module

Modify the configuration information of the ES module to specify the log path

vim modules.d/elasticsearch.yml

Modify to the following

- module: elasticsearch
  server:
    enabled: true
    var.paths:
      - /app/elk/elasticsearch/logs/*_server.json

  gc:
    enabled: true
    var.paths:
      - /app/elk/elasticsearch/logs/gc.log.[0-9]*
      - /app/elk/elasticsearch/logs/gc.log

  audit:
    enabled: true
    var.paths:
      - /app/elk/elasticsearch/logs/*_audit.json

  slowlog:
    enabled: true
    var.paths:
      - /app/elk/elasticsearch/logs/*_index_search_slowlog.json
      - /app/elk/elasticsearch/logs/*_index_indexing_slowlog.json

  deprecation:
    enabled: true
    var.paths:
      - /app/elk/elasticsearch/logs/*_deprecation.json

PS: Log path is modified according to the actual situation

 

4.3. Configure output

Modify the filebeat.yml file to configure the ES information

output.elasticsearch:
  hosts: ["localhost:9200"]
  
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

PS: Address, user name and password are changed according to the actual situation

 

4.4. Start Filebeat

./filebeat -c filebeat.yml -e

 

Five, check the monitoring interface

Enter theKibanaThe console interface, enterStack monitoringThe menu

 

You can view the monitoring information of each component

 

Scan code concern has surprise!