SpringBoot e-commerce project mall (40K + STAR) address: github.com/macrozheng/…

Abstract

The ELK log collection system we built before is mainly used to collect logs of SpringBoot applications. The Logstash plugin is used to transfer logs to Elasticsearch using TCP. Filebeat is a log handler for Elasticsearch. This article will explain how to use Filebeat as a log handler for Elasticsearch. Hope to help you.

Filebeat profile

Filebeat is a lightweight log collector that can forward and summarize logs and files. Filebeat comes with multiple built-in modules (Nginx, MySQL, Redis, Elasticsearch, Logstash, etc.) that greatly simplify the collection, parsing, and visualization of common log formats with a single command.

Filebeat installation and configuration

Before you install Filebeat, you need to install Elasticsearch and Kibana first. Note Use version 7.6.2.

  • We first download Filebeat installation package, download address: www.elastic.co/cn/download…

  • After the download is complete, unzip to the specified directory, notice the three places shown in the figure;

  • Kibana provided good Filebeat collected all kinds of log in the early tutorials, we first enter the home page, access to the address: http://localhost:5601/app/kibana#/home

  • Click on theAdd log dataButton, you can find support for middleware or very rich, covering the commonly used;

  • Click on the Nginx log collection tutorial to see if it is detailed;

  • Configure Filebeat by configuring the Elasticsearch/Kibana connection address and modifying the Filebeat installation directoryfilebeat.ymlThen, modify the content as follows.
output.elasticsearch:
  hosts: ["localhost:9200"]
setup.kibana:
  host: "localhost:5601"
Copy the code

Collect Nginx logs

Let’s start with Nginx and try out Filebeat’s log collection feature.

  • Run the following command to start the Nginx log collection module of Filebeat.
filebeat modules enable nginx
Copy the code
  • After the opening is complete, entermodules.dIn the directory, you’ll findnginx.ymlthedisableIf the suffix is missing, it indicates that it has been opened.

  • I’m going to modifynginx.ymlFile, configure NginxaccessanderrorLog path;
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-nginx.html

- module: nginx
  access:
    enabled: true
    var.paths: ["I: / developer/env/nginx, version 1.8.1 - / logs/access log"]

  error:
    enabled: true
    var.paths: ["I: / developer/env/nginx, version 1.8.1 - / logs/error log"]
Copy the code
  • Since the Nginx log collection module is enabled, we need to set Filebeat with the following command;
filebeat setup
Copy the code

  • Run the following command to start the Filebeat service:
filebeat -e
Copy the code
  • We can check whether the data was collected successfully by using the Nginx tutorial page in Kibana.

  • Click on theNginx logs dashboardButton to view collected Nginx logs.

  • Take a look at the detailed log, no longer need to go to the server for Nginx log!

Collect Elasticsearch logs

Let’s try collecting Elasticsearch logs again.

  • Run the following command to enable the Elasticsearch log collection module of Filebeat:
filebeat modules enable elasticsearch
Copy the code
  • I’m going to modifyelasticsearch.ymlThe log path of Elasticsearch is configured.
# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-elasticsearch.html

- module: elasticsearch
  server:
    enabled: true
    var.paths: ["I: / developer/env/elasticsearch - 7.6.2 / logs/elasticsearch log"]

  slowlog:
    enabled: true
    var.paths: ["I: / developer/env/elasticsearch - 7.6.2 / logs/elasticsearch_index_indexing_slowlog log"."I: / developer/env/elasticsearch - 7.6.2 / logs/elasticsearch_index_search_slowlog log"]

  deprecation:
    enabled: true
    var.paths: ["I: / developer/env/elasticsearch - 7.6.2 / logs/elasticsearch_deprecation log"]
Copy the code
  • Set up and start Filebeat service.
filebeat setup
filebeat -e
Copy the code
  • inDiscoverView Elasticsearch logs on the page.

Collecting MySQL Logs

Try collecting MySQL logs again.

  • Run the following command to start the MySQL log collection module of Filebeat:
filebeat modules enable mysql
Copy the code
  • I’m going to modifymysql.ymlConfigure the MySQL log path, mainly error logs and slow query logs.
# Module: mysql
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.6/filebeat-module-mysql.html

- module: mysql
  error:
    enabled: true
    var.paths: ["C: / ProgramData/MySQL/MySQL Server 5.7 / Data/DESKTOP - 5 nimj19. Err"]

  slowlog:
    enabled: true
    var.paths: ["C: / ProgramData/MySQL/MySQL Server 5.7 / Data/DESKTOP - 5 nimj19 - missile. The log"]
Copy the code
  • Set up and start Filebeat service.
filebeat setup
filebeat -e
Copy the code
  • Click on theMySQL logs dashboardButton to view collected MySQL logs;

  • View the log details collected by MySQL.

conclusion

Filebeat can be used to collect logs from Nginx, Elasticsearch and MySQL files.

In this paper, making github.com/macrozheng/… Already included, welcome everyone Star!