This is the seventh day of my participation in the First Challenge 2022. For details: First Challenge 2022.

1. Introduction

For some reason, I decided to build a set of log collection and distributed link tracking in the project. Before, I used Ali Cloud directly.

There is no doubt that this is a fully proven ELK, FileBeat + LogStash + Kibana + ElasticSearch.

If that’s what you need, go ahead and follow the rut.

2. Introduction

2.1 LogStash

Logstash is just what it sounds like: log collection. Logstash, of course, is not only a log collection tools, it is also a log collection tool, for example you can collect nginx log, Logstash also has a lot of input | filter | output plug-in can be used. But since Logstash is written in Java and requires a lot of resources, *Beat is an Elastic move.

2.2 FileBeat

As a file collection tool, FileBeat is lighter than Logstash.

2.3 ElasticSearch

ElasticSearch, needless to say, accounts for the majority of searches. It is mainly used to store information collected by LogStash for word segmentation search.

2.4 Kibana

Kibana is used to query ElasticSeach data and has various dashboard functions. More comprehensive than ES_HEAD.

3. Download and install

3.1 FileBeat

3.1.1 Download link

3.1.2 configuration

  1. Edit files in the root directory:filebeat.yml
  2. Configuration File Source
Inputs: filebeat. Inputs: enabled: true // This must be set to true paths: - /Users/zyq/project/study-project/track/logs/*.logCopy the code
  1. Configure the output to Logstash
output.logstash:
  hosts: ["localhost:5044"]
Copy the code

3.1.3 start

In the root directory./filebeat -e -c filebeat.yml

3.2 LogStash

3.2.1 Download link

3.2.2 configuration

  1. In the root directoryconfigCreate a foldersync.conf
  2. Enter the contents of the code area below
Input {beats{** set input to fileBeat 5044 port ** port =>" 5044"}} output {elasticSearch {# transform ES index =>" fileBeat" Document_type =>"_doc" hosts =>["127.0.0.1:9200"]}}Copy the code

3.2.3 start

In the bin directory./logstash -f.. /config/sync.conf

3.3 ElasticSearch

3.3.1 Download link

3.3.2 rainfall distribution on 10-12 start

If you install it locally, you don’t need to configure it (because ES now comes with the JDK, so you don’t need to configure Java environment variables), go to the bin directory, and start./ elasticSearch.

3.4 Kibana

3.4.1 trackDownload link

3.4.2 start

If you are installing entirely locally, do not change the Es configuration, go to the bin directory, and start./kibana.

4 the end

After all the software is started, FileBeat will automatically monitor the configured file path, read the log file, and store it in Es. Below is the result via Kibana.