ELK is made up of ElasticSearch, Logstash, and Kiabana. The official website: https://www.elastic.co/products

Elasticsearch is an open source distributed search engine. It features distributed, zero configuration, automatic discovery, automatic index sharding, index copy, restful interface, multiple data sources, and automatic search load

2. Logstash is a completely open source tool that collects, filters, and stores your logs for future use (e.g., searching)

3. Kibana is also an open source and free tool that provides a log analysis friendly Web interface for Logstash and ElasticSearch to help you aggregate, analyze and search important data logs

To prepare the installation environment, download the related software installation packages. https://www.elastic.co https://artifacts.elastic.co/downloads/logstash/logstash-5.3.1.tar.gz https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.3.1.tar.gz https://artifacts.elastic.co/downloads/kibana/kibana-5.3.1-linux-x86_64.tar.gz installation configuration JAVA environment

JDK version: JDK-8u144-linux-x64.tar. gz [root@centos7-1 ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
[root@centos7-1 ~]# uname -r3.10.0-693. El7. X86_64 [~] root @ centos7-1# tar zxf jdk-8u144-linux-x64.tar.gz -C /usr/local/
[root@centos7-1 ~]/usr/local/jdk1.8.0_144 /usr/local/jdk
[root@centos7-1 ~]# cat >>/etc/profile <<EOF
export JAVA_HOME=/usr/local/jdk
export PATH=$PATH: $JAVA_HOME/bin
export CLASSPATH=.CLASSPATH:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
EOF
[root@centos7-1 ~]# source /etc/profile
[root@centos7-1 ~]# java -version
java version "1.8.0 comes with _144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
Copy the code

Install Elasticserach. Modify system parameters

[root@centos7-1 config]# vim /etc/sysctl.conf
Add the following configuration
vm.max_map_count=655360
[root@centos7-1 config]# sysctl -p
vm.max_map_count = 655360
[root@centos7-1 config]# tail -5 /etc/security/limits.conf
* soft nofile 65536
* hard nofile 131072
* soft nproc 65536
* hard nproc 131072
# End of file
Copy the code
[root@centos7-1 config]# vim /etc/security/limits.d/20-nproc.conf
Add the following configuration
elk        soft    nproc     65536
Copy the code

[root@centos7-1 config]# useradd elk
[root@centos7-1 config]# mkdir /elk/data /elk/logs -p
[root@centos7-1 config]# chown -R elk.elk /elk/
[root@centos7-1 config]# chown -R elk.elk /usr/local/elasticsearch/
Copy the code

Installation and Configuration

[root@centos7-1 ~]# tar ZXF elasticSearch -5.3.1.tar.gz -c /usr/local/
[root@centos7-1 ~]-s /usr/local/elasticSearch -5.3.1 /usr/local/elasticSearch
[root@centos7-1 ~]# cd /usr/local/elasticsearch/config/
Copy the code

Modifying a Configuration File

[root@centos7-1 config]# egrep -v "^#|^$" elasticsearch.yml
cluster.name: myelk    # the cluster nameName: centos7-1 path.data: /elk/data path.logs: /elk/logs net. host: 0.0.0.0 http.port: centos7-1 path.data: /elk/data path.logs: /elk/logs net. host: 0.0.0.0 HTTP. 9200 discovery.zen.ping.unicast.hosts: ["host1"."host2"]
Copy the code

Start the service

[root@centos7-1 config]# su - elk
[elk@centos7-1 ~]$ who
root     pts/0        2017-10-13 22:22 (10.0.0.1)
[elk@centos7-1 ~]$ cd /usr/local/elasticsearch/bin/
[elk@centos7-1 bin]$ ./elasticsearch&
Copy the code

Install the Logstash

[root@centos7-1 ~]# tar zxf logstash-5.3.1.tar.gz -C /usr/local/
[root@centos7-1 ~]# /usr/local/logstash-5.3.1/bin/logstash -e 'input {stdin {}} output {stdout {}}' # /usr/local/logstash-5.3.1/bin/logstash -e 'input {stdin {}} output {stdout {}}'
Sending Logstash's logs to /usr/local/logstuck-5.3.1 /logs which is now configured via log4j2.properties [2017-10-16T01:39:36.983][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/local/logstash-5.3.1/data/queue"} [2017-10-16T01:39:37.181][INFO][logstash. Agent] No persistent UUID file  found. Generating new UUID {:uuid=>"a2e3b22a-4785-42f6-a073-f7fad4d60a44", :path=>"/usr/local/logstash-5.3.1/data/uuid"} [2017-10-16T01:39:37.623][INFO][logstash {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "Pipeline. Max_inflight "=>125} [2017-10-16T01:39:37.700][INFO][logstash. Pipeline plugin is now waiting for input: [2017-10-16T01:39:38.042][INFO][logstash. Agent] Successfully started logstash API endpoint {:port=>9600} Hello The 2017-10-16 T05:39:45. 692 z hello centos7-1Copy the code

Origination profile

[root @ logstash centos7-1-5.3.1]# cd config/
[root@centos7-1 config]# ll
total 20
-rw-rw-r-- 1 root root 1738 Apr 17 12:07 jvm.options
-rw-rw-r-- 1 root root 3958 Apr 17 12:07 log4j2.properties
-rw-rw-r-- 1 root root 4433 Apr 17 12:07 logstash.yml
-rw-rw-r-- 1 root root 1701 Apr 17 12:07 startup.options
[root@centos7-1 config]# vim logstash.conf
input { stdin { } }
output {
    stdout { codec=> rubydebug }
}
Copy the code

Logstash uses input and output to define the configuration of input and output for log collection. In this case, input defines an input called “stdin” and output defines an output called “stdout”. No matter what character we type, Logstash returns the character in some format, Where output is defined as “stdout” and the codec parameter is used to specify the logstash output format [root@centos7-1 config]# /usr/local/logstuck-5.3.1 /bin/logstash -f/usr/local/logstash – 5.3.1 / config/logstash. Conf

Install Kibana

[root@centos7-1 ~]# tar zxf kibana-5.3.1-linux-x86_64.tar.gz -C /usr/local/
[root@centos7-1 ~]# CD/usr/local/kibana 5.3.1 - Linux - x86_64 / config /
[root@centos7-1 config]# vim kibana.yml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "10.0.0.5"
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://10.0.0.5:9200"
# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana"
Copy the code

Enable service [root@centos7-1 config]# /usr/local/kibana-5.3.1-linux-x86_64 /bin/kibana&

[root@centos7-1 config]# lsof -i :5601
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
node  11535 root  12u  IPv4  30703  0t0  TCP centos7-1:esmagent (LISTEN)
Copy the code

Logging In to the WEB UI

Test Kibana connection to Elasticsearch

After the components are installed, you need to test whether the components can properly collect, store, and display logs

[root@centos7-1 config]# CD/usr/local/logstash - 5.3.1 / config /
[root@centos7-1 config]# vim logstash.conf
input {
       stdin { }
      }
output {
    elasticsearch {
         action => "index"
         hosts => "10.0.0.5:9200"
         index => "logstash-%{+YYYY-MM}"
   }
}  
[root@centos7-1 config]# / usr/local/logstash 5.3.1 / bin/logstash -f/usr/local/logstash - 5.3.1 / config/logstash. Conf
Copy the code

Log in to the WEB UI to check whether logs are generated