ELK profile

What is ELK? ELK is a combination of Elasticsearch, Logstash, and Kibana open source software, each of which is used to perform different functions. ELK is also called ELKstackwww.elastic.co/The main advantages of ELK are as follows:

Elasticsearch is a real-time full-text index with powerful search capabilities. 2. Configuration is relatively simple: ElasticSearch uses JSON interface, LogStash uses module configuration, kibana configuration file part is even simpler. 3. High retrieval performance: Based on excellent design, although every query is real-time, it can also reach the query second-level response of tens of billions of data. 4. Linear cluster scaling: ElasticSearch and Logstash both allow linear scaling. 5. Front end operation is gorgeous: Kibana's front end design is gorgeous and simple to operate.Copy the code

Elasticsearch

Elasticsearch is a highly scalable full-text search and analysis engine based on Apache Lucene. It can store, search, and analyze large volumes of data in near real time. It can handle large volumes of log data, such as Nginx, Tomcat, and system logs.

Logstash

Data collection engine. It supports dynamic data collection from a variety of data sources, filtering, analysis, enrichment, unified format and other operations on the data, and then stored to the user specified location; Supports common log and custom JSON log parsing.

Kibana

Data analysis and visualization platform. Usually used in conjunction with Elasticsearch to search, analyze, and display data as a statistical chart.

ELK deployment preparations

CentOS 7.4 X86_64 is used in this experiment. Please turn off the firewall and SELinux.

IPAddress	Hostname	Mem
172.18.12.172	elk-node1	4G
172.18.12.163	elk-node2	4G

Uploading software Packages

Elasticsearch – 7.4.2 – Linux – x86_64. Tar. Gz
jdk-8u221-linux-x64.tar.gz
Kibana – 7.4.2 – Linux – x86_64. Tar. Gz
Logstash – 7.4.2. Tar. Gz
Elasticsearch-head
The node – v12.14.0 – Linux – x64. Tar. Xz

Software Package Download AddressExtraction code: SE5P

Install the JDK and configure the environment variables

[root@localhost /]# tar -xzvf jdk-8u221-linux-x64.tar.gz
[root@localhost /]# vim etc/profile
Copy the code

Add as follows:

#set java environmentJAVA_HOME = / sk/jdk1.8.0 _221 JRE_HOME =$JAVA_HOME/jre
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
export JAVA_HOME JRE_HOME PATH CLASSPATH
Copy the code

Make the configuration take effect

[root@localhost /]# source /etc/profile
Copy the code

Viewing the JDK Version

[root@localhost /]# java -version
Copy the code

Elasticsearch deployment

Unzip elasticSearch and configure bin/ elasticSearch.yml

[root@localhost tmp]# tar - ZXVF elasticsearch - 7.4.3. Tar. Gz
[root@localhost tmp]# mkdir /ELK
[root@localhost tmp]# mv elasticsearch - 7.4.3 / ELK/elasticsearch
[root@localhost /]# cd ELK/elasticsearch/config/
[root@localhost config]# vim elasticsearch.yml
Copy the code

The configuration is as follows:

cluster.name: ELK-Cluster    #ELK specifies the name of the cluster that belongs to the same cluster
node.name: elk-node1    The name of the node in the cluster
path.data: /ELK/elasticsearch/data    # Data storage directory
path.logs: /ELK/elasticsearch/data/log    # Log save directory
bootstrap.memory_lock: true    Lock enough memory when the service starts to prevent data from being written to swapNetwork. The host: 172.18.12.164Listen to the IP address
http.port: 9200    The port on which the service listens
discovery.seed_hosts: ["172.18.12.172"."172.18.12.173"]   Configure one unicast
cluster.initial_master_nodes: ["elk-node1"."elk-node2"]  
Copy the code

Configure memory limits

[root@localhost config]# vim jvm.options
Copy the code

The configuration is as follows:

-Xms2g
-Xmx2g
Copy the code

Min and Max memory limits, why are min and Max Settings the same size? Reference: www.elastic.co/guide/en/el…

Create user and data directories

[root@localhost config]# mkdir -p /ELK/elasticsearch/{data,data/log}
[root@localhost elasticsearch]# groupadd elk
[root@localhost elasticsearch]# useradd -g elk elk
[root@localhost elasticsearch]# passwd elk
[root@localhost elasticsearch]# chown -R elk:elk /ELK/elasticsearch/ -R
[root@localhost bin]# su elk
[elk@localhost elasticsearch]# cd bin/
[elk@localhost bin]# ./elasticsearch
Copy the code

ERROR: [3] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: memory locking requested for elasticsearch process but memory is not locked
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Copy the code

Startup error: need to modify the maximum number of open files and memory lock

Modifying kernel parameters

[root@localhost bin]# vim /etc/security/limits.conf
Copy the code

The configuration is as follows:

* soft nofile 65536
* hard nofile 131072
* soft memlock unlimited
* hard memlock unlimited
Copy the code

Example Change the maximum number of threads

[root@localhost bin]# vim /etc/security/limits.d/20-nproc.conf
Copy the code

The configuration is as follows:

*     soft    nproc    unlimited
Copy the code

[root@localhost bin]# vim /etc/sysctl.conf
Copy the code

The configuration is as follows:

vm.max_map_count = 655360
Copy the code

Make the configuration take effect:

[root@localhost bin]# sysctl -p
[root@localhost bin]# reboot
Copy the code

4. Start the service

[root@localhost bin]# su elk
[elk@localhost elasticsearch]# cd bin/
[elk@localhost bin]# ./elasticsearch
[elk@localhost bin]$ yum -y install net-tools
[elk@localhost bin]# netstat -nltup |grep java
Copy the code

Tcp6 00 172.18.12.164:9200 ::* LISTEN 9971/ Java tcp6 00 172.18.12.164:9300 ::* LISTEN 9971/ JavaCopy the code

Configure another bin/ elasticSearch.yml

[root@localhost tmp]# tar - ZXVF elasticsearch - 7.4.3. Tar. Gz
[root@localhost tmp]# mkdir /ELK
[root@localhost tmp]# mv elasticsearch - 7.4.3 / ELK/elasticsearch
[root@localhost /]# cd ELK/elasticsearch/config/
[root@localhost config]# vim elasticsearch.yml
-----------------------------------------------------------------------------------
cluster.name: ELK-Cluster    #ELK specifies the name of the cluster that belongs to the same cluster
node.name: elk-node2    The name of the node in the cluster
path.data: /ELK/elasticsearch/data    # Data storage directory
path.logs: /ELK/elasticsearch/data/log    # Log save directory
bootstrap.memory_lock: true    Lock enough memory when the service starts to prevent data from being written to swapNetwork. The host: 172.18.12.166Listen to the IP address
http.port: 9200    The port on which the service listens
discovery.seed_hosts: ["172.18.12.172"."172.18.12.173"]   Configure one unicast
cluster.initial_master_nodes: ["elk-node1"."elk-node2"]   
-----------------------------------------------------------------------------------
[root@localhost config]# vim jvm.options
-----------------------------------------------------------------------------------
-Xms2g
-Xmx2g
-----------------------------------------------------------------------------------
[root@localhost config]# mkdir -p /ELK/elasticsearch/{data,data/log}
[root@localhost elasticsearch]# groupadd elk
[root@localhost elasticsearch]# useradd -g elk elk
[root@localhost elasticsearch]# passwd elk
[root@localhost elasticsearch]# chown -R elk:elk /ELK/elasticsearch/ -R
[root@localhost bin]# su elk
[elk@localhost elasticsearch]# cd bin/
[elk@localhost bin]# ./elasticsearch
[elk@localhost bin]$ yum -y install net-tools
[elk@localhost bin]# netstat -nltup |grep javaTcp6 00 172.18.12.164:9200 ::* LISTEN 9971/ Java tcp6 00 172.18.12.164:9300 ::* LISTEN 9971/ JavaCopy the code

Elasticsearch – head plug-in

Unzip and install head plugin & NPM

[root@localhost tmp]# tar -zxvf elasticsearch-head.tar.gz
[root@localhost tmp]# mv elasticsearch-head /ELK/elasticsearch-head
[root@localhost tmp]# xz - d node - v12.14.0 - Linux - x64. Tar. Xz
[root@localhost tmp]# tar - XVF node - v12.14.0 - Linux - x64. Tar
[root@localhost ELK]# ln -s /ELK/nodejs/bin/node /usr/bin/node
[root@localhost tmp]# mv node - v12.14.0 - Linux - x64 / ELK/nodejs
[root@localhost ELK]# ln -s /ELK/nodejs/bin/ NPM /usr/bin/ NPM //
[root@localhost elasticsearch-head]# NPM install - registry=https://registry.npm.taobao.org / / use taobao source
[root@localhost elasticsearch-head]# npm run start
[root@localhost elasticsearch-head]# NPM run start & //
Copy the code

2. Visit http://IP___ : 9100

Modify all elasticSearch configuration files, enable cross-domain access, and restart the ElasticSearch service

[root@localhost elasticsearch]# vim /ELK/elasticsearch/config/elasticsearch.yml 
Copy the code

The configuration is as follows:

http.cors.enabled: true     
http.cors.allow-origin: "*"
Copy the code

[root@localhost bin]# ps -ef |grep elasticElk 9971 1 0 13:22 PTS /0 00:00:45 / TMP /jdk1.8.0_221/bin/ Java -xMS2g -XMx2g -xx :+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.JlwA3m72 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFil

[root@localhost bin]# kill -9 9971
[root@localhost bin]# su elk
[elk@localhost bin]$ ./elasticsearch &
Copy the code

4. Access port 9100 and change the connection address to elasticSearch address

5. Submit test data

The difference between Master and Slave:

Master’s responsibilities include collecting statistics about node status and cluster status, creating and deleting indexes, managing index allocation, and shutting down nodes. Savle’s responsibilities include synchronizing data and waiting for the opportunity to become the Master

Logstash deployment

I. Test standard input and output

[root@localhost TMP]$mv logstash-7.5.1 /ELK/logstash [root@localhost ELK]# CD logstash/bin/ [root@elk-node1 logstash]# vim /ELK/logstash/config/logstash.ymlCopy the code

Configuration logstash. Yml

HTTP. Host: "192.168.30.129" HTTP. Port: 9600Copy the code

Test standard input and output

[root@localhost bin]# ./logstash -e 'input { stdin {} } output { stdout { codec => rubydebug} }'
Copy the code

Input: hello word

{" @ version "= >" 1 ", "@ timestamp" = > 2019-12-19 T07:33:46. 725 z, "message" = > "hello word", "the host" = > "localhost"}Copy the code

Test output to file

[root@localhost bin]# ./logstash -e 'input { stdin{} } output { file { path => "/tmp/log-%{+YYYY.MM.dd}messages.gz"}}'
Copy the code

Input:

hello word

[2019-12-19T15:44:02.238][INFO][logstash. Outputs. File][main] Opening file {:path=>"/ TMP /log-2019.12.19messages.gz"} [2019-12-19T15:44:20.886][INFO][logstash. Outputs. File][main] Closing file/TMP /log-2019.12.19messages.gzCopy the code

View file contents:

[root@localhost bin]# tail/TMP /log-2019.12.19messages.gz {"message":"hello World @ ", "version" : "1", "@ timestamp" : "the 2019-12-19 T07:44:01. 261 z", "the host" : "localhost"}Copy the code

Test output to ElasticSearch

[root@linux-elk1 ~]#./logstash -e 'input {stdin{}} output {elasticSearch {hosts => ["172.18.12.164:9200"] index => "mytest-%{+YYYY.MM.dd}" }}'Copy the code

The elasticSearch server verifies the received data

[root @ localhost /] # ll/ELK/elasticsearch/data/nodes / 0 / indices/total amount 0 DRWXRWXR - x. 8 ELK ELK 15:51 on December 19 65 BOpOoG7xRQW7OVV06ufvAw drwxrwxR-x. 8 elK ELK 65 12月 19 14:51 mPIadPvetbygDVJ-i9w7uwCopy the code

Kibana deployment

Unzip and configure Kibana.yml

[root@localhost tmp]# tar -zxvf kibana-7.5.1-linux-x86_64.tar.gz 
[root@localhost tmp]# mv kibana-7.5.1-linux-x86_64 /ELK/kibana
[root@localhost tmp]# cd /ELK/kibana/config/
[root@localhost config]# vim kibana.yml
Copy the code

Configuration is as follows

Host: "172.18.12.164" # elasticSearch.hosts: [" http://172.18.12.164:9200 ", "http://172.18.12.166:9200"] # elasticsearch server address i18n. Locale: "useful - CN" # modified into ChineseCopy the code

Description of kibana. yml configuration file

Two, start and verify

[root@localhost bin]# cd /ELK/kibana/bin/
[root@localhost bin]# ./kibana --allow-root
Copy the code

Elastic Stack of actual combat

Collect yum logs

[root@elk-node1 logstash]# mkdir conf.d
[root@elk-node1 logstash]# touch conf.d/yum-log.conf
[root@elk-node1 logstash]# vim conf.d/yum-log.conf 
Copy the code

Edit the new configuration file:

input {
    file {
        path => "/var/log/yum.log"     
        type= >"systemlog"                    start_position => "beginning"        
        stat_interval => "2"    
    }
}

output {
    elasticsearch {
        hosts => ["172.18.12.172:9200"]       
        index => "logstash-%{type}-%{+YYYY.MM.dd}"}}Copy the code

Check whether the configuration file syntax is correct:

[root@elk-node1 logstash]# /ELK/logstash/bin/logstash -f /ELK/logstash/conf.d/yum-log.conf -t
Copy the code

Thread.exclusive is deprecated, use Thread::Mutex Sending Logstash logs to /ELK/logstash/logs which is now configured via log4j2.properties [the 2019-12-20 T16:53:29, 595] [INFO] [logstash. Setting. Writabledirectory] Creating directory {: setting = > "path. The queue," : the path = > "/ ELK logstash/data/queue"} [the 2019-12-20 T16:53:29, 615] [INFO] [logstash. Setting. Writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", : the path = > "/ ELK logstash/data/dead_letter_queue"} [the 2019-12-20 T16:53:30, 038] [WARN] [. Logstash config. Source. Multilocal] Ignoring the 'charge. Yml' file because modules or command line options are specified [2019-12-20T16:53:3128][INFO ][org.reflections.Reflections] Reflections took 67 ms to scan 1 urls, [2019-12-20T16:53:31.979][INFO][logstash config.test_and_exit mode. Config Validation Result: OK. Exiting LogstashCopy the code

2. Restart the Logstash service

[root@elk-node1 logstash]# /ELK/logstash/bin/logstash -f /ELK/logstash/conf.d/yum-log.conf  &
Copy the code

Elasticsearch+Logstash+Kibana

ELK profile

What is ELK? ELK is a combination of Elasticsearch, Logstash, and Kibana open source software, each of which is used to perform different functions. ELK is also called ELKstackwww.elastic.co/The main advantages of ELK are as follows:

Elasticsearch

Logstash

Kibana

ELK deployment preparations

CentOS 7.4 X86_64 is used in this experiment. Please turn off the firewall and SELinux.

Uploading software Packages

Software Package Download AddressExtraction code: SE5P

Install the JDK and configure the environment variables

Elasticsearch deployment

Unzip elasticSearch and configure bin/ elasticSearch.yml

Configure memory limits

Create user and data directories

Modifying kernel parameters

4. Start the service

Configure another bin/ elasticSearch.yml

Elasticsearch – head plug-in

Unzip and install head plugin & NPM

2. Visit http://IP___ : 9100

Modify all elasticSearch configuration files, enable cross-domain access, and restart the ElasticSearch service

4. Access port 9100 and change the connection address to elasticSearch address

5. Submit test data

The difference between Master and Slave:

Logstash deployment

I. Test standard input and output

Test standard input and output

Input: hello word

Test output to file

Input:

Test output to ElasticSearch

The elasticSearch server verifies the received data

Kibana deployment

Unzip and configure Kibana.yml

Description of kibana. yml configuration file

Two, start and verify

Elastic Stack of actual combat

Collect yum logs

2. Restart the Logstash service

Kibana visual interface configuration

Related Posts

OkHttp open source library use and source code analysis

How do front-end engineers respond to Serverless storm?

How did I make a big visual screen as easy as powerpoint?