Author: Wang Bei
Why ELK is recommended:
What is the most important thing to do when an online server has a problem? When we need to monitor and track the health of the server in real time, what do we take to analyze it? People would say, look at the log, analyze the log. Yes, logging is the closest thing to a server developer’s heart, looking at a large number of logs of various types on a daily basis to locate problems, do statistical analysis, and so on. For example, if there is a bug online, we SSH to the online server, CD to the server’s log directory, and then less the log file of the day, execute shift+G,? Error,n, after a series of operations to find the corresponding abnormal situation, find out the abnormal cause. But, so low, so low efficiency ah, why? First of all, we have to log on to the server, online are basically server cluster, less is more than a dozen units, many hundreds of units, light this one a server to login is a huge and complicated work, secondly, sometimes we even various polymerization (such as compared to multiple machines at the same time point of log), or abnormal based on multiple keyword search, and one and, or, to pay, And, difference, complement, sort and other operations, and the corresponding speed must be strong, then this time no matter how grep AWk estimate is useless. In the face of service cluster, in the face of massive log, there is no centralized visualization of log management, the program ape GG estimates really to kneel. In order to liberate many Webserver programs, in order to liberate productivity and improve efficiency, in order to view and analyze online logs more accurately and in real time, the industry put a big move: with the strong ELK log analysis platform to achieve centralized visual log management. Immediately if there is something wrong with the online environment, can accurate positioning, ELK is master, take in million-man army general heads, confidence, so at this time the ELK is very suitable for, in addition to this, of course, ELK also often puts glorious the operations work in a big way, the application level of real-time monitoring, very suitable for early warning of some important core services.
Here are two simple examples to illustrate the power of ELK:
(1) According to the filtering rules, the pie chart of ERROR ratio of all servers within 24 hours is generated to monitor the health degree of servers. (At present, 7 servers of GACHA project are connected to ELK platform and are on trial. This figure is the abnormal proportion of these 7 servers in the past 24 hours.)
(2) generate the 24 hours each time within the scope of the number of abnormal histogram, click on the bar graph below you can see the specific exception stack (histogram statistics is GACHA project server cluster in the past 24 hours each time period, the number of abnormal histogram is exception list below, in the first picture on the left side of the option to add abnormal display item list, The second image shows the result of selecting the host, Priority, stack trace, and message items, and the third image shows the specific exception stack details.
Ok, see these, is not the heart slightly moved, in fact, ELK is far more powerful than this, play a chicken to continue to see.
ELK’s brief introduction:
ELK consists of ElasticSearch, Logstash and Kiabana. Official website: www.elastic.co/products
Elasticsearch is an open source distributed search engine with features such as distributed, zero configuration, automatic discovery, index automatic sharding, index copy mechanism, restful interface, multi-data source, automatic search load, etc.
(2) Logstash is a completely open source tool that collects, filters, and stores your logs for future use (e.g., search).
(3) Kibana is also an open source and free tool that provides a log analysis friendly Web interface for Logstash and ElasticSearch to help you aggregate, analyze and search important data logs.
From this, we can see that the ELK combination can actually be regarded as an MVC Model. Logstash receives the log transmission request of the application server and collects the log, which is equivalent to controller, ElasticSearch for data source storage and search, which is equivalent to Model. Kibana offers a friendly Web interface, the equivalent of View.
ELK build:
Next, I walk through the entire ELK setup process:
Step 1: Build ElasticSearch
(1) Download ElasticSearch first (install JDk1.7, I won’t go into details), I’m using version 2.4.1 of ElasticSearch:
Wget download. Elastic. Co/elasticsear…
(2) Decompression:
The tar – ZXVF elasticsearch – against 2.4.1. Tar. Gz
(3) Configuration:
CD elasticsearch – against 2.4.1
vim config/elasticsearch.yml
#ES cluster name
cluster.name : es_cluster
# Name of the current node
node.name : node0
Is the current node allowed to be elected as the master node
#node.master: true
# Whether the current node allows data storage
node.data: true
Number of index fragments
index.number_of_shards: 5
# number of index copies
index.number_of_replicas: 1
Index data store path,
path.data : /home/hzwangbei/elasticsearch/data
path.logs : /tmp/elasticsearch/logs
The default ES root directory is config
#path.conf: /path/to/conf
The default ES root directory is the plugins directory
#path.plugins: /path/to/plugins
# Current hostname or IP, which is used by ES to monitor, the best configuration of netease cloud host 0.0.0.0, so that both private IP and machine room IP can be monitored.
Network. Bind_host: 0.0.0.0
Set the IP address for other nodes to interact with this node. Netease Cloud host Set the equipment room IP address here
# network. Publish_host: 0.0.0.0
network.port : 9200
# IndexCache expiration time
#index.cache.field.expire:100m
The default type is resident, which literally means resident, increasing until memory runs out. Soft is when the memory is insufficient, first clear off the occupation, and then put into the memory. Soft is equivalent to the relative memory size. Resident unless the memory is large enough.
index.cache.field.type: soft
The default value is 3s, which is set to ping connection timeout when the cluster automatically discovers other nodes.
Discovery. Zen. Ping. A timeout: 40 s # set whether to open the multicast discovery nodes, the default is true. #discovery.zen.ping.multicast.enabled: false
If es fails to discover any other nodes in the cluster, select the IP address of the master node.
# discovery. Zen. Ping. Unicast. Hosts: [” node 1 IP “, “node 2 IP”, “node 3 IP”]
(4) Plug-in installation
Install the head plugin:
./bin/plugin install mobz/elasticsearch-head
(5) Start ES
./bin/elasticsearch
Unfortunately, the error was reported. Later, I saw in a foreign forum that the user and group of ES startup could not be the same as the user and group of the current directory. I compared it, sure enough, I started ES as root user, while the current directory belongs to the user root.
As you can see, ES started up successfully. Ok, so here we are, we have successfully taken a step towards ELK.
Open the link: http://localhost:9200/_plugin/head/
Kibana. Click Browse Data, select Applog index, and view the logs stored in the Applog index.
All right, so you’re wondering where all this data is coming from. As I mentioned earlier, ES is a Model layer, so of course the data is collected from the Controller layer, the Logstash layer. Lets starts our second step, setting up the Logstash.
Step 2: Install the Logstash
Logstash function module diagram
In fact, it is a collector, and we need to specify Input and Output for it (of course, Input and Output can be multiple). Since we need to export Log4j logs from the Java code to ElasticSearch, the Input is Log4j and the Output is ElasticSearch.
(1) Download logstash
Wget download. Elastic. Co/logstash/lo…
(2) Decompress
The tar – ZXVF logstash – 2.4.0. Tar. Gz
(3) Create the config folder in the root directory and create the log4j_to_es
cdLogstash - 2.4.0 mkdir config vimlog4j_to_es.conf
input {
log4j {
mode => "server"
host => "localhost"// IP address of the netease cloud host room port => 4567}} filter {#Only matched data are send to output.
}
output {
elasticsearch {
action => "index" #The operation on ES
hosts => "localhost:9200" #ElasticSearch host, can be array.
index => "applog" #The index to write data to.}}Copy the code(4) Start the logstash
./bin/logstash agent -f config/log4j_to_es.conf
Logstash is a collector. We just configured the log4J collection configuration in our configuration file, so in our project, we can add a socket appender to the log4J configuration
# appender socket
log4j.rootLogger=${log4j.level}, stdout,socket
log4j.appender.socket=org.apache.log4j.net.SocketAppender
log4j.appender.socket.Port=4567
#log4j.appender.socket.Threshold=ERROR
log4j.appender.socket.RemoteHost=localhost
log4j.appender.socket.layout=org.apache.log4j.PatternLayout
log4j.appender.socket.layout.ConversionPattern=%d %-5p %c%x - %m%n
log4j.appender.socket.ReconnectionDelay=10000Copy the codeOk, so with ES and Logstash set up, you can basically collect and store logs. However, this is not enough. At the beginning of the article, we gave two examples of pie charts and bar charts. How are these two kinds of charts generated? Then take a look at our View layer Kibana and start our third step: Installing Kibana.
Step 3: Kibana installation
(1) Download Kibana
Wget download. Elastic. Co/kibana/kiba…
Note that there are many versions of Kibana, and you should refer to the ES version when downloading. Because our ES is installed with 2.4.1, we use Kibana4.6.1 here. When selecting Kibana version, we must ensure that it is compatible with ES version.
(2) Decompress Kibana
Tar – ZXVF kibana 4.6.1 – Linux – x86_64. Tar. Gz
(3) Configure Kibana
root@wangbei-ThinkPad-E560:/usr/local/elk# CD kibana - 4.6.1 - Linux - x86_64
root@wangbei-ThinkPad-E560:/usr/localElk/kibana - 4.6.1 - Linux - x86_64# vim config/kibana.yml
server.port: 5601
server.host: "localhost"
elasticsearch.url: http://localhost:9200
kibana.index: ".kibana"Copy the code(4) Start Kibana
Root @ wangbei ThinkPad – E560: / usr/local/elk/kibana – 4.6.1 – Linux – x86_64 #. / bin/kibana
Well, here, done, browser open: http://localhost:5601
To use Kibana later, you need to configure at least one Index name or Pattern, which is used to determine the Index in ES during analysis. Kibana will automatically load the doc field under this Index and automatically select the appropriate field for the time field in the icon:
Once you’ve created the index, you can see the power of ELK!
Fast search, powerful use of search syntax, great graphics drawing, it is blockhouse!
The most important is centralized log management, visual presentation, real-time monitoring!
No more SSH less grep, complicated log analysis is done!
Have to sigh, good welfare really come!
Note:
The use of ELK line should also pay attention to ES memory configuration, network Settings and other factors.
Netease Cloud Free experience pavilion, 0 cost experience 20+ cloud products!
For more information about NETEASE’s r&d, product and operation experience, please visit netease Cloud Community.
Relevant article: “recommended” to know things by learning | artificial intelligence, machine learning and deep learning how to application in the field of network security?