Install the ELK

Elasticsearch download for elasticSearch

www.elastic.co/downloads/e…

Logstash Download address:

www.elastic.co/downloads/l…

Download kibana

www.elastic.co/downloads/k…

Installation reference (it is recommended to download the package from the official website and unpack it, brew installation will lack x-pack plug-in) :

www.cnblogs.com/liuxiaoming…

steps

Part 1. Elasticsearch

To start ElasticSearch, visit http://localhost:9200

sh ./bin/elasticsearch
Copy the code

2. Logstsh part

Log file format
log_format  wwwlogs  '$remote_addr - $remote_user [$time_local] "$request" '
               '$status $body_bytes_sent "$http_referer" '
               '"$http_user_agent" $http_x_forwarded_for $request_time';
Copy the code
[07/Sep/2017:17:24:53 +0800]"GET / 1. HTTP / 1.1 PHP"200, 25327,"-" "Mozilla / 5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"[07/Sep/2017:17:24:54 +0800]"The GET/favicon. Ico HTTP / 1.1"404, 571,"http://192.168.44.111:88/1.php" "Mozilla / 5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"[08/Sep/2017:17:14:00 +0800]"The GET/HTTP / 1.1"403, 169,"-" "Curl /7.19.7 (x86_64-REDHat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"[08/Sep/2017:17:14:00 +0800]"The GET/HTTP / 1.1"403, 169,"-" "Curl /7.19.7 (x86_64-REDHat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2"[20/Aug/2019:04:23:00 +0800]"The GET/HTTP / 1.1"403, 571,"-" Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"[20/Aug/2019:05:29:45 +0800]"The GET/HTTP / 1.1"403, 169,"-" "Mozilla / 5.0 (X11; Ubuntu; Linux x86_64; The rv: 36.0) Gecko / 20100101 Firefox / 36.0"
183.136.190.62 - - [08/Sep/2019:16:27:28 +0800] "The GET/HTTP / 1.1"403, 571,"-" Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"[09/Sep/2019:17:35:29 +0800]"GET/robots. TXT HTTP / 1.1"404, 571,"-" "Mozilla / 5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36"
Copy the code
Go to the Config folder and download the IP address category query library provided by the Logstash file
cd config/

wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
Copy the code
Adding a Configuration File
// Create a new configuration file vim. /config/ip-test.conf // Specify the configuration input {file {path =>"/ usr/local/Cellar/logstash - 6.6.0 / config/nginx log"
        type= >"nginx"
        start_position => "beginning"
    }
}

filter {
    grok {
        match => {"message"= >"%{IPORHOST:remote_addr} - - \[%{HTTPDATE:time_local}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{INT:status} %{INT:body_bytes_sent} \"-\" \"%{DATA:http_user_agent}\""}
    }
    geoip {
        source= >"remote_addr"
        target => "geoip"
        database => "/ usr/local/Cellar/logstash - 6.6.0 / config/GeoLite2 - City. The MMDB." "
        add_field => ["[geoip][coordinates]"."%{[geoip][longitude]}"]
        add_field => ["[geoip][coordinates]"."%{[geoip][latitude]}"]
    }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => true
        index => "logstash-map-%{+YYYY-MM}"// Run the check configuration file command to check whether the configuration is incorrectly written./bin/logstash-f ./config/ip-test.conf -t
Copy the code
Configuration to explain
  • Grok: A plug-in for rematching content
  • Geoip: plug-in for querying IP
  • Source: the field that needs to be handled by the geoIP plugin. The field is usually IP
  • Target: which field should the parsed geoIP address be stored in? The default field is geoIP
  • Database: IP address category query database
  • Add_field: the two lines here add latitude and longitude, the map shows the regions by latitude and longitude
Start the logstash and the data will be saved by ElasticSearch
// The startup command must follow the configuration file path./bin/logstash-f {configPath}
./bin/logstash -f. / config/IP - test. Conf1. Logstsh partsCopy the code
Elasticsearch data display

3. Kibana part

Add the Amap interface to the last line of the Kibana configuration file./config/kibana.yml as follows
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&
z={z}'
Copy the code
Delete bundles
rm -rf ./optimize/bundles
Copy the code
To launch Kibana, visit http://localhost:5601
./bin/kibana
Copy the code
Set Index Patterns, create logStash *, and select @TIMESTAMP as the Time Filter field name

Create VisualIzations, select coordinate Map type, select LogStash * as the analysis object, and modify the GeoCoordinates option on the left to see the IP source distribution Map

In the upper right corner, you can also add time filtering conditions and automatic refresh frequency