Basic overview of ELK
ELK is short for Elasticsearch, Logstash, and Kibana. It is often used to deploy distributed system log services.
- Elasticsearch is a global real-time full-text search and analysis engine that collects, analyzes, and stores data. Is a set of open REST and JAVA API architecture to provide efficient search capabilities, scalable distributed system. It is built on top of the Apache Lucene search engine library.
- Logstash: A tool for collecting, analyzing, and filtering logs. It supports almost any type of logging, including system logging, error logging, and custom application logging. It can receive logs from many sources, including syslog, messaging (such as RabbitMQ), and JMX, and it can output data in a variety of ways, including email, WebSockets, and Elasticsearch.
- Kibana: Web-based graphical visualization interface for searching, analyzing, and visualizing log data stored in Elasticsearch metrics. It utilizes Elasticsearch’s REST interface to retrieve data, allowing users not only to create custom dashboard views of their own data, but also to query and filter data in special ways.
Basic architecture diagramelk-architecture] :
The ElasticSearch service is deployed
The basic steps of Docker deployment application service: Search[query image]->Pull[Pull image]->Run[deploy image]
1. Query Elasticsearch image: Docker search Elasticsearch
Ps [Notes] :
1. Generally, pull image resources are pulled from the official Docker repository [Docker-hub], or the Docker cloud repository aliyun-Docker built by ourselves
2. ELK images selected for this tutorial are based on ELK’s official Docker repository, Elastic-io
2. Pull Elasticsearch mirror: docker pull docker. Elastic. Co/Elasticsearch/Elasticsearch: 7.3.1
Ps [Notes] :
1. This tutorial adopts 7.3.x version, currently the latest version 7.4.x[mainly used 7.3.x version to build ali Cloud to avoid pit problems]
2. There may be a [NET/HTTP: TLS Handshake timeout] problem. Try it several times
3. Modify the name of mirror: docker tag docker. Elastic. Co/elasticsearch/elasticsearch: 7.3.1 elasticsearch: the latest
Ps [Notes] :
Docker tag source-image[source image] target-image[target image] docker tag source-image[source image] target-image This is equivalent to renaming the mirror
2. To pull kibana [docker. Elastic. Co/kibana/kibana: 7.3.1] and logstash [docker. Elastic. Co/logstash/logstash: 7.3.1] have suggested changes.
4. Run the following command to deploy the image service: docker run -itd -p 9200:9200 -p 9300:9300 –restart=always –privileged=true –name elasticsearch-server -e “discovery.type=single-node” -e ESJAVAOPTS=”-Xms=512m -Xms=512m” elasticsearch:latest
The/usr/share/elasticsearch/config/usr/share/elasticsearch/logs to check the container list: docker ps –format “table {{.ID}}t{{.Names}}t{{.Ports}}”
1. Open ports [9200 and 9300]->9200 as the Http protocol, mainly used for external communication. 9300 as the Tcp protocol, jar communication is through Tcp protocol, usually deployed cluster communication through 9300. Recommended [Host custom port :9200]
2.–restart=always: Configure the container restart policy. When the host restarts, it does not need to be manually started
3.– Privileged: Privileged container [true-root, false- privileged container user]
4. For the deployment network mode, the default bridge mode and host mode are recommended
Docker exec-it container-id[container ID] or container-name[container name] /bin/bash Example: docker exec-it container-id[container id] /bin/bash docker exec -it f2d2e97da375 /bin/bash #f2d2e97da375-> container-id
Modifying a configuration file:
[root@f2d2e97da375 elasticsearch]# ls LICENSE.txt NOTICE.txt README.textile bin config data jdk lib logs modules plugins [root@f2d2e97da375 elasticsearch]# [root@f2d2e97da375 elasticsearch]# cd config [root@f2d2e97da375 config]# ls elasticsearch.keystore elasticsearch.yml jvm.options log4j2.properties role_mapping.yml roles.yml users users_roles [root@f2d2e97da375 config]# vi elasticsearch.ymlCopy the codeAdd cross-domain configuration: http.cers. enabled: true && http.cers. allow-origin: “*”
Cluster.name: "docker-cluster" network.host: 0.0.0.0 http.coron. enabled: true http.coron. allow-origin: "*"Copy the codeThen exit the container and restart the container on the host: docker restart container-id[container ID] or container-name[container name]docker restart f2d2e97DA375
[root@f2d2e97da375 config]# exit
exit
[root@centos-meteor ~]# docker restart f2d2e97da375
f2d2e97da375
[root@centos-meteor ~]# Copy the codePs [Notes] :
1. Container access method: including using docker attach command or Docker exec command,
The Docker exec command is recommended. The reason:
* Docker attach: Using exit to exit the container will cause the container to stop
* Docker exec: Exit the container without stopping the container
* reference docker into several methods of container blog – [docker into several methods of container] (https://www.cnblogs.com/yanshicheng/p/9452545.html)
2. If Docker has installed the visual interface Portainer, it is recommended to use this method to enter the container:
The elasticSearch-head service is deployed
Elasticsearch-head: A Web front – end interface for ElasticSearch cluster. It is built using Nodjs and is used to view ElasticSearch information
Docker pull mobz/ elasticSearch-head :5
[root@centos-amber ~]# docker pull mobz/elasticsearch-head:5
5: Pulling from mobz/elasticsearch-head
75a822cd7888: Pull complete
57de64c72267: Pull complete
4306be1e8943: Pull complete
871436ab7225: Pull complete
0110c26a367a: Pull complete
1f04fe713f1b: Pull complete
723bac39028e: Pull complete
7d8cb47f1c60: Pull complete
7328dcf65c42: Pull complete
b451f2ccfb9a: Pull complete
304d5c28a4cf: Pull complete
4cf804850db1: Pull complete
Digest: sha256:55a3c82dd4ba776e304b09308411edd85de0dc9719f9d97a2f33baa320223f34
Status: Downloaded newer image for mobz/elasticsearch-head:5
docker.io/mobz/elasticsearch-head:5
[root@centos-amber ~]# Copy the codeDocker tag mobz/ elasticSearch-head :5 Elasticsearch-head :latest elasticsearch-head: specifies the name of the elasticsearch-head image
[root@centos-amber ~]# docker tag mobz/elasticsearch-head:5 elasticsearch-head:latest [root@centos-amber ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE grafana/grafana latest 05d1bcf30d16 7 days ago 207MB nginx latest 540a289bab6c 3 weeks ago 126MB prom/prometheus latest 2c8e464e47f4 3 weeks ago 129MB moxm/sentinel-dashboard latest 0CCaAC81584E 4 weeks ago 167MB Portainer latest 4cda95efb0e4 4 weeks ago 80.6MB Portainer/Portainer latest 4cda95efb0e4 4 weeks ago 80.6MB Portainer/Portainer latest 4cda95efb0e4 4 Weeks ago 80.6MB Apache/Skywalking - UI latest FA66CA9C9862 2 months ago 123MB Apache/Skywalking - oAP-server latest 376 a37cdf65c 2 have a line 190 MB docker. Elastic. The co/kibana kibana 7.3.1 b54865ba6b0b 2 have a line 1.01 GB Docker. Elastic. Co/elasticsearch/elasticsearch 7.3.1 3 d3aa92f641f 2 have a line 807 MB elasticsearch latest 3 d3aa92f641f 2 Months AGO 807MB PROM /node- EXPORTER latest E5A616E4B9CF 5 months ago 22.9MB Google/CAdvisor latest EB1210707573 12 Elasticsearch -head latest B19a5C98e43b 2 years ago 824MB mobz/ elasticSearch -head 5 B19a5C98e43b 2 years ago years ago 824MB tutum/influxdb latest c061e5808198 3 years ago 290MB [root@centos-amber ~]#Copy the code3. Deploy elasticSearch-head: docker run -itd –restart=always –privileged=true -p 9100:9100 –name elasticsearch-head-server Docker ps –format “table {{.id}}t{{.names}}t{{.ports}}”
Set up and deploy the Kibana service
1. Pull Kibana mirror: docker pull docker. Elastic. Co/Kibana/Kibana: 7.3.12. Modify Kibana mirror name: docker tag docker. Elastic. The co/Kibana/Kibana: 7.3.1 Kibana: latest3. Deploying Kibana image containers: docker run -itd -p 5601:5601 –restart=always –privileged=true –linkelasticsearch-server:elasticsearch –name kibana-server -e ELASTICSEARCH_URL=http://elasticsearch:9200 kibana:latest
Build and deploy the Logstash service
1. Pull Logstash mirror: docker pull docker. Elastic. Co/Logstash/Logstash: 7.3.12. Modify Kibana mirror name: docker tag docker. Elastic. The co/logstash/logstash: 7.3.1 logstash: latest3. Deploying Kibana image containers: docker run -itd –restart=always –privileged=true -p 5043:5043 –name logstash-server –link elasticsearch-server:elasticsearch logstash:latest4. Go to container – modify configuration logstash. Yml:
HTTP. Host: "0.0.0.0" xpack. Monitoring. Elasticsearch. Url: http://host-ip:9200 xpack. Monitoring. Elasticsearch. Username: elastic xpack.monitoring.elasticsearch.password: changmeCopy the code[Precautions] : 1. Host-ip is the local IP address. 5.
# # the default configuration = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = # input {# beats # # # port = > {5044}} # # # the output {stdout {codec = > Rubydebug # # #}} = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = # add configuration input {file {codec = > json path = > Grok {match => {"message" => grok {match => {"message" => "%{DATA:timestamp}\|%{IP:serverIp}\|%{IP:clientIp}\|%{DATA:logSource}\|%{DATA:userId}\|%{DATA:reqUrl}\|%{DATA:reqUri}\|% {DATA:refer}\|%{DATA:device}\|%{DATA:textDuring}\|%{DATA:duringTime:int}\|\|"} } } output { elasticsearch{ hosts=> "http://host-ip:9200" } }Copy the code6. Exit containers and restart elK-related containers on the host: docker restart Elk-related container services
Ps [notes] : If Docker installed a visual interface Portainer, you can operate in the interface:
7. Visit http://remote-ip:5601/ and then operate the Kibana panel
Set up and deploy the APM-Server and Filebeat services
The procedure is similar to the above except that the configuration file and port may be different: Docker pull docker. Elastic. Co/beats/filebeat: 7.3.1 docker pull docker. Elastic. The co/apm/apm – server: 7.3.1
Change the image name: Docker tag docker. Elastic. Co/beats/filebeat: 7.3.1 filebeat: latestdocker tag docker. Elastic. The co/apm/apm – server: 7.3.1 apm-server:latest
Deployment container: docker run -itd –restart=always –privileged=true -p 5044:5044 –name filebeat-server –link logstash-server:logstash filebeat:latest
docker run -itd –restart=always –privileged=true -p 8200:8200 –name apm-server –link elasticsearch-server:elasticsearch apm-server:latest –strict.perms=false -e -E output.elasticsearch.hosts=[“elasticsearch:9200”]
Finally, modify the configuration file to integrate related resources and restart the container service
Ps [Notes] : You can refer to the official documentation: elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/7.4/docker.htmlkibana:https://www.elasti Arthur c. o/guide/en/kibana / 7.3 / docker. Htmllogstash: https://www.elastic.co/guide/en/logstash/7.3/docker.htmlfilebeat:https://ww W.e lastic, co/guide/en/beats/filebeat / 7.3 / running – on – docker. Htmlapm – server: https://www.elastic.co/guide/en/apm/server/7.3 /running-on-docker.html
Development SpringBoot+Elasticsearch integration actual combat
[1] Integrated Maven configuration:
Select * from ElasticSearch; select * from ElasticSearch
<! -- https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch --> <dependency> Elasticsearch </groupId> <artifactId> elasticSearch </artifactId> <version>6.4.2</version> </dependency> <! -- https://mvnrepository.com/artifact/org.elasticsearch.client/transport --> <dependency> < the groupId > org. Elasticsearch. Client < / groupId > < artifactId > transport < / artifactId > < version > 6.4.2 < / version > < exclusions > <exclusion> <groupId>org.elasticsearch</groupId> <artifactId>elasticsearch</artifactId> </exclusion> </exclusions> </dependency>Copy the code(2) Use SpringDataElasticSearch to create a connection
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-data-elasticsearch</artifactId> </dependency>Copy the codeOr:
Dependencies {the compile (' org. Springframework. The boot: spring - the boot - starter ') / / using SpringDataElasticSearch only need to add a dependence on the box compile('org.springframework.boot:spring-boot-starter-data-elasticsearch') }Copy the codeAdvantages and disadvantages of the two methods: (1) Advantages: It is separated from the framework and does not need to consider compatibility with Spring version during the integration process. Disadvantages: It uses native API to operate ES, resulting in a large amount of code and difficulty in writing
(2) Advantages: Native API encapsulation provides ElasticsearchRepository, operation ES is very simple, the same as JPA disadvantages: born in the Spring family, easy to conflict with SpringBoot, SpringData version
[2] Parameter connection configuration mode:
ElasticSearch port 9200 is used to enable HTTP REST APIS to access ElasticSearch. Port 9300 elasticSearch. IP =192.168.30.128 ElasticSearch. port=9300 elasticSearch. pool=5 elasticsearch.cluster.name=my-applicationCopy the codeNode. The name: “elasticsearch server -” network. Host: 0.0.0.0 network. Bind_host: 0.0.0.0 network. Publish_host: 0.0.0.0 HTTP. Cors. Enabled: truehttp cors. Allow – origin: “*” is the bootstrap. Memory_lock: truetransport. TCP. Port: 9300 transport.tcp.com press: truehttp. Maxcontentlength: 128 MB copyright statement: this article original articles for bloggers, follow the relevant copyright agreement, if reproduced or share please attach the original source links and sources.