Docker deployment FileBeat + ELK 7.14.1

1. Pre-preparation

  • Deployment environment: centos + Docker
  • Mirror selection:
    • Elasticsearch: 7.14.1
    • Kibana: 7.14.1
    • Logstash: 7.14.1
    • Elastic/filebeat: 7.14.1
  • Environmental workflow 
    Graph TD Start --> fileBeat; Filebeat reads log files --> logstash receives fileBeat data; Logstash accepts FileBeat data --> ElasticSearch accepts fileBeat data; Elasticsearch receives data --> Kibana displays data;

2. Install the image

1. Create a Docker bridge network

Create a network for interaction between containers

    docker network create elk
Copy the code

2. Create a folder

Create a folder to store configuration files.

3. Create configuration files for each software

  1. es
vim elasticsearch.yml # start cluster.name: "elasticsearch" node.name: Initial_master_nodes: "es-node-1" network. Host: "es-node-1" network. Allow-origin: "*" # Indicates that all domain names are supported # endCopy the code
  1. logstash
Vim logstash. Yml # # start module address configuration file path. The config: / usr/share/logstash/config / *. Conf xpack. Monitoring. Enabled: true xpack.monitoring.elasticsearch.hosts: ["http://elasticsearch:9200"] #end vim logstash.cof input{ beats { port => 5044 codec => "json" id => 'beat_test' } } output{ elasticsearch { hosts => ["http://elasticsearch:9200"] index => "logtest" } stdout { codec => rubydebug } }Copy the code
  1. kibana
Vim Kibana.yml server.host: "0.0.0.0" elasticSearch. hosts: [ "http://elasticsearch:9200" ] monitoring.ui.container.elasticsearch.enabled: trueCopy the code
  1. filebeat

vim filebeat.yml


filebeat.inputs:

  - type: log

    enabled: true

    paths:

      - /usr/share/filebeat/data/*.log

    fields_under_root: true

output.logstash:

  hosts: ["logstash:5044"]

Copy the code

4. Write the docker – compose

Vim docker-compose. Yml version: '3' Services: elasticSearch: image: elasticSearch :7.14.1 Ports: -9200 :9200 restart: always container_name: elasticsearch volumes: # [localfile]:[docker container file or address] - /usr/local/docker/ELK/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml networks: Restart: always container_name: Kibana Networks: -elk Volumes: - / usr/local/docker/ELK kibana/kibana yml: / usr/share/kibana/config/kibana yml logstah: image: logstash: 7.14.1 restart: always container_name: logstash networks: - elk ports: - 5044:5044 - 5045:5045 volumes: - /usr/local/docker/ELK/logstash/logstash.conf:/usr/share/logstash/config/logstash.conf - /usr/local/docker/ELK/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml - / usr/local/docker/ELK/log /, / usr/share/logstash/data/filebeat: image: "elastic/filebeat: 7.14.1" restart: always container_name: filebeat networks: - elk volumes: - / usr/local/docker/ELK filebeat/filebeat yml: / usr/share/filebeat/filebeat yml log on local - # docker container / usr/local/docker/ELK/filebeat/log: / usr/share/filebeat/log # filebeat - read the local address / usr/local/docker/ELK filebeat/data: / usr/share/filebeat/data networks: # using local created network ELK: external: trueCopy the code

5. Run

 docker-compose up -d

Copy the code

3. Problems encountered

  1. The file hangs in question

> A: At the beginning of the configuration of the image startup directory, it was copied directly from the Internet. Finally, it was found that the startup was successful, but the configuration did not take effect. Finally, it was found that the configuration file path in the Docker image was incorrect, so the container startup configuration did not take effect. /usr/share/* = /usr/share/* = /usr/share/*

  1. Kibana Server is not ready yet

> ANSWER: 1) ES failed to start 2) Network problem, Kibana could not connect to ES

  1. Logstash connection problem

> Logstash container configuration has a configuration sample file linked to ES. After startup, it will connect to localhost:9200. I didn’t notice it at first because I thought there was something wrong with my configuration file