Logging is an integral part of a robust program, and it is the foundation for an exhaustive analysis of the execution of a system. A good log system can greatly improve the efficiency of bug location.
A good log system is formed by the combination of log printing format, log printing time and log output mode. Of course, there must be a good solution to this industry-wide need. Therefore, we only need to scale the characteristics of the system and existing resources as a condition, choose different logging framework that can make the logging system quickly established, we just need to design a reasonable log output, which saves a lot of things for us.
These are all about how to plan and design the logging system in the generation stage of the log. When a large number of logs are generated, how to use or use these logs to troubleshoot problem statistics is the topic of today’s discussion.
With the expanding of business, the log level will become more and more strong, to quickly in the vast amounts of data and the data they want to become a complicated question, especially now micro service, a module of the log frequently distribution in the tens of thousands of servers, if each filter logs, obviously is not high). An excellent solution to this is the Elastic Stack (ELKB).
Development of Elastic log collection
The log collection stack provided by Elastic goes through the following phases:
ELK phase
Elasticsearch+Logstash+Kibana
At this stage:
Elasticsearch is a distributed search engine for full text search
Logstash collects and filters log data for Elasticsearch
Kibana provides visual graphical interface for Elasticsearch
ELKB phase
Elasticsearch+Logstash+Kibana+Beats
On the basis of ELK, Beats is added so that Beats shares the log collection function of the original Logstash
Beats has several specialized collection tools, focusing more on data collection, among which Filebeat is commonly used to collect file data
Beats provides the following tools:
ELKkB phase
Elasticsearch+Logstash+Kibana+kafka+Beats
In addition to ELKB, Kafka is a high-throughput distributed publish-subscribe messaging system
After Kafka is added, the data obtained by Beats is first fed to Kafka and Logstash and then processed from Kafka, making the processing process smoother
In subsequent notes, we will build an ELKB system from 0. The technical version of the example is as follows:
Elasticsearch 7.12.0
Logstash 7.12.0
Kibana 7.12.0
Beats 7.12.0
It is also useful to view es data from elasticsearch-head, but this is not required
Download link below:
Elasticsearch latest version | elasticsearch7.12.0 version – Windows | – macos elasticsearch7.12.0 version
Kibana latest version | kibana7.12.0 version – Windows | – macos kibana7.12.0 version
Latest version logstash | logstash7.12.0 version – Windows | – macos logstash7.12.0 version
Latest version filebeat | filebeat7.12.0 version – windowsX64 | – macos filebeat7.12.0 version
The specific process of system construction, we officially began to explore….