Like attention, no more lost, your support means a lot to me!

🔥 Hi, I’m Chouchou. GitHub · Android-Notebook has been included in this article. Welcome to grow up with Chouchou Peng. (Contact information at GitHub)

preface

  • ELK Stack is an open source data search/analysis solution with core components including Elasticsearch, Logstach, Kibana, and later Beants. For students on the client side, they will generally only touch the Kibana component.
  • In this article, I will briefly summarize the architectural logic behind the ELK Stack and summarize the use of Kibana. Please be sure to like and follow if you can help, it really means a lot to me.

directory

1. The background

Traditional log analysis is to directly grep/awk log information from log files. In this way, text search is slow and multidimensional query is difficult. In large systems, the architecture of distributed deployment is often adopted (different service modules are deployed on different server nodes). When faults occur, a centralized log management tool is required to collect and analyze logs on all nodes to improve fault locating efficiency.

Tip:

  • The Linux grep command is used to search for qualified strings in a file.
  • Awk is a text analysis tool.

2. ELK Stack architecture

ELK Stack components include Elasticsearch, Logstash, Beats, and Kibana. Earlier versions of ELK had only Elasticsearch, Logstash, and Kibana components, which is where the letters ELK come from. In 2015, Beats was added to ELK component. For the convenience of naming, the updated iteration versions of ELK component are all called ELK Stash. On the basis of

The following is a brief introduction to the responsible functions of several components, as we are students of the client side, as long as you understand the general purpose.

  • Elasticsearch: Open source distributed search and analysis engine (RESTful style);

  • Logstash: Open source data collection tool that allows you to collect, filter, and transform data from multiple sources before indexing it to Elasticsearch. In terms of implementation, Logstash adopts C/S architecture. The Logstash Agent is installed on the host that needs to collect data, and the Logstash Server collects data from each host.

  • Kibana is an open source data visualization and management tool for Elasticsearch that provides real-time charts (e.g. histogram, line chart, pie chart, map, etc.).

  • Beats: Lightweight data acquisition tool, a collection of multiple single-purpose data collectors. Early ELK components used Logstash to collect logs. However, Logstach has high resource consumption and Beats has a lighter performance footprint. Although Beats and Logstash have overlapping functions, they can still forward data to Logstash to gain stronger data conversion capability.

In addition, ELK Stack components often need to work with Kafka or Redis message queues to reduce the risk of data loss. In the absence of message queues, data loss occurs if a service-side component fails. In the case of message queues, data is stored in case of faults, reducing the risk of data loss.

3. Use of Kibana

For students on the client side, they should be more concerned about Kibana on the analysis side. In this section, we will focus on summarizing the use of Kibana.

3.1 Data Search

Data search is the most basic and key feature of Kibana. You can explore your data interactively on the Discover page.

Familiarize yourself with a few areas of the page:

  • Search criteria: You can use a simple text query, Lucene syntax, or jSON-based Elasticsearch query DSL;
  • Time filtering: Select the time span for searching data. Note that the longer the time span, the more time it will take to search data.
  • Item filtering: Select the item to search for data. Generally, the log of each item is collected independently and collected through Logstash to Elasticsearch.
  • Field filtering: Filters search results to display only data containing specific field values.

In the data display area, you can see all the filtered data. Expand a specific data field, where the MSG field value is the payload reported in your project.

3.2 visual

On the Visualize page you can create visual controls, and on the Dashboards page you can display visuals that you have created together.

The resources

  • Elastic website
  • What is Elasticsearch? — Official documents
  • Kibana User Manual – official documentation
  • Introduction to ELK components and principles by Beiguo-hao

Creation is not easy, your “three lian” is chouchou’s biggest motivation, we will see you next time!