On January 29, Elastic Stack released version 6.6, which brings new features like:

  • Index Lifecycle Management
  • Frozen Index
  • Geoshape based on Bkd Tree
  • SQL adds support for Date histograms
  • .

Of these features, Index Lifecycle Management(Index Lifecycle Management, hereafter ILM) is the most popular in the community. Today we have a quick look at this feature from the following aspects:

  • Why are indexes alive? What is the index lifecycle?
  • How does ILM divide the index life cycle?
  • How does ILM manage the index life cycle?
  • In actual combat

Index Lifecycle Indicates the Index Lifecycle

Let’s start with the first question:

Why are indexes alive?

An Index is a logical concept for organizing data in Elasticsearch. It is a combination of documents with the same or similar fields. It is composed of many shards, such as book and people, which can be used as index names. It can be simply likened to the table of a relational database.

Life is life and death; The life and death of indexes is creation and deletion.

Create index (s) and delete index (s) from Elasticsearch.

This is where Elasticsearch comes in.

In service search scenarios, users can store service data, such as product data, order data, and user data, in Elasticsearch to achieve quick full-text search. Data like this is basically cumulative and will not be deleted. Generally delete the words, or the service upgrade, or the service is off. In this scenario, there is basically only life, not death, so there is no such thing as management.

In the log service scenario, users can store all kinds of logs, such as system logs, firewall logs, middleware logs, database logs, Web server logs, and application logs to Elasticsearch in real time for query and analysis. This type of data has a temporal dimension, that is, temporal data. Because the amount of log data is too large, users do not store all Elasticsearch data. In Elasticsearch, users store hot data, for example, the data generated in the latest 7 days or 30 days, but the data generated in the last 7 days or 30 days is deleted. To facilitate operation, users generally create indexes based on the date. For example, the log index name of nginx can be nginx_log-2018.11.12 or nginx_log-2018.11.13. To query or delete logs of a specific day, you only need to perform operations based on the index of the corresponding date. In this scenario, then, the life and death of a large number of indexes plays out every day.

An index from birth to death process, that is, a life cycle. Examples are as follows:

Health: create index nginx_log-2019.02.05 on February 5, 2019 and start processing read/write requests for log data Delete index nginx_log-2018.02.05 on March 5, 2019. Other indexes, such as nginx_log-2019.02.06, will also go through the same life cycle.

How does ILM divide the index life cycle?

We now know what a life cycle is, and the simplest life cycle requires only two phases: life and death. In practice, however, the life cycle has multiple phases. Let’s look at how ILM divides the life cycle.

ILM divides the index life cycle into four phases:

  • Hot stage
  • A Warm stage
  • Cold phase
  • The Delete phase

If we take a human life cycle as an analogy, it looks something like this:

Hot stage

Index data is being actively updated and queried

Hot stage can be likened to the stage from human infant to youth. In this stage, it will continuously input and output knowledge (data reading and writing), and continuously grow tall and grow up (data volume increases) to become a useful youth.

During this phase, a large number of data reads and writes are required. Therefore, nodes with high configuration are required. You are advised to set the memory to disk ratio of a node to 32, for example, 64GB memory combined with 2TB SSDS.

A Warm stage

Index data is no longer updated, but is still queried

The Warm stage can be likened to the stage from youth to middle age of human beings. In this stage, it basically does not input knowledge (data writing), but mainly outputs knowledge (data reading) to contribute value to the society.

As this phase is mainly responsible for data reading, nodes with medium configuration can meet the requirements. You can increase the memory to disk ratio of nodes to 64 96, for example, 64GB memory with 46 TB HDD disks.

Cold phase

The index has not been updated and is rarely queried. But the information of the index data needs to be searched, and if it is searched, it is slow

The Cold stage can be compared to the middle to old age stage in humans, in which it retires, outputs its knowledge (data reading) when society needs it, and mostly sits still.

Since only a small amount of data is read at this stage, nodes with low configuration can meet the requirements. The node memory to disk ratio can be further improved to 96 or above, such as 128, that is, 64GB memory with 8 TB HDD.

The Delete phase

Indexes are no longer required to be safely deleted

The Delete stage can be likened to the stage when human beings die of old age. After glowing and heating up, they die quietly, Rest in Peace~

ILM has a very detailed breakdown of the index lifecycle, but it does not mandate the four phases, and users can compose their own lifecycle according to their own needs.

How does ILM manage the index life cycle?

The so-called life cycle management is to control four life phase transitions: when Hot to Warm, when Warm to Cold, when Delete, etc.

Timing is necessary for phase transformation, and time is the best dimension for temporal data, and ILM is also measured by time. For example, in the diagram below, the transition is Hot by default, Warm by 3 days after index creation, Cold by 7 days, and deleted by 30 days. The relevant field for this setting is min_age, as explained later.

ILM refers to different lifecycle management policies as policies, which consist of different actions in different phases.

Actions are a series of actions that operate indexes, such as Rollover, Shrink, and Force Merge. The actions available at different stages vary as follows:

  • Hot Phase
    • Rollover

    A rolling index operation is used to control the size of a single index by creating new indexes for data reads and writes when the index size or the number of documents reaches a certain value. One thing to note here is that if Rollover is enabled, The Times of all phases are no longer based on the index creation time, but on the index Rollover time.

  • Warm Phase
    • Allocate Sets the number of copies, and modifies the sharding allocation rules (for example, migrate the sharding to a moderately configured node)
    • Read-onlly Indicates that the current index is read-only
    • Force Merge Merge segment operations
    • Shrink shrinks the number of shard fragments
  • Cold Phase
    • The same goes for the Allocate
  • Delete Phase
    • Delete Delete

It’s pretty simple from above, but Kibana also provides a UI interface to set up these policies, as follows:Is the setup of ILM clear from the picture above?

Of course, ILM has its own API. For example, the corresponding API request in the picture above is as follows:

PUT /_ilm/policy/test_ilm2
{
    "policy": {
        "phases": {
            "hot": {
                "actions": {
                    "rollover": {
                        "max_age": "30d"."max_size": "50gb"}}},"warm": {
                "min_age": "3d"."actions": {
                    "allocate": {
                        "require": {
                            "box_type": "warm"
                        },
                        "number_of_replicas": 0
                    },
                    "forcemerge": {
                        "max_num_segments": 1
                    },
                    "shrink": {
                        "number_of_shards": 1}}},"cold": {
                "min_age": "7d"."actions": {
                    "allocate": {
                        "require": {
                            "box_type": "cold"}}}},"delete": {
                "min_age": "30d"."actions": {
                    "delete": {}}}}}}Copy the code

I’m not going to do that here, but if you’re interested, you can check the official manual.

Now that you have a Policy, how do you apply it to an Index?

The method is to set the following index configuration:

Name Specifies the Policy name, such as test_ilm2 index.lifecycle. Rollover_alias. The Index configuration can be modified directly (PUT Index_name /_settings) or through Index Template.

We will not expand here, we refer to the following actual combat will understand.

In actual combat

The target

Now you need to collect nginx logs. Only the logs generated in the latest 30 days need to be saved. Ensure that the query performance of the logs generated in the latest 7 days is good and the search response time is less than 100ms.

In order to quickly see the effect, the following actual operation will be replaced with 30 days, 7 days, 40 seconds, 20 seconds.

ES Cluster Architecture

Here we briefly introduce the composition of ES cluster used in this actual combat. The ES cluster consists of three nodes, each of which has a property named box_type, as follows:

GET _cat/nodeattrs? S =attr es01_hot 172.24.0.5 172.24.0.5 box_type Hot ES02_warm 172.24.0.4 172.24.0.4 box_type warm ES03_cold 172.24.0.3 172.24.0.3 box_type coldCopy the code

It can be seen from the above that we have 1 hot node, 1 warm node and 1 cold node, which are respectively used for the corresponding ILM stage. That is, indexes in the hot stage are all located on hot, and indexes in the warm stage are all located on warm. Indexes for the Cold phase are all located on Cold.

Create the ILM Policy

As required, our Policy is set as follows:

  • The index name is prefixed with nginx_logs and is Rollover every 10 documents
  • 5 seconds after Rollover, the phase changes to Warm
  • 20 seconds after the Rollover, the Cold phase changes
  • Delete 40 seconds after Rollover

The API request is as follows:

PUT /_ilm/policy/nginx_ilm_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_docs": "10"}}},"warm": {
        "min_age": "5s"."actions": {
          "allocate": {
            "include": {
              "box_type": "warm"}}}},"cold": {
        "min_age": "20s"."actions": {
          "allocate": {
            "include": {
              "box_type": "cold"}}}},"delete": {
        "min_age": "40s"."actions": {
          "delete": {}}}}}}Copy the code

Create Index Template

We create the required index based on the index template, as shown below:

PUT /_template/nginx_ilm_template
{
  "index_patterns": ["nginx_logs-*"]."settings": {
    "number_of_shards": 1."number_of_replicas": 0."index.lifecycle.name": "nginx_ilm_policy"."index.lifecycle.rollover_alias": "nginx_logs"."index.routing.allocation.include.box_type": "hot"}}Copy the code

The configuration is explained as follows:

  • Index.lifecycle. Name Specifies the ILM Policy applied to the index
  • Index.lifecycle. Rollover_alias Specifies the alias to use for Rollover
  • Index. The routing. The allocation. The include. Box_type indicate the index distribution in the hot new node

Create the initial Index Index

The first index of ILM needs to be created manually, and the Rollover startup must end with a numeric type, such as nginx_logs-000001. The API for index creation is as follows:

PUT nginx_logs- 000001.
{
  "aliases": {
    "nginx_logs": {
      "is_write_index":true}}}Copy the code

The index distribution is as follows:

Example Change the ILM Polling Interval

The ILM Service polls the Policy in the background. The default interval is 10 minutes, but we changed it to 1 second for faster results.

PUT _cluster/settings
{
  "persistent": {
    "indices.lifecycle.poll_interval":"1s"}}Copy the code

Let’s start

Everything is ready, let’s get started!

First perform the new document operation below 10 times.

POST nginx_logs/_doc
{
  "name":"abbc"
}
Copy the code

After Rollover executes, a new index is created, as shown below:

After 5 seconds, nginx_LOGs-000001 enters the Warm phase

After 15 seconds, nginx_LOGs-00001 goes to the Cold phase

After 25 seconds, nginx_logs-00001 is deleted

At this point, a complete ILM Policy execution process is complete, and subsequent nginx_LOGs-000002 flows follow this setting.

conclusion

ILM is a comprehensive implementation of Elasticsearch best practices in logging scenarios by the Elastic team, making it much easier to use Elasticsearch. Understanding the core concepts of ILM means understanding the best practices of Elasticsearch. I hope this article will help you get started with ILM.

conclusion

Welcome to wechat public account “Code zonE”, focusing on sharing Java, cloud computing related content, including SpringBoot, SpringCloud, microservices, Docker, Kubernetes, Python and other related technology dry goods, looking forward to meeting you!