One, foreword

In everyday Elasticsearch, there are many application scenarios such as storing system logs, behavior data, etc. These scenarios are characterized by a very large amount of data and the number of indexes will continue to grow over time. However, these scenarios are basically only recently for a period of time the data have the use value, or will be frequently used (hot), and historical data is almost no effect or rarely use cold (data), at this time requires a certain strategy for index of maintenance and management and even delete clean, otherwise with data volume more and more waste in addition to the disk and memory space, [Fixed] It also severely affects the performance of ElasticSearch

 

Index Lifecycle Management, a new feature released after Elastic Stack 6.6, supports full Lifecycle Management of indexes, and provides a UI for configurable policies on Kibana. This article focuses on how to configure and use Elasticsearch index lifecycle management.

 

Second, life cycle

2.1. Stage introduction

The index life cycle is divided into four phases: hot, warm, cold, and delete, in which hot is mainly responsible for the rollover of the index.

Rollover: A new index created by rolling updates will be added to the index alias and will be specified as the write index.

PS: Of the four stages, only the HOT stage is required

The index enters the life cycle phase according to the time parameter MIN_AGE. If not set, the default is 0ms. Min_age is usually calculated from the time the index was created, or if the index is set to scroll, then min_age is calculated from index scroll. Note that the operation for the current stage must complete before checking the min_age parameter and moving on to the next stage.

 

2.2. Stage actions

Phase/action Priority setting Cancel the following Scroll to the index Fragmentation distribution read-only Forced segment merge Shrinkage index Freezing index delete
hot Square root Square root Square root x x x x x x
warm Square root Square root x Square root Square root Square root Square root x x
cold Square root Square root x Square root x x x Square root x
delete x x x x x x x x Square root

 

Example 2.3.

The index syslog-2020.10.01 is taken as an example. After 1 day of index creation, it becomes Warm stage, after 30 days, it becomes Cold stage, and after 30 days, it is deleted

The date of action phase
2020-10-01 Create indexesSyslog - 2020.10.01, handles read and write requests Hot stage
2020-10-02 Syslog - 2020.10.01Is changed to read only A warm stage
2020-11-01 Syslog - 2020.10.01Is read-only and migrated to cold node storage Cold phase
2020-12-01 Remove the indexSyslog - 2020.10.01 The delete phase

 

3. Simulation process

3.1. Create index life cycle policy

Suppose the Policy is set as follows:

  • Indexing is done every 10 documentsRollover
  • RolloverAfter 5 seconds toWarmphase
  • RolloverSwitch to after 20 secondsColdphase
  • RolloverDelete after 40 seconds
curl -XPUT "http://$IP:9200/_ilm/policy/my_ilm_policy" \
    -H 'Content-Type: application/json' \
    -u elastic:changeme \
    -d '{
      "policy": {
        "phases": {
          "hot": {
            "actions": {
              "rollover": {
                "max_docs": "10"
              }
            }
          },
          "warm": {
            "min_age": "5s",
            "actions": {
              "allocate": {
                "include": {
                  "box_type": "warm"
                }
              }
            }
          },
          "cold": {
            "min_age": "20s",
            "actions": {
              "allocate": {
                "include": {
                  "box_type": "cold"
                }
              }
            }
          },
          "delete": {
            "min_age": "40s",
            "actions": {
              "delete": {}
            }
          }
        }
      }
    }'

The IP, user name, and password are changed as the case may be

 

3.2. Association strategy

There are two types of association strategy, namely using index template association and index direct association

3.2.1. Index Template Association

Index templates to create the required indexes and associate ILM policies

curl -XPUT "http://$IP:9200/_template/my_test_template" \
    -H 'Content-Type: application/json' \
    -u elastic:changeme \
    -d '{
        "index_patterns": ["my-test-*"], 
        "settings": {
            "number_of_shards": 1,
            "number_of_replicas": 0,
            "index.lifecycle.name": "my_ilm_policy", 
            "index.lifecycle.rollover_alias": "my-test",
            "index.routing.allocation.include.box_type": "hot"
        }
    }'

The IP, user name, and password are changed as the case may be

Index.Lifecycle. Name: Indicates the ILM Policy Index.Lifecycle. Rollover_Alias to which this index applies: Indicate when Rollover using the alias of the index. The routing. The allocation. The include. Box_type: indicate the index distribution in the hot new node

 

3.2.2. Index Direct Association

Associate policies separately for existing indexes

curl -XPUT "http://$IP:9200/my-test-*/_settings" \
    -H 'Content-Type: application/json' \
    -u elastic:changeme \
    -d '{
      "index": {
        "lifecycle": {
          "name": "my_ilm_policy"
        }
      }
    }'

The IP, user name, and password are changed as the case may be

 

3.3. Look at the stage of the index

http://$IP:9200/my-test-*/_ilm/explain

 

3.4. Update Policy

  1. If no index applies the policy, then we can update the policy directly.
  2. If an index applies this policy, the phase currently executing will not be modified synchronously. When the current phase ends, the next phase of the new version of the policy will be moved to.
  3. If a policy is changed, the phase currently in execution will not change, and the new policy will manage the next lifecycle after the current phase is over.

 

3.5. Kibana graphical operation

Most of the above steps can be done inKibanaIn the way of graphical interface to operate

 

Pay attention to: If you use a graphical interface to create the policy, the delete stage is missing
actionsThe content cannot be deleted

 

Modify the polling interval (optional)

The ILM Service polls the Policy execution in the background, with a default interval of 10 minutes, which can be changed to 1 second to test for faster results.

curl -XPUT "http://$IP:9200/_cluster/settings" \
    -H 'Content-Type: application/json' \
    -u elastic:changeme \
    -d '{
        "persistent": {
          "indices.lifecycle.poll_interval":"1s"
        }
    }'

The IP, user name, and password are changed as the case may be

 

Start and stop index lifecycle management

ILM is enabled by default

All indexes managed by ILM will continue to execute their policies. Sometimes some indexes, or even all indexes in the cluster, may not be needed. For example, when a cluster topology change is required, there may be planned maintenance Windows, which may affect the ILM operations that are running. Thus, ILM has two ways to disable operations.

When ILM is stopped, the snapshot lifecycle management operation also stops, which means that no planned snapshots are created (currently in-progress snapshots are not affected).

In general, ILM will run by default. To view the current health Status of ILM, use the Get Status API to view the current Status of ILM.

GET  _ilm/status

If the request does not encounter an error, you will receive the following results:

{
    "operation_mode": "RUNNING"
}

 

Operation mode of ILM:

Phase/action Priority setting
The running Normal operation, all policies are executing normally
stop ILM has received a stop request, but is still processing some policies
Has stopped This indicates a state where no policies are being executed

 

5.1. Stop the ILM

You can pause the ILM service so that no further steps are performed using the Stop API.

POST  _ilm/stop

 

When it stops, all other policy measures will stop. This will be reflected in the status API

{
    "operation_mode": "STOPPING"
}

 

The ILM service then asynchronously runs all policies to a place where they can be safely stopped. After ILM confirms that it is safe, it moves to the Stopped mode

{
    "operation_mode": "STOPPED"
}

 

5.2. Start the ILM

To Start ILM and continue executing the policy, use the Start API.

POST  _ilm/start

 

The Start API will send a request to the ILM service to begin normal operations immediately.

{
  "operation_mode": "RUNNING"
}

 

VI. API List

The following APIs can be used to manage indexing policies. See the official documentation management index life cycle.

  • Policy Management API

    • Create a lifecycle policy
    • Get the lifecycle policy
    • Delete lifecycle policies

  • Index Management API

    • Move the index to the step
    • Retry index policies
    • Remove the policy from the index

  • Operations Management API

    • Get the ILM operation mode
    • Start the ILM
    • Stop the ILM
    • Explain the API

 

Scan code concern has surprise!