The preface

Elasticsearch is a real-time distributed search analysis engine. Teambition uses Elastisearch as a search engine to provide search service for users. When we decide to store certain data, we need to use PUT/Teambition to create index. When creating index, we need to determine the data structure completely. At the same time index Settings and many fixed configurations will not be changed. When data structures need to be changed, the Elastic team has a number of tools available to help developers rebuild indexes.

Decide if you need to rebuild?

Rebuilding is very painful, if there is no good foundation, the service may be interrupted, when the amount of data is very large, the reconstruction recovery time may be very long, and even errors in the reconstruction process, etc. So avoid rebuilding indexes unless you have to. The reason why Teambition rebuilds the index is slight, because it is not important.

Use the right tools

Kibana is the perfect partner for Elasticsearch. Starting with ES 5.0, Kibana is a powerful replacement for almost all the old ES 1.x and ES 2.x plugins. The point is, it’s free!

A prerequisite for

The running service must use an alias to access the index for the simple reason that the search service will eventually use the rebuilt index and the original index will be deleted. If your service is using index names directly, create aliases to update the service before rebuilding.

POST /_aliases {"actions": [{"add": {"index": "teambition", // original index" alias": "teambition_latest" // alias of service}}]}Copy the code

Prerequisite 2

Check the Disk Usage of Elasticsearch. If not, please make sure you have enough space.

Create a new index

Create a new index just as you would create a normal index. It is worth mentioning that when a large amount of data is generated, the refresh interval should be set to improve the reconstruction speed. Refresh_intervals = -1, number_of_replicas = 0. Practice tells me it’s about 100 to 400 percent improvement.

PUT /teambition_20180328 { "settings": {... }, "mapping": {... }}Copy the code

Record the offset of synchronized data

Elasticsearch uses Kafka to import MongoDB data to Elasticsearch. It is true that Elasticsearch uses Kafka to import Elasticsearch data. Regardless of which method of synchronizing data is used, the offset of the synchronized data needs to be recorded. Rebuilding an index can be time-consuming and cannot update new data while the synchronization process is still updating data to the old index. This ambition uses the kafka-admin API to log offset.

Start rebuilding the index

Copy data into the new index using the ReIndex API provided by the Elasticsearch team. Here are a few options:

  1. When just changing the mapping data structure, you can just use the REindex API. For example, delete fields and update the word segmentation method of fields.
  2. This parameter is used when you need to write a new field that is computed from the old fieldscriptParameters. For example, calculate the sum of a field and a piece of data. Script has a lot of bugs. When a script goes wrong, reindex runs for a long time and fails. Even if data is recovered, reindex needs to be run again.
  3. When it has very complicated logic, well, write your own program.

If the reindex interface is called, the interface will return after the reindex is complete, and the timeout is only 30 seconds. If the reindex time is too long, it is recommended to add wait_FOR_completion =false, so that the reindex will directly return taskId

POST _reindex? wait_for_completion=false { "source": { "index": "teambition" }, "dest": { "index": "teambition_20180328" }, "script": {... }}Copy the code

Rebuilding the index

Rebuilding the index takes a long time, so take a coffee break (hit a ball, sleep, take a trip).

When refresh_intervals and number_of_replicas are not set, the reindex rate ranges from 500 to 1000 doc/ SEC, and may be lower if script is included. After setting, it can be 4000~8000 doc/ SEC. Teambition 70M Documents takes about 4 hours.

You can use GET _tasks/{taskID} to see the rebuilding process, including time taken, number of remaining docs, and so on.

If errors are found, you can abandon the task and start again using the PUT _tasks/{taskID}/cancel interface.

Recovering Synchronized Data

After rebuilding the index, don’t forget to set number_of_replicas and refresh_intervals to their original values in setting. Start a new index synchronization process (starting from offset)

Creating a new Alias

The following statement is used to bind the new index and unbind the old index:

POST _aliases
{
  "actions": [{"add": {
    "index": "teambition_20180328",
    "alias": "teambition_latest"
  }}, {"remove": {
    "index": "teambition",
    "alias": "teambition_latest"
  }}]
}
Copy the code

Delete the index

Delete the old index to release disk space. Stop the original synchronization process.

DELETE teambition
Copy the code

conclusion

Changing an index can be a really time-consuming and taxing task, especially if an error occurs and the whole person suffers. So try to figure out if you can meet the requirements when you create an index, which is almost impossible because of the evil product manager.

There is also a very important content that has not been described in detail is the synchronization process. The synchronization process mentioned above is the program that synchronizes MongoDB data to ES. This program also needs the ability to suspend synchronization, repeat synchronization and so on.