This is the 13th day of my participation in the November Gwen Challenge. Check out the event details: The last Gwen Challenge 2021
Elasticsearch is a popular distributed open source search and data analysis engine, featuring high performance, easy expansion, and strong fault tolerance. It enhances Apache Lucene’s search capabilities and takes control of indexing and querying massive amounts of data to a new level.
In this article, we will put together some best practices for ElasticSearch development in your daily life, based on the characteristics and experience of ElasticSearch development.
Cluster deployment optimization suggestions
Choose the right hardware configuration: use SSDS whenever possible
The biggest bottleneck of Elasticsearch is often disk read/write performance, especially random read performance. Using SSDS (PCI-E SSD cards or SATA SSDS) is 5 to 10 times faster than using mechanical hard disks (SATA disks or SAS disks), but the write performance is not significantly improved.
For scenarios that require high performance in document retrieval, you are advised to use SSDS as storage devices and configure memory and hard disks at a ratio of 1:10.
If the requirements for concurrent query of log analysis are low, you can use mechanical disks as the storage and configure the memory and hard disks at the ratio of 1:50.
It is recommended that the data stored on a node be less than 2TB and less than 5TB to avoid slow query and system instability.
Configure the JVM with half the machine’s memory, but no more than 32GB is recommended
Change the conf/jvm.options configuration and set -xms and -xmx to the same value. It is recommended to set the value to about half of the machine memory and reserve the remaining half for the operating system cache.
It is recommended that the JVM memory be less than 2 GB. Otherwise, ES may fail to start or overflow due to insufficient memory. It is recommended that the JVM memory be less than 32 GB, otherwise, THE JVM will disable the memory object pointer compression technology, resulting in memory waste.
-xms30g -XMx30g is recommended if the memory size is greater than 64 GB.
When the JVM heap memory is large, memory garbage collection pauses for a long time. Therefore, you are advised to configure the ZGC or G1 garbage collection algorithm.
Large clusters are configured with proprietary master nodes to avoid brain splitting
The primary Elasticsearch node manages cluster meta information, adds and deletes indexes, adds and deletes nodes, and broadcasts the latest cluster status to each node.
In a large cluster, you are advised to configure a dedicated primary node to manage the cluster, not store data, and not bear data read and write pressure.
The configuration is as follows:
# elasticSearch.yml # elasticSearch.yml
node.master: true
node.data: false
node.ingest: false
# data node configuration (conf/ elasticSearch.yml)
node.master: false
node.data: true
node.ingest: true
Copy the codeBy default, each node of Elasticsearch is both a candidate primary node and a data node. The minimum_master_nodes parameter is recommended to be set to more than half the number of candidate primary nodes. This parameter tells Elasticsearch not to vote for master nodes when there are not enough candidate master nodes. Wait until there are enough master nodes to vote.
For example, for a 3-node cluster, the minimum number of primary nodes is changed from the default of 1 to 2.
# elasticSearch.yml/elasticSearch.yml
discovery.zen.minimum_master_nodes: 2
Copy the codeTuning the Linux operating system
Disable swap partitions to prevent memory replacement from degrading performance.
The configuration is as follows:
Comment out the line containing swap in /etc/fstab
sed -i '/swap/s/^/#/' /etc/fstab
swapoff -a
The maximum number of files that can be opened by a single user can be set to 65536 or more as recommended
echo "* - nofile 65536" >> /etc/security/limits.conf
# Increase the number of single-user threads
echo "* - nproc 131072" >> /etc/security/limits.conf
# Maximum number of map memory regions that can be used by a single process
echo "vm.max_map_count = 655360" >> /etc/sysctl.conf
The parameter changes take effect immediately
sysctl -p
Copy the codeIndex performance tuning recommendations
Strictly set the storage and indexing switch, do not store all data or all indexes
Elasticsearch supports persistence and indexing, but the main scenario of Elasticsearch is full text search, which is expensive. It is recommended to index only the fields that need to be indexed according to the business, which not only saves the index cost, but also makes the operation faster and more efficient!
Do not use multi-type indexes to avoid sparse indexes
Elasticsearch can read and write data of different types on the same index. Data is isolated by type level. The version of Elasticsearch has the same id as that of Elasticsearch. The version of Elasticsearch has the same id as that of Elasticsearch. The version of Elasticsearch has the same ID as that of Elasticsearch.
The number of fragments is set based on the amount of service data and cluster size
The default number of fragments for Elasticsearch is 5. You can set the number of fragments for different service indexes based on actual conditions. If the amount of index data is very small and the future increment is limited, you can reduce the number of fragments. Synchronization management and fault recovery of a cluster have certain risks.
If the index data volume is very large, you need to set it based on the data volume and cluster scale. The recommended value for a fragment is 1 billion, and the recommended maximum data volume is 4 billion. The number of fragments is also affected by the cluster size. If the cluster has too few nodes, you are advised not to set too many fragments. As a result, multiple fragments of the same index reside on the same Elasticsearch node, causing slow recovery. Typically, there are 20-25 shards per 1GB of heap memory.
The number of copies is not greater than 3
The default number of copies of Elasticsearch is 1, that is, there are two copies of data from one primary and one replica. To improve data security and avoid data loss, set the number of copies to 2, that is, there are three copies of data from one primary and two replicas. However, there is no need to set a large number of copies. Replication of Elasticsearch can affect cluster performance, especially for clusters where refresh_interval is set to a small value.
Disable automatic index creation
Elasticsearch indexes are automatically created by default and must be disabled.
Set action. Auto_create_index to false because automatic index creation cannot avoid metadata governance problems caused by misoperations.
After indexes are created, automatic type mapping is disabled
The index of Elasticsearch can be automatically mapped by default and must be disabled. If the type does not match the actual type, set index.mapper.dynamic to false.
String mapping setting with keyword preferred
The default Elasticsearch index supports keyword and text. Text is used for keyword segmentation. Keyword does not support keyword segmentation.
Mapping Settings, use nested types sparingly
The default Elasticsearch index supports mapping nesting. In normal cases, you are advised not to use the nested type. If the nested type is used, the depth is not greater than 2.
Join design, careful use of nested and parent-child
Elasticsearch indexes do not support Joins by default. You are advised to perform this operation by yourself. If filtering is required in both parent and child documents, or the field of both parent and child documents needs to be returned, the nested function is recommended. In other scenarios, parent-child function is recommended.
In general, nested queries are 5-10 times faster than parent-child queries, but parent-child updates documents more conveniently.
Index the number of fields should not be too high, and the field type should take up as little memory as possible
By default, the index of Elasticsearch supports a maximum of 1000 fields. The maximum number of fields of Elasticsearch increases the cluster cache consumption. You are advised to limit the number of fields to 50.
You are advised to use other storage engines, such as mysql and hbase, instead of storing non-indexed and aggregated fields in Elasticsearch.
Disable _all and _source
By default, _all and _source are enabled for Elasticsearch indexes. In addition to normal storage and indexes, original write records and other metadata information are stored. Filed records of Elasticsearch can be replaced by normal storage and index records. You can also exclude or include a query or search.
Set reasonable delay fragment balancing time
The Elasticsearch cluster is highly dependent on network stability. By default, the fragmentation delay is 1 MB. If a copy is restored to the cluster within 1 minute, it will not be fragmented again. If the network is disconnected for a long time during O&M, you are advised to change this parameter to prevent fragments from being created and moved back and forth, and to provide buffer time for o&M and node recovery.
Query performance tuning suggestions
Provide explicit timeouts in queries
Almost all Elasticsearch apis allow users to specify timeouts.
Find and get rid of time-consuming operations, save resources, and build stable services that will help both your application and your Elasticsearch cluster.
Use Scan and Scroll carefully
Elasticsearch supports a maximum of 10000 records by default. The value of from+size cannot be greater than 10000.
In deep paging scenarios, Scan and Scroll can be used, but use Scan and Scroll with caution. For large indexes, frequent Scan and Scroll will greatly affect the stability of the Elasticsearch cluster, and may cause JVM suspension or even crash.
- For real-time scenarios: Disable this function
- Perform this operation with caution in offline scenarios
Use the size:0 and includes/ Excludes qualifying field to return
Elasticsearch has a significant performance difference before and after adding the size:0 clause.
Return required fields unless required by the business, otherwise, fields that do not need to be returned are controlled via Includes and Excludes.
Use rest apis to invoke apis, and use transport mode with caution
The native API of Elasticsearch supports the Transport mode to access a cluster. This mode communicates with the cluster in netty mode. The continuous write system can use the Transport mode. Normal clients can use restful interfaces based on DSL to query indexes. Httpclient access efficiency is higher.
Avoid prefix fuzzy matching
Elasticsearch is supported by *? Regular expression is used to perform fuzzy matching. If fuzzy matching, especially prefix fuzzy matching, is performed on an index with a large amount of data, it takes a long time and may even cause memory overflow. Try to avoid doing this in a production environment with high concurrent query requests.
When a customer requests “License plate number :A8848” for fuzzy query of license plate number, the load of the whole cluster is high. By preprocessing data, add redundant field “license plate number.keyword”, and advance all license plate numbers according to 1 yuan, 2 yuan, 3 yuan… 7 yuan is stored in this field. Example: Shanghai,A,8,4, Shanghai A,A8,88,84,48, Shanghai A8… Shanghai A88488. The original performance problem can be solved by querying “license plate number.keyword :A8848”.
Avoid deep page turning
By default, Elasticsearch allows you to view only the top 10000 results. If you want to view the bottom 10000 records, the response time is very long. Using search_after is more lightweight. If only 10 results are returned each time, then each shard only needs to return 10 results after search_after. The total amount of data returned depends only on the number of shards and the number of shards needed this time. It has nothing to do with the number of historical reads.
Configure the aggregation node to be queried
The query aggregation node can send particle query requests to other nodes, collect and merge results, and respond to the client issuing the query. By configuring higher CPU and memory specifications for query aggregation nodes, you can speed up query operation and improve cache hit ratio.
A customer uses 25 8-core CPU32 GB memory nodes in the Elasticsearch cluster, and the query QPS is about 4000. Add six 16-core CPU32G memory nodes as query aggregation nodes, observe server CPU and JVM heap memory usage, adjust cache, fragment, and copy parameters, and query QPS reaches 12000.
The configuration is as follows
# query aggregate node configuration (conf/elasticsearch.yml)
node.master: false
node.data: false
node.ingest: false
Copy the codeConfigure the appropriate word dividers
Elasticsearch comes with a lot of built-in tokenizers, including Standard, CJK, nGram, etc. You can also install your own/open source tokenizers. Select an appropriate tokenizer based on the business scenario, and avoid using the default standard tokenizer.
Commonly used participles are as follows:
- Standard: the default word segmentation, English according to the space segmentation, Chinese according to a single Character segmentation.
- CJK: Segmentation of Chinese and Japanese Korean words according to binary index can ensure recall rate.
- NGram: English can be split into letters and used with ES phrase search (match_phrase).
- IK: Popular Chinese word segmentation, can be according to Chinese semantic segmentation, you can customize the dictionary.
- Pinyin: Allows users to type in pinyin and find relevant keywords.
- Aliws: Alibaba develops its own word segmentation, supports a variety of models and word segmentation algorithms, has a rich thesaurus and accurate word segmentation results, which is suitable for e-commerce and other scenes requiring high accuracy.
Other Optimization Suggestions
Different service scenarios are isolated at the cluster level
Do not store data of different service scenarios in one Elasticsearch cluster. You are advised to deploy real-time and offline services separately.
Because the read/write parameters of the Elasticsearch cluster are shared by all indexes, peak times can affect each other.
You are advised to isolate different clusters based on different read/write frequencies and real-time requirements, and set cluster parameters based on different service scenarios.
Do not store large text or binary data
Elasticsearch supports large text and binary data. However, large text and binary data occupy the cluster’s memory and disk space, affecting the cluster’s DISK I/O and cache efficiency.
You are advised to store the large text or binary data on another file system or storage system and use Elasticsearch to store the large text or binary data.
Naming conventions
The index name of Elasticsearch is disabled. The initial index name, used by default within Elasticsearch or stack-related components. The index name to start with.
Index names that start with _ are prohibited. By default, field names used internally by Elasticsearch start with _.
The index name is case insensitive and consists of letters and digits, starting with a letter.
The length of index name and field name cannot exceed 32 characters.
Reference documentation
- Best practice guide for Elasticsearch developers
- 21 recommendations for Tuning Index and query performance for Elasticsearch