Writing in the front

Arrange some interview questions with Internet companies, these interview questions are often asked, also as a Java engineer need to master the knowledge, after all, the combination of theory and practice, is king, fragmentation, eat some knowledge every day, happy every day, if you have any help to you, remember the point attention and point oh.

Related articles

MyBatis (MyBatis) (ZooKeeper) (Dubbo) (Elasticsearch) (Redis) (MySQL) (Dubbo) Java concurrent programming interview questions (1)

How many shards are there, how many are there, and how many tuning methods are there

Interviewer: I want to know the application scenario and scale of ES that the applicant contacted with before, and whether he has done index design, planning and tuning on a relatively large scale.

Answer: truthfully combined with their own practice scenarios can be answered. For example, the ES cluster architecture has 13 nodes, and the index is 20+ index according to channel. The index is increased by 20+ index according to date, and the index is 10 fragments, and the index is increased by 100 million + data every day. The index size of each channel is controlled within 150GB.

Indexing-only tuning means:

  • Design phase tuning

    • According to the incremental requirements of the service, the index is created based on the date template, and the roll over API is used to roll the index.
    • Use aliases for index management;
    • Do the index at the same time every morningforce_mergeOperation to free up space;
    • Hot data is stored on SSD to improve retrieval efficiency. Cold data occurs regularlyshrinkOperation to reduce storage;
    • takecuratorIndex lifecycle management;
    • Only for the fields that need word segmentation, the reasonable setting of word segmentation;
    • In the Mapping stage, the attributes of each field are fully combined, such as whether to be retrieved and stored.

  • Write the tuning

    • The number of copies before writing is set to 0.
    • Close before writerefresh_intervalIf the value is set to -1, the refresh mechanism is disabled.
    • Write process: takebulkBatch write;
    • Restore the number of copies and refresh interval after writing;
    • Use automatically generated ids whenever possible.

  • Query tuning

    • disablewildcard;
    • Disable the batchterms(hundreds of scenes);
    • Make full use of the inverted index mechanism to keyword type as much as possible;
    • When the amount of data is large, the index can be determined based on time before retrieval;
    • Set a proper routing mechanism.

  • Other tuning deployment tuning, business tuning, etc.

As part of the above, the interviewer will have a general assessment of your previous practice or operations experience.

What is the inverted index of ElasticSearch

Interviewer: I want to know your understanding of basic concepts.

Answer: A simple explanation is ok.

Traditionally, our retrieval is to find the position of corresponding keywords through the article one by one. The inverted index, through the word segmentation strategy, forms the mapping relation table between words and articles, and this dictionary + mapping table is the inverted index. With the inverted index, it can realize O(1) time complexity of the efficient retrieval of articles, greatly improving the retrieval efficiency.

  • The academic solution: an inverted index, as opposed to what words are included in an article, starts with a word and records where that word appears in the text. It consists of two parts — a dictionary and an inverted list.

  • The underlying implementation of inversion indexes is based on: Finite State metadata (FST) data structure. The data structure that Lucene has used extensively since version 4+ is FST. FST has two advantages:

    • Small space footprint. By reusing the prefixes and suffixes of words in the dictionary, the storage space is reduced.
    • Fast query speed. O(len(STR)) query time complexity.

Select * from elasticSearch; select * from elasticSearch

Interviewer: I want to know the operation and maintenance ability of large data volume.

Answer: Index data planning, should do a good job in the early planning, is the so-called “design first, coding after”, so as to effectively avoid the sudden data surge caused by the cluster processing capacity insufficient online customer search or other business affected. How to tune, as mentioned in Question 1, is detailed here:

  • Dynamic index level based on template + time + Rollover API rolling index creation, example: design stage definition: blog cited template format: blog_index_ timestamp form, increasing data every day. The advantage of this is that the data volume does not surge so that the data volume of a single index is very large, close to the 32nd power -1 of upper limit 2, and the index storage reaches TB+ or even larger. Once a single index is large, storage and other risks come with it, so think ahead + avoid early.
  • Storage level

    Hot data (for example, data generated in the latest three days or one week) is stored separately, and other data is stored separately. For cold data, new data is not written to the systemforce_merge 加 shrinkCompression operation, saving storage space and retrieval efficiency.
  • Once the deployment level is not planned in advance, it is an emergency policy. Combined with the dynamic expansion feature of ES itself, dynamic new machines can relieve the cluster pressure. Note: if the master nodes are properly planned before, dynamic new machines can be completed without restarting the cluster.

How does ElasticSearch implement master voting

Interviewer: To understand the underlying principles of ES clustering, not just the business level.

Answer: Preconditions:

  • Only nodes that are candidate primary nodes (master: true) can become primary nodes.

  • The purpose of the minimum number of primary nodes (MIN_master_nodes) is to prevent brain splitting. This I looked at a variety of online analysis of the version and source code analysis of the books, clouds in the fog. Select the Master node and return the corresponding Master node successfully, otherwise return null. The election process is roughly described as follows:

    • Step 1: Confirm that the number of candidate primary nodes reaches the standard,elasticsearch.ymlSet the value of theDiscovery. Zen. Minimum_master_nodes;
    • Step 2: Comparison: First determine whether the master qualification, with candidate master node qualification priority return; If both nodes are candidate primary nodes, the value with a small ID is the primary node. Note that the id here is of type string.

Off-topic: how to get the node ID.

1GET /_cat/nodes? v&h=ip,port,heapPercent,heapMax,id,name 2ip port heapPercent heapMax id nameCopy the code

Describe the process of indexing documents for Elasticsearch in detail

The interviewer: To understand the underlying principles of ES, look beyond the business level.

answerThe index document here should be understood as the document writing ES, the process of creating the index. Document writing includes single-document writing and bulk writing. This section describes the single-document writing process. Remember this diagram from the official documentation.

  • Step 1: The customer writes data to a node in the cluster and sends a request. (If no routing/coordination node is specified, the requested node acts as the routing node.)
  • Step 2: After node 1 receives the request, it uses the document id to determine that the document belongs to Shard 0. The request will be forwarded to another node, let’s say node 3. Therefore, the primary shard of shard 0 is allocated to node 3.
  • Step 3: Node 3 performs a write operation on the master shard. If successful, the request is forwarded to the replica shards of node 1 and node 2 in parallel and waits for the result to return. All replica shards report success, node 3 reports success to the coordinator node (node 1), and node 1 reports success to the requesting client.

If the interviewer asks again: How do you get sharded documents in step 2? A: Obtained by the routing algorithm. The routing algorithm is the process of calculating the target fragment ID based on the route and document ID.

1shard = hash(_routing) % (num_of_primary_shards)
Copy the code

How does Elasticsearch search work?

Interviewer: You want to understand the underlying principles of ES search, not just the business level. Answer: The search is decomposed into two phases of “Query then fetch”. The purpose of the Query phase is to locate the position without fetching it. The steps are as follows:

  • Suppose an index data has 5 master +1 copies with 10 shards, one of which will be hit in one request.
  • Each fragment is queried locally, and the results are returned to the local ordered priority queue.
  • The results of step 2 are sent to the coordination node, which produces a global sorted list.

The purpose of the FETCH phase is to fetch data. The routing node retrieves all documents and returns them to the client.

How to optimize the Settings for Linux when Elasticsearch is deployed

Interviewer: I want to know the operation and maintenance capability of ES cluster.

  • Close the cacheswap;
  • Heap memory is set to: Min (node memory /2, 32GB);
  • Set the maximum number of file handles.
  • Adjust thread pool + queue size according to service needs;
  • Disk storageraidMode – Storage conditional useRAID10To improve the performance of a single node and avoid storage failures of a single node.

What is the internal structure of Lucence?

Interviewer: I want to know the breadth and depth of your knowledge.



Lucene is an index and search process, including index creation, index, and search. You can build on that a little bit.

How does Elasticsearch implement Master voting?

  • The primary of Elasticsearch isZenDiscoveryThe module is responsible for the Ping (RPC between nodes to find each other) andUnicastThe unicast module contains a host list to control which nodes need to be pinged.
  • For all nodes that can become master (node.master: true), select the first node (0th bit) in the order of the known nodes according to the nodeId dictionary.
  • If the number of votes for a node reaches a certain value (n/2+1) and the node elects itself, that node is master. Otherwise, the election is re-run until the conditions described above are met.
  • Note: The master node is responsible for cluster, node and index management, not document level management. The data node can turn off HTTP functionality *.

What if 10 of the 20 Elasticsearch nodes select one master and the other 10 select another master?

  • If the number of master candidates in the cluster is not less than 3, you can set the minimum number of votes (discovery.zen.minimum_master_nodes) more than half of all candidates to solve the brain split problem;
  • When the number of candidates is two, only the master candidate can be changed, and the other candidates can be used as data nodes to avoid the brain-splitting problem.

How do clients select specific nodes to execute requests when connecting to a cluster?

The TransportClient uses the Transport module to remotely connect to an ElasticSearch cluster. It does not join the cluster, but simply obtains one or more initialized transport addresses and communicates with them in a polling manner.

Describe the process of indexing documents for Elasticsearch in detail

By default, the coordination node participates in the calculation using the document ID (routing is also supported) to provide the appropriate shard for the route.

shard = hash(document_id) % (num_of_primary_shards)
Copy the code
  • When the shard node receives a request from the coordinator node, it writes the request to the Memory Buffer and then to the Filesystem Cache periodically (every second by default). This process from Momery Buffer to Filesystem Cache is called refresh;
  • Of course, in some cases, Momery Buffer and Filesystem Cache data may be lost. ES ensures data reliability through the translog mechanism. The data in Filesystem cache is flushed when the data is written to the disk.
  • During Flush, the buffer in memory is cleared, the content is written to a new segment, the segment’s fsync creates a new commit point and flusher the content to disk, and the old translog is deleted and a new Translog is started.
  • Flush is triggered when it is timed (default: 30 minutes) or when translog becomes too large (default: 512 MB).



Added: About Lucene seinterfaces:

  • Lucene indexes are composed of multiple segments, which themselves are a fully functional inverted index.
  • Segments are immutable, allowing Lucene to incrementally add new documents to the index without having to rebuild the index from scratch.
  • For each search request, all segments in the index are searched, and each segment consumes CPU clock cycles, file handles, and memory. This means that the higher the number of segments, the lower the search performance.
  • To solve this problem, Elasticsearch merges small segments into a larger segment, commits the new merged segments to disk, and removes those old segments.

Describe how Elasticsearch updates and deletes documents in detail

  • Delete and update are also write operations, but documents in Elasticsearch are immutable and therefore cannot be deleted or changed to show their changes.
  • Each segment on disk has a corresponding.delFile. When a delete request is sent, the document is not actually deleted.delThe file is marked for deletion. The document will still match the query, but will be filtered out of the results. When segments are merged, the.delDocuments in files marked for deletion will not be written to the new segment.
  • When a new document is created, Elasticsearch assigns a version number to the document, and when the update is performed, the old version of the document is in.delThe file is marked for deletion and the new version of the document is indexed to a new segment. Older versions of documents still match the query, but are filtered out of the results.

Describe the process of Elasticsearch in detail

  • The search is performed as a two-phase process called Query Then Fetch;
  • During the initial query phase, the query is broadcast to each shard copy (primary shard or duplicate shard) in the index. Each shard performs a search locally and builds a priority queue matching documents of size from + size. PS: Filesystem Cache is queried during search, but some data is still stored in Memory Buffer, so the search is performed in near real time.
  • Each shard returns the IDS and sorted values of all documents in its respective priority queue to the coordinator node, which merges these values into its own priority queue to produce a globally sorted list of results.
  • Next comes the fetch phase, where the coordination node identifies which documents need to be fetched and submits multiple GET requests to the related shard. Each shard loads and enriches the document, then returns the document to the coordination node if necessary. Once all documents have been retrieved, the coordination node returns the result to the client.
  • Supplement: The search type of Query Then Fetch refers to the data of this slice when scoring the document relevance, which may not be accurate when the number of documents is small. DFS Query Then Fetch adds a pre-query processing. Ask Term and Document Frequency, this score is more accurate, but performance deteriorates.

What are the optimizations for Linux Settings when Elasticsearch is deployed?

  • Machines with 64 GB of ram are ideal, but 32 GB and 16 GB machines are also common. Less than 8 GB is counterproductive.
  • If you have to choose between faster CPUs and more cores, more cores is better. The extra concurrency provided by multiple cores far outweighs a slightly faster clock rate.
  • If you can afford an SSD, it will go far beyond any rotating media. Ssd-based nodes have improved query and index performance. SSDS are a good choice if you can afford them.
  • Avoid clustering across multiple data centers, even if they are in close proximity. Clustering across large geographical distances is definitely avoided.
  • Make sure the JVM running your application is exactly the same as the server’s JVM. In several places in Elasticsearch, Java’s native serialization is used.
  • By setting theGateway. recover_after_nodes, gateway.expected_nodes, gateway.recover_after_timeExcessive shard swapping can be avoided when the cluster restarts, which can shorten data recovery from hours to seconds.
  • Elasticsearch is configured to use unicast discovery by default to prevent nodes from unintentionally joining a cluster. Only nodes running on the same machine automatically form a cluster. It is best to use unicast instead of multicast.
  • Do not arbitrarily change the size of the garbage collector (CMS) and individual thread pools.
  • Give Lucene (less than) half of your memory (but no more than 32 GB!). Through theES_HEAP_SIZEEnvironment variable Settings.
  • Swapping memory to disk is fatal to server performance. If memory is swapped to disk, a 100 microsecond operation can become 10 milliseconds. And think about all those 10-microsecond delays that add up. It’s not hard to see how awful performance considerations are.
  • Lucene uses a lot of files. Elasticsearch also uses a lot of sockets to communicate between nodes and HTTP clients. All of this requires sufficient file descriptors. You should increase your file descriptor and set it to a large value, such as 64,000.

Added: Index phase performance enhancement method

  • Use batch requests and resize them: 5-15 MB of data per batch is a good starting point.
  • Storage: Use SSDS
  • Segment and merge: Elasticsearch defaults to 20 MB/s, which should be a good setup for mechanical disks. If you’re using SSD, consider going up to 100-200 MB/s. If you’re doing bulk imports and don’t care about search at all, you can turn merge limiting off completely. And you can add moreindex.translog.flush_threshold_sizeSetting from the default of 512 MB to a larger value, such as 1 GB, which accumulates larger segments in the transaction log during a purge trigger.
  • If your search results don’t require near-real-time accuracy, consider indexing each of themindex.refresh_intervalChanged to 30 s.
  • If you’re doing bulk imports, consider setting them upindex.number_of_replicas: 0Close the copy.

For the GC side, what should I look out for when using Elasticsearch?

  • SEE: elasticsearch. Cn/article / 32
  • The index of the inverted dictionary needs to be resident in memory, not GC, and needs to be monitored on a data nodesegment memoryGrowth trend.
  • All kinds of caches,field cache, filter cache, indexing cache, bulk queueWait, to set a reasonable size, and should the heap be sufficient in the worst-case scenario, i.e., when all kinds of caches are full, and there is still heap space to allocate to other tasks? Avoid using clear Cache to free memory.
  • Avoid searches and aggregations that return large result sets. Scenarios that do require a large amount of pull data can be usedscan & scroll apiTo implement.
  • Cluster STATS resides in memory and cannot be expanded horizontally. A super-large cluster can be divided into multiple clusters connected by a tribe Node.
  • To determine whether heap is sufficient, the heap usage of a cluster must be continuously monitored based on actual application scenarios.

How to implement Elasticsearch aggregation for large data (tens of millions of magnitude)?

The first approximation aggregation provided by Elasticsearch is cardinality metrics. It provides the cardinality of a field, the number of distinct or unique values for that field. It is based on the HLL algorithm. HLL will first for our input hash algorithm, and then according to the result of the hash operation base of the bits do probability estimation is obtained. It features configurable precision to control memory usage (more precision = more memory); Small data sets are very accurate; We can configure parameters to set the fixed amount of memory required for deduplication. Whether unique values are in the thousands or billions, the amount of memory used depends only on the precision of your configuration.

How does Elasticsearch guarantee read/write consistency in concurrent cases?

  • Optimistic concurrency control can be used by version numbers to ensure that the new version is not overwritten by the old version, leaving the application layer to handle specific conflicts;
  • In addition, consistency levels are supported for write operationsquorum/one/all, defaults to quorum, that is, write operations are allowed only when most shards are available. But even if most are available, there may be a failure to write to the copy due to network reasons, so that the copy is considered faulty and the shard is rebuilt on a different node.
  • For read operations, you can set Replication to sync(the default), which makes the operation return only after both the primary and secondary shards have completed; If Replication is set to Async, you can also set the search request parameter _preference to primary to query the primary shard to ensure that the document is of the latest version.

How do I monitor the Status of the Elasticsearch cluster?

Marvel makes it easy to monitor Elasticsearch via Kibana. You can view your cluster health and performance in real time, as well as analyze past cluster, index, and node metrics.

Describe the overall technical architecture of your e-commerce search.

Do you know dictionary trees?

The common dictionary data structure is as follows:



The core idea of Trie is to use the common prefix of the string to reduce the cost of query time to improve efficiency. It has three basic properties:

  • The root node contains no characters, and each node except the root node contains only one character.
  • From the root node to a node, the characters passing through the path are concatenated to be the string corresponding to the node.
  • All children of each node contain different characters.

  • As you can see, the number of nodes at each level of the trie tree is 26^ I. So to save space, we can also use dynamic linked lists, or we can use arrays to simulate dynamics. The cost of space is no more than the number of words x the length of words.
  • Implementation: for each node to open a letter set size array, each node to hang a linked list, using left child right brother notation to record the tree;
  • For the Chinese dictionary tree, the child nodes of each node are stored in a hash table, so as not to waste too much space, and the query speed can retain the complexity of the hash O(1).

How is spelling correction implemented?

  • Spelling correction is based on editing distance. Edit distance is a standard method of representing the minimum number of operation steps required to convert from one string to another through insert, delete, and replace operations.
  • How to calculate the edit distance: For example, to calculate the edit distance for Batyu and Beauty, first create a 7×8 table (batyu length is 5, coffee length is 6, add 2 each), then fill in the following positions with black numbers. The other cells are computed by taking the minimum of the following three values: the upper-left digit if the uppermost character is equal to the left-most character. Otherwise, the top left digit +1. (0 for 3,3) left digit +1 (2 for 3,3 cells) top digit +1 (2 for 3,3 cells). Finally, the value in the lower right corner is the value of editing distance 3.



For spelling correction, we consider constructing a Metric Space in which any relation meets the following three basic conditions:

D (x,y) = 0 — if the distance between x and y is 0, then x=y d(x,y) = d(y,x) — the distance from x to y is the same as the distance from y to x d(x,y) + d(y,z) >= d(x,z) — triangle inequality

  • According to the triangle inequality, then another character that is within the range of n from query turns B, and its distance from A is at most D + N and at least D-n.
  • The construction process of BK tree is as follows: each node has any child nodes, and each edge has a value to represent the editing distance. The edge of all child nodes to the parent node is marked with n to indicate that the editing distance is exactly N. For example, we have a tree whose parent is “book” and two children, “cake” and “books”. The edge from “book” to “books” is numbered 1, and the edge from “book” to “cake” is numbered 4. After constructing the tree from the dictionary, whenever you want to insert a new word, calculate the edit distance of the word from the root node and look for edges with a value of D (neweord, root). The recursion is compared with each child until there are no children, and you can create a new child and store the new word there. For example, to insert “boo” into the tree in the example above, we first examine the root node, looking for the edge where D (” book “, “boo”) = 1, and then examine the children of the edge numbered 1 to get the word “books”. We calculate the distance D (” books “, “boo”)=2 and insert the new word after “books” with the edge number 2.
  • Search for similar words as follows: Calculate the edit distance D between the word and the root node, and then recursively find the edges of each child node labeled d-n to D +n (inclusive). If the distance d between the checked node and the search word is less than n, the node is returned and the query continues. Cake (” cape “, “cape”)=1 if the maximum tolerance distance is 1, then d(” book “, “cape”)=4 and d(” book “, “cape”)= 5 and d(” cake “, “cape”)=1 So I’m going to return cake, and then I’m going to find the cape and cart nodes that are 0 to 2 away from cake, and I’m going to get cape.