2. Operate document data in batches

Bulk data is written to documents in batches using the _BULK API. The request mode is as follows: POST Request Address: _BULK request parameters: The _BULK operation document contains at least two rows of parameters (or even rows of parameters). The first row of parameters specifies the operation type and the operation object (index,type, and ID). The second row is the operation data.

{"actionName":{"_index":"indexName", "_type":"typeName","_id":"id"}} 
{"field1":"value1", "field2":"value2"} 
Copy the code

ActionName: Indicates the operation type, including create,index,delete, and update. (1) Create documents in batches

POST _bulk 
{"create":{"_index":"article", "_type":"_doc", "_id":3}} 
{"id":3, "title" : "li bai", "content" : "li bai 666", "tags" : [" Java ", "to like"], "create_time" :1554015482530} 
{"create":{"_index":"article", "_type":"_doc", "_id":4}} 
{"id":4, "title" : "li bai", "content" : "li bai NB", "tags" : [" Java ", "to like"], "create_time" :1554015482530} 
Copy the code

Returns the result

{
  "error" : {
    "root_cause" : [
      {
        "type" : "json_parse_exception",
        "reason" : "Invalid UTF-8 start byte 0xb1\n at [Source: (byte[])\"POST /_bulk? pretty=true HTTP/1.1\r\nx-forwarded-for: 127.0. 01.\r\nx-forwarded-port: 61337\r\nx-forwarded-proto: http\r\nx-forwarded-host: localhost:5601\r\ncontent-type: application/json\r\nhost: localhost\r\ntransfer-encoding: chunked\r\nConnection: close\r\n\r\n14d\r\n{\"create\":{\"_index\":\"article\", \"_type\":\"_doc\", \"_id\":3}}\n{\"id\":3Li bai, \ "title \" : \ '\ "and \" content \ ": \" li bai 666 \ ", \ "tags \" : [\ Java \ ", \ "oriented to like \ n \], \" create_time \ ":1554015482530}\n{\"create\":{\"_index\":\"article\", \"_type\":\"_doc\", \"_id\":4}}\n{\"id\":4Li bai, \ "title \" : \ '\ ", \ "c \ [truncated89 bytes]; line: 1.column: 3]"
      }
    ],
    "type" : "json_parse_exception",
    "reason" : "Invalid UTF- 8 - start byte 0xb1\n at [Source: (byte[])\"POST /_bulk?pretty=true HTTP/1.1\r\nx-forwarded-for: 127.0.0.1\r\nx-forwarded-port: 61337\r\nx-forwarded-proto: http\r\nx-forwarded-host: localhost:5601\r\ncontent-type: application/json\r\nhost: localhost\r\ntransfer-encoding: chunked\r\nConnection: close\r\n\r\n14d\r\n{\"create\ ": {\" _index \ ": \" article \ ", \ "_type \" : \ "_doc \", \ "_id \" : 3}} \ n {\ \ "id" : 3, \ "title \" : \ "li bai \", \ "content \" : \ "li baiAwesome!\ "and \" tags \ ": \" Java \ ", \ "oriented to like \ n \], \" create_time \ ": {1554015482530} \ n \"create\ ": {\" _index \ ": \" article \ ", \ "_type \" : \ "_doc \", \ "_id \" : 4}} \ n {\ \ "id" : 4, \ "title \" : \ "li bai \", \ \ "c" [truncated 89 bytes); line: 1, column: 3]" }, "status" :400
}

Copy the code

​

(2) Create index or replace index in full

POST _bulk 
{"index":{"_index":"article", "_type":"_doc", "_id":3}} 
{"id":3, "title" : "du fu (a)", "content" : "du fu 666", "tags" : [" Java ", "object oriented"], "create_time" :1554015482530} 
{"index":{"_index":"article", "_type":"_doc", "_id":4}}
{"id":4, "title" : "du fu (2)", "content" : "du fu NB", "tags" : [" Java ", "object oriented"], "create_time" :1554015482530} 

Copy the code

** If the original document does not exist, create ** ** if the original document exists, replace (fully modify the original document) **

Returns the result

#! Deprecation: [types removal] Specifying types in bulk requests is deprecated.
{
  "took" : 651,
  "errors" : false,
  "items" : [
    {
      "index" : {
        "_index" : "article",
        "_type" : "_doc",
        "_id" : "3",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 0,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "article",
        "_type" : "_doc",
        "_id" : "4",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 1,
        "_primary_term" : 1,
        "status" : 201}}}]Copy the code

(3) Batch delete delete

POST _bulk 
{"delete":{"_index":"article", "_type":"_doc", "_id":3}} 
{"delete":{"_index":"article", "_type":"_doc", "_id":4}} 
Copy the code

Returns the result

#! Deprecation: [types removal] Specifying types in bulk requests is deprecated.
{
  "took" : 32,
  "errors" : false,
  "items" : [
    {
      "delete" : {
        "_index" : "article",
        "_type" : "_doc",
        "_id" : "3",
        "_version" : 2,
        "result" : "deleted",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 2,
        "_primary_term" : 1,
        "status" : 200
      }
    },
    {
      "delete" : {
        "_index" : "article",
        "_type" : "_doc",
        "_id" : "4",
        "_version" : 2,
        "result" : "deleted",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 3,
        "_primary_term" : 1,
        "status" : 200}}}]Copy the code

(4) Batch update

POST _bulk 
{"update":{"_index":"article", "_type":"_doc", "_id":3}} {" doc ": {" title" : "ES solution required internal strength"}} {" update ": {" _index" : "article", "_type" : "_doc", "_id" :4}} 
{"doc":{"create_time":1554018421008}} 
Copy the code

Returns the result

#! Deprecation: [types removal] Specifying types in bulk requests is deprecated.
{
  "took" : 3,
  "errors" : true,
  "items" : [
    {
      "update" : {
        "_index" : "article",
        "_type" : "_doc",
        "_id" : "3",
        "status" : 404,
        "error" : {
          "type" : "document_missing_exception",
          "reason" : "[_doc][3]: document missing",
          "index_uuid" : "uTqmplLRR3CpwQXWtAvJ7Q",
          "shard" : "0",
          "index" : "article"
        }
      }
    },
    {
      "update" : {
        "_index" : "article",
        "_type" : "_doc",
        "_id" : "4",
        "status" : 404,
        "error" : {
          "type" : "document_missing_exception",
          "reason" : "[_doc][4]: document missing",
          "index_uuid" : "uTqmplLRR3CpwQXWtAvJ7Q",
          "shard" : "0",
          "index" : "article"
        }
      }
    }
  ]
}

Copy the code

1. Query DSL overview

3. Search conditions exist

3.1.1 Fuzzy matching

Fuzzy matching is mainly for fields of the text type, which will segment the content. In the query, it will also segment the search conditions, and then find matched data through the inverted index. Fuzzy matching is mainly realized by parameters such as match

• Match: Use match keywords to blur matching conditions
• Prefix: prefix matching regexp: data is matched by regular expression
• Complex use of match
• The match condition also supports the following parameters:
• Query: Specifies the matched value
• Operator: Indicates the matching condition type
• And: match all conditional participles
• Or: Conditional segmentation after a match (default)
• Minmum_should_match: Specifies the minimum number of matches

3.2 Combination Query condition (Multi-Condition Query)

Combined condition query is a complete query condition formed by combining leaf condition query statements

• Bool: The relation between the conditions is AND, OR or not
• Must: All conditions must be satisfied, i.e. all conditions must be and
• Relation should: one of each condition can be satisfied, namely, each condition
• Or
• Must_not: All conditions are not met, that is, conditions are not
• Relationship between
• Filter: Does not count correlation scores. It does not count _score
• Relevancy scores are more efficient
• Constant_score: Relevance score is not calculated
• Subconditions such as must/filter/shoud/must_not are pass
• A term/terms/range/ids/exists/match leaves conditions as parameters

Note: The above parameters, when there is only one search condition, must and so on correspond to an object, when there are multiple conditions, correspond to an array

3.3 Join query (multi-document merge query)

• Parent /child document query: parent/child
• Nested document queries: nested

query DSL

​

In the query context, the query answers the question “Is this document relevant because it doesn’t match this query?” How do you verify a match that’s easy to understand, how do you compute the correlation? Index data in ES will store a _score value, the higher the score value represents the more matched. In addition, the calculation of the score for a particular search is very complex, so it takes a certain amount of time.

filter DSL

In the context of a filter, the query answers the question “Does this document match?” The answer is simple, yes or no. It doesn’t compute any points and doesn’t care about the sorting questions returned, so it’s a little more efficient. The filter context is the context in which the filter parameter is used, such as must_NOT or filter in bool queries. In addition, if the filter is used frequently, ES will automatically cache the contents of the filter, which improves the performance of the query. Some filtering cases:

3.5 Query: A case

Query the name term precisely by name. The term query does not perform a word segmentation query on the field

Adopt exact matching

Note: Term is used for precise query, and the query field mapping type belongs to keyword. For example:

POST /es_db/_doc/_search {" query ": {" term" : {" name ":" admin "}}} # equivalenceSQL: select * from student where name = 'admin' 
Copy the code

Multi-field fuzzy matching query and precise query multi_match

POST /es_db/_doc/_search {" query ": {" multi_match" : {" query ":" zhang ", "fields" : / "address", "name"}}} # equivalenceSQL: select * from student where name like 'Joe Smith % %' or address like Zhang '% thirty percent' 
Copy the code

Query_string specifies field conditions, including AND AND OR conditions

POST /es_db/_doc/_search {" query ": {" query_string" : {" query ":" admin OR changsha ", "fields" : (" name ", "address")}}}Copy the code

Paging, output field, sort comprehensive query

POST /es_db/_doc/_search 
{ 
"query" : { 
"range" : { 
"age" : { 
"gte":25, 
"lte":28 
} 
} 
}, 
"from": 0, 
"size": 2, 
"_source": ["name", "age", "book"], 
"sort": {"age":"desc"} 
}
Copy the code

Conclusion:

1. Match the match: Fuzzy match, you need to specify the field name, but the input will be split, for example, “Hello world” will be split into Hello and world, and then match, if the field contains Either Hello or world, or if the field contains both, the result will be queried, so match is a partially matched fuzzy query. The query conditions are relatively loose. 2. term term: This query is equivalent to match in some cases, like if we were to query for the single word hello, we would get the same result as match, but if we were to query for “hello world”, we would get a very different result, because the input is not segmented, which means that when we query, “Hello world” “Hello world” “Hello World” When saving “hello World “, elasticSearch will split the field into “Hello world” and” World “, there is no “Hello world”, so the query result will be empty. This is also the difference between term queries and match. 3. Match_phase Match_phase: the input will be segmented, but the result should also contain all the participles in the same order. In the case of “Hello world”, the result must contain” Hello “and” world”, and they must be connected in a fixed order. “Hello that word” is not satisfied, and “world Hello” is not satisfied. Query_string Query_string: Similar to match, but match requires a field name. Query_string searches for all fields in a wider range.

Document mapping

The mapping in ElasticSearch is similar to the schema information in MySQL.

1. Mapping in ES can be divided into dynamic mapping and static mapping

2.1 Deleting an Index

DELETE /Es_db # returns {"acknowledged" :true
}

Copy the code

2.3 Creating a Document (ES automatically creates a mapping based on the data type)

PUT /es_db/_doc/1 
{ 
"name": "Jack", 
"sex": 1, 
"age": 25, "book", "introduction to Java to master", "address" : "the guangzhou baiyun airport" # returns {} "_index" : "es_db", "_type" : "_doc", "_id" : "1", "_version" :1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Copy the code

3 Static Mapping

3.2 Creating An Index

PUT /es_db
Copy the code

3.4 Creating Documents Based on static Mappings

PUT /es_db/_doc/1 
{ 
"name": "Jack", 
"sex": 1, 
"age": 25, "book", "introduction to elasticSearch to master", "address" : "guangzhou car vicious" # returns {} "_index" : "es_db", "_type" : "_doc", "_id" : "1", "_version" :1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Copy the code

4. Core datatype

• String: string. The string type contains text and keyword.
• Text: This type is used to index long text. Before creating an index, the text will be segmented into word combinations to create an index. Es is allowed to retrieve these words. Text cannot be used for sorting and aggregation.
• Keyword: This type is non-word segmentation and can be used for search filtering, sorting, and aggregation. The keyword type cannot be used for word segmentation fuzzy retrieval with text.
• Numeric types: long, INTEGER, short, byte, double, float
• Date type: Date
• Boolean: Boolean

Specify an IK toggle of type TEXT when creating a static map

2. Word segmentation query

POST /es_db/_doc/_search {" query ": {" match" : {" address ":" wide "}}} return to # #! Deprecation: [types removal] Specifying types in search requests is deprecated.
{
  "took" : 81,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}


POST /es_db/_doc/_search {" query ": {" match" : {" address ":" guangzhou "}}} return to # #! Deprecation: [types removal] Specifying types in search requests is deprecated.
{
  "took" : 81,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}
Copy the code

4. Implement _version optimistic lock to update documents

Note: ES 7.x does not support this keyword

PUT /db_index/_doc/1? version=1 
{ 
"name": "Jack", 
"sex": 1, 
"age": 25} # return {"error" : {"root_cause" : [{"type" : "action_request_validation_exception", "reason" : "Validation Failed: 1: internal versioning can not be used for optimistic concurrency control. Please use `if_seq_no` and `if_primary_term` instead;" } ], "type" : "action_request_validation_exception", "reason" : "Validation Failed: 1: internal versioning can not be used for optimistic concurrency control. Please use `if_seq_no` and `if_primary_term` instead;" }, "status" :400
}

Copy the code

Java API operation ES

Subsequent independent article explanation

X. Setting up ES cluster environment

2. Modify the elasticsearch. Yml

Node1.hxl. cn The server uses the HXL user to modify the configuration file

The mkdir ‐ p/usr/local/es/Elasticsearch ‐7.101./The log mkdir ‐ p/usr/local/es/Elasticsearch ‐7.101./data 

cd /usr/local/es/Elasticsearch ‐7.101./Config rm ‐rf elasticSearch.yml vim elasticsearch.yml cluster.name: HXL ‐es node.name: node1.hxl.cn path.data:/usr/local/es/Elasticsearch ‐7.101./data 
path.logs: /usr/local/es/Elasticsearch ‐7.101./log 
network.host: node1.hxl.cn 
http.port: 9200discovery.seed_hosts: ["IP1", "IP2", "IP3"] cluster.initial_master_nodes: [" Node 1 name ", "node 2 name "," node 3 name "] bootstrap.system_call_filter:false 
bootstrap.memory_lock: false 
http.cors.enabled: trueHTTP. Cors. Allow ‐ origin: "*"Copy the code

4. Modify the ES configuration file on node2 and node3

Node2.hxl. cn and node3.hxl.cn also need to modify the ES configuration file node2.hxl.cn Run the following command as user HXL to modify the ES configuration file

The mkdir ‐ p/usr/local/es/Elasticsearch ‐7.101./The log mkdir ‐ p/usr/local/es/Elasticsearch ‐7.101./data 

cd /usr/local/es/Elasticsearch ‐7.101./Config vim elasticsearch.yml cluster.name: HXL ‐es node.name: node2.hxl.cn path.data:/usr/local/es/Elasticsearch ‐7.101./data 
path.logs: /usr/local/es/Elasticsearch ‐7.101./log 
network.host: node2.hxl.cn 
http.port: 9200discovery.seed_hosts: ["IP1", "IP2", "IP3"] cluster.initial_master_nodes: [" Node 1 name ", "node 2 name "," node 3 name "] bootstrap.system_call_filter:false 
bootstrap.memory_lock: false 
http.cors.enabled: trueHTTP. Cors. Allow ‐ origin: "*"Copy the code

Node3.hxl. cn Run the following command as user HXL to modify the configuration file

The mkdir ‐ p/usr/local/es/Elasticsearch ‐7.101./The log mkdir ‐ p/usr/local/es/Elasticsearch ‐7.101./data 

cd /usr/local/es/Elasticsearch ‐7.101./Config vim elasticsearch.yml cluster.name: HXL ‐es node.name: node3.hxl.cn10 path.data:/usr/local/es/Elasticsearch ‐7.101./data 
path.logs: /usr/local/es/Elasticsearch ‐7.101./log 
network.host: node3.hxl.cn 
http.port: 9200discovery.seed_hosts: ["IP1", "IP2", "IP3"] cluster.initial_master_nodes: [" Node 1 name ", "node 2 name "," node 3 name "] bootstrap.system_call_filter:false 
bootstrap.memory_lock: false 
http.cors.enabled: trueHTTP. Cors. Allow ‐ origin: "*"Copy the code

View cluster status:

GET _cat/nodes? vGET _cat/health? vCopy the code

1.2 Creating a Soft Connection

Node1.hxl. cn Run the following command to create a soft connection

Sudo ln ‐ s/usr/local/es/The node ‐ v81.. 0Linux ‐ ‐ x64/lib/node_modules/npm/bin/Np m ‐ cli. Js/usr/local/bin/NPM sudo ln ‐ s/usr/local/es/The node ‐ v81.. 0Linux ‐ ‐ x64/bin/node /usr/local/bin/nod 
e 
Copy the code

2 Local Installation

2.4 node1 Machine changes app.js

Modify app.js on the first machine

cd /usr/local/es/Elasticsearch ‐ head/_site 
vim app.js 


#在Vim中输入「:4354", locate to the first4354Line # modify HTTP://localhost:9200For HTTP://node1.hxl.cn:9200 
Copy the code

2.6 Accessing ElasticSearch-Head

Elasticsearch-head: select select * from elasticSearch-head; select select * from elasticSearch-head; select select * from elasticSearch-head; Go to/elasticSearch-7.6.1 /data/ and delete the nodes file. Restart the ES process on nodes 2 and 3