preface

Elasticsearch is an open source distributed search service created by Shay Banon. Since its release in February 2010, Elasticsearch has grown to become a very important part of the search and data analysis solution space.

Visualization tool

As with relational databases, we need a visualization tool for Elasticsearch. The most common one is Kibana, which is the K of ELK. The remaining L is Logstash.

kibana

Kibana is a very powerful tool and with Kibana we can do three things:

  1. Search, observe, and Protect: From discovering documents to analyzing logs to discovering security vulnerabilities, Kibana is your portal to these and other features.
  2. Visualization is data analysis: analyzing data found in charts, gauges, maps, etc., and combining them into dashboards.
  3. Management monitoring is protectionElastic Stack: Manages indexing and extraction pipes, monitors the health of the Elastic Stack cluster, and controls which users have access to which functions.

Kibana installation

The installation of Kibana is also relatively easy. Click here to download the corresponding version and decompress it. The decompressed home directory is $KIBANA_HOME:

Again, we need to go to the config directory and modify the kibana configuration file kibana.yml.

Kibana configuration

Like Elasticsearch, once Kibana is installed, we need to configure some of the most important configuration files:

  • elasticsearch.hosts

Configure the Elasticsearch service to be connected. If multiple Elasticsearch services are configured, they must belong to the same Elasticsearch cluster. The default value is:

elasticsearch.hosts: ["http://localhost:9200"]
Copy the code
  • server.name

Identifies the current unique instance of Kibana. The default is the host name, which serves only as a description identifier.

  • server.host

Specify the host name for Kibana, which by default is localhost, indicating that only local access is allowed. If remote connections are required, change this configuration to a non-local loopback address such as an external IP address or domain name, or use 0.0.0.0 to allow all remote host connections.

  • server.port

Configure the kibana port number. The default is 5601.

  • elasticsearch.requestTimeout

Number of milliseconds to wait for background or Elasticsearch response. Default value: 30000.

  • elasticsearch.pingTimeout

Wait for Elasticsearch ping return milliseconds, the default value is equal to the Elasticsearch. RequestTimeout configuration of the time.

  • Elasticsearch. Username and elasticsearch. Password

If the account password is configured for Elasticsearch, you need to configure the account and password here.

  • path.data

The path in Kibana that stores persistent data that is not saved in Elasticsearch.

  • logging.dest

Specify the log path, default is stdout.

  • server.basePath

Specifies the base path to access Kibana. The default path is /.

  • server.rewriteBasePath

Whether to keep the server.basePath path after redirection. The default is false in Kibana 6.3 and earlier, and true in Kibana 7.0 and later.

Kibana startup

After configuring the configuration file, you can go to the bin directory and run the./kibana command to start Kibana. To start Kibana in the background, you can run the./kibana & command.

Once launched, you can visit Kibana at http://ip:5601/{basePath}

Select * from Dev Tools for Elasticsearch; select * from Dev Tools for Elasticsearch;

In addition to Dev Tools, Kibana has many other powerful features that you can try out on your own.

Elasticsearch start operation

Elasticsearch is a REST API that allows you to manage indexes, change instance parameters, and check node and cluster status. And CRUD operations are performed on the document data in the index.

The operating index

An index is a database, so let’s look at the basic operations of an index.

Create index

Create an index using the PUT method:

PUT /my-index-001
Copy the code

If you define a mapping, you can add the mapping along with the index:

PUT /my-index-002
{
  "mappings": {
    "properties": {
      "field1": { "type": "text"}}}}Copy the code

You can also create an alias for an index:

PUT /my-index-003
{
  "mappings": {
    "properties": {
      "field1": { "type": "text"}}},"aliases": {
    "alias_name": {}}}Copy the code

Once you have an alias, the query index can be queried using the alias.

What does an index alias do

Index aliases can be useful in some scenarios, such as when we go live and we need to rebuild the index in Elasticsearch if we need to change the field due to some business change. We can specify the same alias when we rebuild the index, but if our code is querying through the alias, then we can rebuild the index seamlessly.

Remove the index

The DELETE method is used to DELETE an index.

DELETE /my-index-0001
Copy the code

Index of the query

The GET method is used to query index information, which can return index information such as setting, mapping, aliases and sharding.

GET /my-index-001
Copy the code

Exist index

The HEAD method is used to determine whether the index exists.

HEAD /my-index-001
Copy the code
  • clone index

Before cloning an index, you must first make an index read-only (and ensure that the cluster status is green) :

PUT /my-index-004/_settings
{
  "settings": {
    "index.blocks.write": true}}Copy the code

To clone the index, use the POST method:

POST /my-index-004/_clone/cloned-my-index-004
Copy the code

The document operation

Now that you’ve learned the basics of index, you can learn the basics of document.

Inserted into the document

Insert document using POST method:

POST /my-index-001/_doc/? pretty{"@timestamp": "2099-11-15T13:12:00"."message": GET /search HTTP/1.1 200 1070000."user": {    "id": "kimchy"  }}
Copy the code

Elasticsearch automatically generates an ID value because no ID is generated in the above statement.

To insert a document with the specified ID, use the _resource statement:

PUT my-index-001/_create/1{  "@timestamp": "2099-11-15T13:12:00"."message": GET /search HTTP/1.1 200 1070000."user": {    "id": "kimchy"  }}
Copy the code

Or specify op_type=create:

PUT my-index-001/_doc/2? op_type=create{"@timestamp": "2099-11-15T13:12:00"."message": GET /search HTTP/1.1 200 1070000."user": {    "id": "kimchy"  }}
Copy the code

Query the document

Simple queries can use the GET method:

GET /my-index-001/_search
Copy the code

Select * from Elasticsearch (hits); select * from Elasticsearch (hits);

If you want to query only documents with a specific ID, you can use the following statement:

GET /my-index-001/_doc/1
Copy the code

If we want to specify a return field or not to return a field, we can use parameters such as _source.

GET my-index-001/_doc/1? _source_includes=*.id&_source_excludes=@timestamp
Copy the code

If we do not want to return any of our own stored fields, we can use the following statement:

GET my-index-001/_doc/1? _source=false
Copy the code

Delete the document

To DELETE a document, use the DELETE method. The syntax is DELETE /

/_doc/<_id>.

If id is specified, the timeout period is 5 minutes:

DELETE /my-index-001/_doc/1? timeout=5mCopy the code

If you want to delete all fields, you can use the delete_by_query statement:

POST my-index-001/_delete_by_query{  "query": {    "match_all": {}}}Copy the code

Similarly, delete_by_query can also specify conditional deletions:

POST /my-index-001/_delete_by_query{  "query": {    "match": {      "user.id": "elkbee"}}}Copy the code

Update the document

Update document usually uses the POST method, using the _update type, let’s demonstrate an update statement:

  • Insert a piece of data totestIndex:
PUT test/_doc/1{  "name":The Twin Lone Wolves.."age":18."address":"Shenzhen, Guangdong"}
Copy the code
  • rightnameField to update:
POST test/_update/1{  "doc": {    "name": "Gemini and Lone Wolf 2"  }}
Copy the code
  • Execute query statementGET test/_searchQuery, then findnameThe field has been updated:

Update document with a script

Update statements can also use script for more flexible operations (increasing the age field by 4 as shown below) :

POST test/_update/1{  "script" : {    "source": "ctx._source.age += params.count"."lang": "painless"."params" : {      "count" : 4}}}Copy the code

Like delete_by_query, update statements can use update_by_query:

POST test/_update_by_query{  "script": {    "source": "ctx._source.age += params.count"."lang": "painless"."params" : {      "count" : 4}},"query": {    "match": {      "name": "Gemini and Lone Wolf 2"}}}Copy the code

Reindex operation

Sometimes we need to copy the data of one index to another index. In this case, we can use the reindex operation. The difference between the clone operation and the previous one is that the reindex operation only migrates the document data, but does not transfer the setting. Information such as mapping and sharding is migrated to the new index, and the old index does not need to be set to read-only state when reindex is performed.

POST _reindex? wait_for_completion=false{  "source": {    "index": "old-index"  },  "dest": {    "index": "new-index"  }}
Copy the code

By default, the wait_for_completion parameter is true, meaning that the current operation will block until it succeeds, or an error will be reported if it has not completed by the timeout period. Therefore, if there is a large amount of data, you can change the wait_for_completion parameter to false.

conclusion

This article introduces how to install and configure Elasticsearch index and Document using Kibana. Of course, these statements are not enough if you want to use Elasticsearch in depth. The advanced query and analysis statements are at the heart of Elasticsearch.