preface

  • Businesses need to take a snapshot of ElasticSearch to AWS S3, copy the snapshot to Windows, and revert to ElasticSearch. As shown in the figure below:

  • The environment

    Elasticsearch Windows Server 2019 Ubuntu 20.04

ES Cluster 1 installs the S3 plug-in

  • The official document: https://www.elastic.co/guide/…
  • Plug-ins should be installed on all nodes of the ES cluster
  • Download S3 plug-ins and SHA hashes and upload them to the ES server

  • View and verify

    # ll-ah total 4.6m drwxr-xr-x 2 qhy qhy 4.0K Mar 9 01:55./ drwxr-xr-x 7 qhy qhy 4.0K Mar 9 01:50.. / rw - rw - r - 1 qhy qhy 4.6 M Mar 9 01:55 repository - s3-7.10.1. Zip - rw - rw - r - 1 qhy qhy 154 Mar 9 01:55 Repository -S3-7.10.1.zip. SHA512 # shasum -a 512-c Repository -S3-7.10.1.zip. SHA512: OK

  • The installation

    # / usr/share/elasticsearch/bin/elasticsearch - plugin install file:///home/qhy/es_down/repository-s3-7.10.1.zip - > Installing file:///home/qhy/es_down/repository-s3-7.10.1.zip - > Downloading file:///home/qhy/es_down/repository-s3-7.10.1.zip [= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =] 100% @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ java.lang.RuntimePermission accessDeclaredMembers java.lang.RuntimePermission getClassLoader java.lang.reflect.ReflectPermission suppressAccessChecks java.net.SocketPermission * connect,resolve java.util.PropertyPermission es.allow_insecure_settings read,write See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y -> Installed repository-s3
  • restartESservice

Add KEY to ES cluster 1

  • The official document: https://www.elastic.co/guide/…
  • Key should be added on all nodes of the ES cluster

  • Check which keys are available

    /usr/share/elasticsearch/bin/elasticsearch-keystore list

  • Add access_key

    /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key

  • Add secret_key

    /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key

  • reload

    POST _nodes/reload_secure_settings

Create the S3 snapshot repository

  • The official document: https://www.elastic.co/guide/…
  • Note that China is not supportedregionwithendpointalternative
PUT _snapshot/my_s3_repository { "type": "s3", "settings": { "endpoint": "s3.cn-northwest-1.amazonaws.com.cn", "bucket": "zt-hadoop-cn-northwest-1", "base_path": "ES_SNAPSHOT "," MAX_SNAPSHOT_BYTES_PER_SEC ": "200MB ", # Adjust the speed of snapshot creation, default 40MB" MAX_RESTOR_BYTES_PER_SEC ": "200MB" # Adjust snapshot recovery speed, default unlimited}}
  • View all registered snapshot repositories
GET _snapshot/_all
  • Delete snapshot repository
DELETE _snapshot/my_s3_repository
  • Adjust cluster recovery sharding speed and number of concurrency
PUT _cluster/settings { "transient": { "indices.recovery.max_bytes_per_sec": "200 MB", # 40 MB by default "cluster. Routing. Allocation. Node_concurrent_recoveries" : "5 # default 2}}"
  • View the cluster configuration (including default configuration)
GET _cluster/settings? flat_settings&include_defaults

Create a snapshot

  • The official document: https://www.elastic.co/guide/…
PUT /_snapshot/my_s3_repository/snapshot_zt
{
  "indices": "zt_product_doc_index_20210223_3",
  "ignore_unavailable": true,
  "include_global_state": false
}
  • Look at amy_s3_repositoryAll snapshots in the repository
GET _snapshot/my_s3_repository/_all
  • To viewsnapshot_ztThe profile state of the snapshot
GET _snapshot/my_s3_repository/snapshot_zt
  • To viewsnapshot_ztThe detailed state of the snapshot
GET _snapshot/my_s3_repository/snapshot_zt/_status
  • To delete a snapshot
DELETE _snapshot/my_s3_repository/snapshot_zt

Restore snapshots from S3

  • The official document: https://www.elastic.co/guide/…

    POST /_snapshot/my_s3_repository/snapshot_zt/_restore
{
  "indices": "zt_product_doc_index_20210223_3",
  "index_settings": {
"index.number_of_replicas": 0
  },
  "rename_pattern": "zt_product_doc_index_20210223_3",
  "rename_replacement": "zt_product_doc_index_20210223_3_restore"
}

  • Add a copy of the

    PUT zt_product_doc_index_20210223_3_restore/_settings
{
  "index.number_of_replicas" : "1"
}

Restore snapshots from the Windows shared directory

  • Files on S3 have been copied to the Windows shared directory and mounted to the/MNT /winshare directory on the ES Cluster 2 server

    # tree-d/MNT/Winshare/sigma ── es_snapshot sigma ── indices sigma ─ km6svccrtugjp-634aduyg ├─ 0 sigma ─ 1 sigma ─ 2
  • See Ubuntu 20.04 to read and write the Windows 10 shared directory for Linux mount

  • Add the following configuration to the elasticsearch.yml file and restart the ES service

    path.repo:
- /mnt/winshare

  • Create a snapshot repository for your Windows system

    PUT /_snapshot/my_backup
{
"type": "fs",
"settings": {
  "location": "/mnt/winshare/es_snapshot"
}
}

  • The official documentation

    POST /_snapshot/my_backup/snapshot_zt/_restore
{
"indices": "zt_product_doc_index_20210223_3",
"index_settings": {
  "index.number_of_replicas": 0
},
"rename_pattern": "zt_product_doc_index_20210223_3",
"rename_replacement": "zt_product_doc_index_20210223_3_smb"
}

  • Add a copy of the

    PUT zt_product_doc_index_20210223_3_smb/_settings
{
"index.number_of_replicas" : "1"
}

This article from the
qbit snap