background

The certificate of the original ES cluster needs to be replaced because the certificate expires. The certificate update takes effect only after the entire cluster is restarted, and the service sensitivity is high. Therefore, the entire cluster cannot stop external services.

The solution

  • The OP is responsible for deploying a new ES cluster with the same version configuration as the original cluster

  • RD is responsible for developing the program

    • Import historical data to a new cluster
    • Business services are implemented in two setsESclustering

  • After the two clusters have the same data, the OP and RD work together to switch service traffic

  • OP updates the existing cluster certificate

ElasticSearch cluster

To prepare

  • ESVersion 5.5.1
  • Search Guard
  • JAVAVersion jdk1.8.0 _161

The cluster configuration

Three Master nodes (56 cores, 128 GB, and 3.7 TB common disks) 10.90.104.133 10.90.105.133 10.90.106.133 Five Data nodes (56 cores, 128 GB, and 3.7 TB common disks) 10.90.107.132 10.90.108.132 10.90.109.133 10.90.110.133 10.90.111.133 Port: 9201/9301Copy the code

System adjustment

Temporary $sudo sysctl -w vm.max_map_count=262144 Permanent $grep vm.max_map_count /etc/sysctl.conf $echo vm.max_map_count=262144 >> /etc/sysctl.conf

or 
$ vi /etc/sysctl.conf
vm.max_map_count=262144
Copy the code

The installationJAVA

$wget --no-cookies --no-check-certificate --header $wget --no-cookies --no-check-certificate --header"Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie"Create a directory, http://download.oracle.com/otn-pub/java/jdk/8u161-b12/2f38c3b165be4555a1fa6e98c45e0808/jdk-8u161-linux-x64.tar.gz Unzip the file $mkdir -p /usr/local/ Java /jdk1.8.0_161/ $cp jdk-8u161-linux-x64.tar.gz /usr/local/ Java/jdk1.8.0 _161 / $cd /usr/localJava / $tar ZXF jdk-8u161-linux-x64.tar.gz && rm -rf jdk-8u161-linux-x64.tar.gz Use alternatives to switch the default Java command (command --install <link> <name> <path> <priority>)
$ alternatives --install /usr/bin/java java $JAVA_18_161/bin/java 2
$ alternatives --config java

There are 4 programs which provide 'java'. Selection Command ----------------------------------------------- 1 The/usr/lib/JVM/Java - 1.7.0 - its - 1.7.0.91-2.6.2.3. El7. X86_64 / Java jre/bin / * + 2 / usr/lib/JVM/Java -- 1.8.0 comes with its 1.8.0.65-3. B17. El7. X86_64 / jre/bin/Java 3 / usr/lib/JVM/jre - 1.6.0 - its. X86_64 / bin/Java 4 /usr/local/ Java /jdk1.8.0_161/bin/ Java Enter to keep the current selection[+], ortypeJavac alternatives --install /usr/bin/jar jar /usr/local/ Java /jdk1.8.0_161/bin/jar 2 Alternatives --install /usr/bin/javac javac /usr/local/ Java /jdk1.8.0_161/bin/javac 2 alternatives --set jar /usr/local/ Java /jdk1.8.0_161/bin/jar alternatives --set javac /usr/local/ Java /jdk1.8.0_161/bin/javac Set the environment variable $vim /etc/bashrcexport JAVA_HOME=/usr/local/ Java/jdk1.8.0 _161export JRE_HOME=/usr/local/ Java/jdk1.8.0 _161 / jreexport PATH=/root/perl5/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/ Java/jdk1.8.0 _161 / bin: / usr /local/ Java /jdk1.8.0_161/jre/bin Check the Java version. $Java -version Java version"1.8.0 comes with _161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
Copy the code

The installationElasticSearch

Create user $adduser -u-s/sbin/nologin ElasticSearch Creates a directory $mkdir -p /usr/local/elasticsearch
$ mkdir -p /data/elasticsearch/my-project
$ mkdir -p /data/logs/elasticsearch/my-project/{log, PID} download ES 5.5.1 $cd /usr/local/ elasticsearch $wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.1.tar.gz unzip the file $tar ZXF Elasticsearch -5.5.1.tar.gz && rm -rf ElasticSearch -5.5.1.tar.gz $mv ElasticSearch -5.5.1 my-project-esCopy the code

The installationSearch GuardThe plug-in

Download plug-in $cd{PROJECT_NAME} $ wget Install plugin $http://search.maven.org/remotecontent?filepath=com/floragunn/search-guard-5/5.5.1-15/search-guard-5-5.5.1-15.zip Bin/elasticsearch - plugin install search - guard - 5-5.5.1-15. ZipCopy the code

Generate a certificate

Download the tool (no special path required) $gitcloneCertificate of https://github.com/floragunncom/search-guard-ssl.git to generate $cd search-guard-ssl/example-pki-scripts/
$ ./clean.sh
$ ./gen_root_ca.sh kubernetes- kubernetes-                      # COMMAND <CA_PASS> <TS_PASS>
$ ./gen_node_cert.sh k8s-es-node kubernetes- kubernetes-        # COMMAND <NODE_NAME> <KS_PASS> <CA_PASS>
$ ./gen_client_node_cert.sh sgadmin kubernetes- kubernetes-     # COMMAND <CLIENT_NAME> <KS_PASS> <CA_PASS>Copy the certificate to the config directory $cp * JKS /usr/local/elasticsearch/my-project/config/
Copy the code

configurationElasticSearch

Backup configuration $cd /usr/local/my-project/config/ $mv elasticSearch. Yml {,bak} Node Master Yml cluster.name: my-project-es node.name: 10.90.104.133 node.master:true
node.data: false
node.ingest: false

path:
    data:
    - /data/elasticsearch/my-project
    logs: /data/logs/elasticsearch/my-project/log

thread_pool.index.queue_size: 1000
thread_pool.bulk.queue_size: 1000

bootstrap.memory_lock: trueNetwork. Host: 10.90.104.133 HTTP. port: 9201 transport.tcp.port: 9301 HTTP: enabled:false
    compression: true
    cors:
        enabled: true
        allow-origin: "*"allow-headers: Authorization discovery.zen.ping.unicast.hosts: [10.90.104.133:9301,10.90. 105.133:9301,10.90. 106.133:9301] discovery. Zen. Minimum_master_nodes: 2 searchguard.ssl.transport.keystore_filepath: node-k8s-es-node-keystore.jks searchguard.ssl.transport.keystore_password: kubernetes- searchguard.ssl.transport.truststore_filepath: truststore.jks searchguard.ssl.transport.truststore_password: kubernetes- searchguard.ssl.transport.enforce_hostname_verification:false
searchguard.authcz.admin_dn:
- CN=sgadmin,OU=client,O=client,L=test,C=DE memory configuration: $vim Jvm.options... Xms20g Xmx20g ... Yml cluster.name: my-project-es node.name: $vim elasticSearch. yml cluster.name: my-project-es node.name: 10.90.107.132 node. The master:false
node.data: true
node.ingest: true

path:
    data:
    - /data/elasticsearch/my-project
    logs: /data/logs/elasticsearch/my-project/log

thread_pool.index.queue_size: 1000
thread_pool.bulk.queue_size: 1000

bootstrap.memory_lock: trueNetwork. Host: 10.90.107.132 HTTP. port: 9201 transport.tcp.port: 9301 HTTP: enabled:true
    compression: true
    cors:
        enabled: true
        allow-origin: "*"allow-headers: Authorization discovery.zen.ping.unicast.hosts: [10.90.104.133:9301,10.90. 105.133:9301,10.90. 106.133:9301] discovery. Zen. Minimum_master_nodes: 2 searchguard.ssl.transport.keystore_filepath: node-k8s-es-node-keystore.jks searchguard.ssl.transport.keystore_password: kubernetes- searchguard.ssl.transport.truststore_filepath: truststore.jks searchguard.ssl.transport.truststore_password: kubernetes- searchguard.ssl.transport.enforce_hostname_verification:false
searchguard.authcz.admin_dn:
- CN=sgadmin,OU=client,O=client,L=test,C=DE memory configuration: $vim Jvm.options... Xms63g Xmx63g ...Copy the code

User permission and password

$ cd /usr/local$sh tools/hash.sh -p ICdsy1B3j68Tr4w2$2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbeChange the default user password and add user myuser $vim sgconfig/sg_internal_users.yml# This is the internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh
admin:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  #password is: ICdsy1B3j68Tr4w2
logstash:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  #password is: ICdsy1B3j68Tr4w2
kibanaserver:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  #password is: ICdsy1B3j68Tr4w2
kibanaro:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  #password is: ICdsy1B3j68Tr4w2
  roles:
    - kibanarole
readall:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  #password is: ICdsy1B3j68Tr4w2
myuser:
  hash: $2a$12$BrsaHcr6q0SawxbJtjN46ORVcYsdLH1hQSwVTW4vbUAOwUckpWCbe
  # password is: ICdsy1B3j68Tr4w2Modify the role mapping file (assign admin permissions) $vim sgconfig/sg_roles_mapping.yml...Add only new users to this permission. Leave other Settings as default
sg_all_access:
  users:
    - admin
    - myuser
    
...

Copy the code

Add a service management script

$ vim /usr/lib/systemd/system/es-my-project.service [Unit] Description=Elasticsearch Documentation=http://www.elastic.co  Wants=network-online.target After=network-online.target [Service] Environment=ES_HOME=/usr/local/elasticsearch/my-project
Environment=CONF_DIR=/usr/local/elasticsearch/my-project/config
Environment=PID_DIR=/data/logs/elasticsearch/my-project/pid
WorkingDirectory=/usr/local/elasticsearch/my-project

User=elasticsearch
Group=elasticsearch

ExecStartPre=/usr/local/elasticsearch/my-project/bin/elasticsearch-systemd-pre-exec

ExecStart=/usr/local/elasticsearch/my-project/bin/elasticsearch \
        -p \${PID_DIR}/elasticsearch.pid \
        -Edefault.path.conf=\${CONF_DIR}

StandardOutput=journal
StandardError=inherit
LimitNOFILE=65536
LimitNPROC=2048
LimitMEMLOCK=infinity
TimeoutStopSec=0
KillSignal=SIGTERM
KillMode=process
SendSIGKILL=no
SuccessExitStatus=143

[Install]
WantedBy=multi-user.target
Copy the code

Modify the user

chown -R elasticsearch:elasticsearch /usr/local/elasticsearch
chown -R elasticsearch:elasticsearch /data/logs/elasticsearch
chown -R elasticsearch:elasticsearch /data/elasticsearch
chown -R elasticsearch:elasticsearch /usr/lib/systemd/system/es-my-project.service
Copy the code

Start the service

Add sudo systemctl daemon-reload sudo systemctlenable es-my-project.service
sudo systemctl start es-my-project
Copy the code

Sguard Initializes the password

The SGuard configuration takes effect. You only need to run the sgConfig script on any node in the cluster. The ES cluster will not be restarted during the operation. $cd /usr/local/ elasticSearch /my-project/plugins/search-guard-5/tools $./sgadmin.sh --hostname 10.90.104.133 --port 9301 \ -cd.. /sgconfig/ \ -ks /usr/local/elasticsearch/my-project/config/sgadmin-keystore.jks \
    -kspass kubernetes- \
    -ts ./usr/local/elasticsearch/my-project/config/truststore.jks  \
    -tspass kubernetes- \
    -nhnv --diagnose -cn my-project-es
Copy the code

Viewing Cluster Status

$curl -u myuser: ICdsy1B3j68Tr4w2 - XGET http://10.90.107.132:9201/_cluster/health? pretty {"cluster_name" : "my-project-es"."status" : "green"."timed_out" : false."number_of_nodes": 3."number_of_data_nodes": 3."active_primary_shards": 4."active_shards" : 9,
  "relocating_shards": 0."initializing_shards": 0."unassigned_shards": 0."delayed_unassigned_shards": 0."number_of_pending_tasks": 0."number_of_in_flight_fetch": 0."task_max_waiting_in_queue_millis": 0."active_shards_percent_as_number": 100.0}Copy the code

Kibana deployment

Dockerfile

The FROM docker.xxx.com/library/kibana-searchguard:5.5.1 ENV CLUSTER_NAME = my - project - es \ SERVER_BASEPATH = \ ELASTICSEARCH_URL="http://10.90.107.132:9201" \
    XPACK_SECURITY_ENABLED=false \
    XPACK_GRAPH_ENABLED=false \
    XPACK_ML_ENABLED=false \
    ELASTICSEARCH_USERNAME=kibanaserver \
    ELASTICSEARCH_PASSWORD=ICdsy1B3j68Tr4w2

# port is 5601
Copy the code

Build based on internal warehouse images

Build image + build container

$ docker build -t my-project-kibana .
$ docker run -d my-project-kibana
Copy the code

Login kibana

Initialization Settings are performed

Replacement of the certificate

Copy the newly generated certificate file *. JKS to the original cluster config and restart the ES cluster and Search Guard.