In my previous series of articles:
- How to deploy Elastic Stack-Overview using Ansible
- Deploying Elastic Stack-ElasticSearch with Ansible
- Deploying Elastic Stack-Kibana with Ansible
We’ve learned how to deploy our Elasticsearch and Kibana using Ansible. I want you to complete the three exercises above before proceeding to the following exercises. In a real production environment, we would use Elasticsearch security. Configure passwords for built-in users. If you are not familiar with how to configure security for Elasticsearch and Kibana, please read my previous article:
- Elasticsearch: Set Elastic account security
- Configure security and create built-in user accounts using ElasticSearch -keystore
Above, I showed you two ways to configure security. The first approach is best suited to scenarios with a small number of servers, while the second approach is better suited to scripts or tools like Ansible for deployment. In today’s tutorial, I’ll use the second method for deployment.
You can find the source at github.com/liu-xiao-gu…
Configure security for the Elastic Stack
Defining built-in users
Below, we will provide a step-by-step guide to configuring security for Elasticsearch and Kibana. If you’re not familiar with the built-in users of Elastic Stack, please read our official documentation “Built-in Users.” To be able to create these built-in users, in the previous file schema, we created a new file directory vars:
$ pwd
/Users/liuxg/ansible/elasticsearch
$ mkdir vars
$ ls
inventory playbooks roles vars
Copy the codeIn the vars directory, we create a new file named credentials.yml:
vars/credentials.yml
# Bootstrap credetials
es_api_basic_auth_username: elastic
es_api_basic_auth_password: s3cr3t
# A built-in superuser.
elastic_username: elastic
elastic_password: goodwitch
# The user Kibana uses to connect and communicate with Elasticsearch.
kibana_username: kibana
kibana_password: badsanta
# The user Logstash uses when storing monitoring information in Elasticsearch.
logstash_system_username: logstash_system
logstash_system_password: dragonprince
# The user the Beats use when storing monitoring information in Elasticsearch.
beats_system_username: beats_system
beats_system_password: avatar
# The user the APM server uses when storing monitoring information in Elasticsearch.
apm_system_username: apm_system
apm_system_password: mashaandthebear
# The user Metricbeat uses when collecting and storing monitoring information in Elasticsearch. It has the remote_monitoring_agent and remote_monitoring_collector built-in roles.
remote_monitoring_user_username: remote_monitoring_user
remote_monitoring_user_password: gossipgirl
Copy the code
Here we define the user names and passwords of some of the built-in users of the Elastic Stack. These variables are used by all software in the Elastic Stack, so they are independent of any roles.
At the same time, we define some common variables. In the Vars directory, we will create a file called main.yml:
vars/main.yml
Elastic_host: 192.168.0.4 ELAStic_port: 9200 KiBANA_host: 192.168.0.4 kiBANA_port: 5601 ELAStic_protocol: http filebeat_http_port: 5067 auditbeat_http_port: 5068 heartbeat_http_port: 5069 packetbeat_http_port: 5070 web_server_1:192.168.0.4Copy the code
Due to the introduction of the above two files, we need to modify the deploy-demo.yml file we defined earlier:
playbooks/deploy-demo.yml
--- # This playbook will deploy webserver - hosts: all become: yes roles: - .. /roles/add-elastic-repo # This playbook will deploy ELK stack - hosts: elk become: yes vars_files: - .. /vars/credentials.yml - .. /vars/main.yml roles: - .. /roles/elasticsearch - .. /roles/kibanaCopy the code
Start the safety
We find the templates/elasticsearch. Yml file, and add the following items:
xpack.security.enabled: true
Copy the code[elasticSearch.yml]
templates/elasticsearch.yml
# ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: {{ cluster_name }} # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: {{ node_name }} # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: {{ path_data }} # # Path to log files: # path.logs: {{ path_logs }} # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: {{ network_host }} # # Set a custom port for HTTP: # http.port: {{ http_port }} # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # # Discovery. Seed_hosts: ["host1", "host2"] # # Bootstrap the cluster using an initial set of master-eligible nodes: # #cluster.initial_master_nodes: ["node-1", "node-2"] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true discovery.type: {{ discovery_type }} # -------------------------------- X-pack security Configuration ---------------- xpack.security.enabled: trueCopy the code
To facilitate our design, we’ll modify the defaults/main.yml file under elasticSearch and define the following variables:
elasticsearch/defaults/main.yml
---
# defaults file for elasticsearch
cluster_name: demo-elk
node_name: elk-1
path_data: /var/lib/elasticsearch
path_logs: /var/log/elasticsearch
network_host: 0.0.0.0
http_port: 9200
discovery_type: single-node
# X-pack configuration
xpack_security_enabled: true
es_home: /usr/share/elasticsearch
es_conf_dir: /etc/elasticsearch
es_group: elasticsearch
es_owner: root
es_mode: 0660
es_api_protocol: "http"
es_api_host: "localhost"
es_api_port: 9200
es_api_uri: "{{ es_api_protocol }}://{{ es_api_host }}:{{ es_api_port }}"
es_security_api: _security
es_validate_certs: no
es_force_basic_auth: yes
Copy the codeOn top of that, I added the x-Pack Configuration section. If you are not familiar with variable definitions in this section, please refer to my previous article “Configuring security and creating built-in user accounts using ElasticSearch -keystore”. Here es_HOME defines the directory where Elasticsearch is installed, and es_conf_dir defines the directory where the configuration file is installed. Es_api_url specifies how to configure the built-in user interface using the REST API interface.
Create a folder security under tasks under ElasticSearch role and create two keystore.yml and setup-built-in.
$PWD/Users/liuxg ansible/elasticsearch/roles/elasticsearch tree -l/tasks $2. ├ ─ ─ the main, yml └ ─ ─ security ├ ─ ─ Keystore. Yml └ ─ ─ the setup - built - in - the user. YmlCopy the codeLet’s start with keystore.yml:
keystore.yml
---
# ----------- Create KeyStore ----------
- name: create elasticsearch keystore
become: yes
command: >
{{ es_home }}/bin/elasticsearch-keystore create
args:
creates: "{{ es_conf_dir }}/elasticsearch.keystore"
environment:
ES_PATH_CONF: "{{ es_conf_dir }}"
- name: Set elasticsearch keystore permissions
become: yes
file: state=file path={{ es_conf_dir }}/elasticsearch.keystore owner={{ es_owner }} group={{ es_group }} mode={{ es_mode }}
- name: Check if elasticsearch keystore is setup
become: yes
command: >
{{ es_home }}/bin/elasticsearch-keystore list
register: list_keystore
changed_when: False
environment:
ES_PATH_CONF: " {{ es_conf_dir }} "
check_mode: no
- name: Create bootstrap password for elastic user
become: yes
shell: echo "{{ es_api_basic_auth_password }}" | {{ es_home }}/bin/elasticsearch-keystore add -x 'bootstrap.password'
when:
- es_api_basic_auth_username is defined and list_keystore is defined and es_api_basic_auth_username == 'elastic' and 'bootstrap.password' not in list_keystore.stdout_lines
environment:
ES_PATH_CONF: " {{ es_conf_dir }} "
no_log: true
# ------------including builtin user setup --------------
- name: Including builtin user setup
include: setup-built-in-user.yml
Copy the codeIn the previous sections, I covered “Configuring security and creating a built-in user account with ElasticSearch -keystore”. If you don’t know much, I suggest you read the article first.
In the keystore.yml file above, we also have include: setup-built-in-user.yml. Setup-built-in -user. Yml is defined as follows:
setup-built-in-user.yml
---
- name: Restarting Elasticsearch
service:
name: elasticsearch
state: restarted
#------------------------------- Setup Built-in User passwords-----------
- name: Update elastic user password
uri:
url: "{{ es_api_uri }}/{{ es_security_api }}/user/{{es_api_basic_auth_username}}/_password"
method: POST
body_format: json
body: "{ \"password\":\"{{ elastic_password }}\" }"
status_code: 200
user: "{{es_api_basic_auth_username}}"
password: "{{es_api_basic_auth_password}}"
force_basic_auth: "{{ es_force_basic_auth }}"
validate_certs: "{{ es_validate_certs }}"
- name: Update Kibana user password
uri:
url: "{{ es_api_uri }}/{{ es_security_api }}/user/{{ kibana_username }}/_password"
method: POST
body_format: json
body: "{ \"password\":\"{{ kibana_password }}\" }"
status_code: 200
user: "{{es_api_basic_auth_username}}"
password: "{{ elastic_password }}"
force_basic_auth: "{{ es_force_basic_auth }}"
validate_certs: "{{ es_validate_certs }}"
- name: Update logstash_system user password
uri:
url: "{{ es_api_uri }}/{{ es_security_api }}/user/{{ logstash_system_username }}/_password"
method: POST
body_format: json
body: "{ \"password\":\"{{ logstash_system_password }}\" }"
status_code: 200
user: "{{es_api_basic_auth_username}}"
password: "{{ elastic_password }}"
force_basic_auth: "{{ es_force_basic_auth }}"
validate_certs: "{{ es_validate_certs }}"
- name: Update beats_system user password
uri:
url: "{{ es_api_uri }}/{{ es_security_api }}/user/{{ beats_system_username }}/_password"
method: POST
body_format: json
body: "{ \"password\":\"{{ beats_system_password }}\" }"
status_code: 200
user: "{{es_api_basic_auth_username}}"
password: "{{ elastic_password }}"
force_basic_auth: "{{ es_force_basic_auth }}"
validate_certs: "{{ es_validate_certs }}"
- name: Update APM system user password
uri:
url: "{{ es_api_uri }}/{{ es_security_api }}/user/{{ apm_system_username }}/_password"
method: POST
body_format: json
body: "{ \"password\":\"{{ apm_system_password }}\" }"
status_code: 200
user: "{{es_api_basic_auth_username}}"
password: "{{ elastic_password }}"
force_basic_auth: "{{ es_force_basic_auth }}"
validate_certs: "{{ es_validate_certs }}"
- name: Update remote_monitoring_user password
uri:
url: "{{ es_api_uri }}/{{ es_security_api }}/user/{{ remote_monitoring_user_username }}/_password"
method: POST
body_format: json
body: "{ \"password\":\"{{ remote_monitoring_user_password }}\" }"
status_code: 200
user: "{{es_api_basic_auth_username}}"
password: "{{ elastic_password }}"
force_basic_auth: "{{ es_force_basic_auth }}"
validate_certs: "{{ es_validate_certs }}"
Copy the codeThis part is also very straightforward. It uses the REST API interface directly. Their use was shown in my previous article “Configuring security and creating built-in user accounts with ElasticSearch -keystore”.
We need to further modify the main. Yml file in the Tasks directory so that the keystore.yml and setup-built-in.
elasticsearch/tasks/main.yml
---
# tasks file for elasticsearch
# Installing Elasticsearch
- name: Installing Elasticsearch
apt:
name: elasticsearch
# Replce default elasticsearch.yml
- name: Replace default elasticsearch.yml
template:
src: elasticsearch.yml
dest: /etc/elasticsearch/elasticsearch.yml
# Start Elasticsearch service
- name:
service:
name: elasticsearch
state: started
enabled: yes
# Xpack security configuration
- name: Including xpack security setup configuration
include: security/keystore.yml
Copy the code
Ok, so far we have completed the requirements for Elasticsearch configuration. We need to make corresponding changes to Kibana as well. We need to add the following section to the kibana.yml file:
elasticsearch.username: "{{ kibana_username }}"
elasticsearch.password: "{{ kibana_password }}"
Copy the codekibana/templates/kibana.yml
# Kibana is served by a back end server. This setting specifies the port to use. server.port: {{ server_port }} # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. # The default is 'localhost', which usually means remote machines will not be able to connect. # To allow connections from remote users, set this parameter to a non-loopback address. server.host: "{{ server_host }}" # Enables you to specify a path to mount Kibana at if you are running behind a proxy. # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath # from requests it receives, and to prevent a deprecation warning at startup. # This setting cannot end in a slash. #server.basePath: "" # Specifies whether Kibana should rewrite requests that are prefixed with # `server.basePath` or require that they Are rewritten by your reverse proxy. # This setting was effectively always' false 'before Kibana 6.3 and will # default To 'true' starting in Kibana 7.0. #server. RewriteBasePath: false # The maximum payload size in bytes for incoming server requests. #server.maxPayloadBytes: 1048576 # The Kibana server's name. This is used for display purposes. server.name: "{{ server_name }}" # The URLs of the Elasticsearch instances to use for all your queries. elasticsearch.hosts: ["{{ elasticsearch_host }}"] # Kibana uses an index in Elasticsearch to store saved searches, visualizations and # dashboards. Kibana creates a new index if the index doesn't already exist. #kibana.index: ".kibana" # The default application to load. #kibana.defaultAppId: "home" # If your Elasticsearch is protected with basic authentication, these settings provide # the username and password that the Kibana server uses to perform maintenance on the Kibana # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which # is proxied through the Kibana server. elasticsearch.username: "{{ kibana_username }}" elasticsearch.password: "{{ kibana_password }}" # Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively. # These settings enable SSL for outgoing requests from the Kibana server to the browser. #server.ssl.enabled: false #server.ssl.certificate: /path/to/your/server.crt #server.ssl.key: /path/to/your/server.key # Optional settings that provide the paths to the PEM-format SSL certificate and key files. # These files are used to verify the identity of Kibana to Elasticsearch and are required when # xpack.security.http.ssl.client_authentication in Elasticsearch is set to required. #elasticsearch.ssl.certificate: /path/to/your/client.crt #elasticsearch.ssl.key: /path/to/your/client.key # Optional setting that enables you to specify a path to the PEM file for the certificate # authority for your Elasticsearch instance. #elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ] # To disregard the validity of SSL certificates, change this setting's value to 'none'. #elasticsearch.ssl.verificationMode: full # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of # the elasticsearch.requestTimeout setting. #elasticsearch.pingTimeout: 1500 # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value # must be a positive integer. #elasticsearch.requestTimeout: 30000 # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side # headers, set this value to [] (an empty list). #elasticsearch.requestHeadersWhitelist: [ authorization ] # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration. #elasticsearch.customHeaders: {} # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. #elasticsearch.shardTimeout: 30000 # Logs queries sent to Elasticsearch. Requires logging.verbose set to true. #elasticsearch.logQueries: false # Specifies the path where Kibana creates the process ID file. #pid.file: /var/run/kibana.pid # Enables you to specify a file where Kibana stores log output. #logging.dest: stdout # Set the value of this setting to true to suppress all logging output. #logging.silent: false # Set the value of this setting to true to suppress all logging output other than error messages. #logging.quiet: false # Set the value of this setting to true to log all events, including system usage information # and all requests. #logging.verbose: false # Set the interval in milliseconds to sample system and process performance # metrics. Minimum is 100ms. Defaults to 5000. #ops.interval: 5000 # Specifies locale to be used for all localizable strings, dates and number formats. # Supported languages are the following: English - en , by default , Chinese - zh-CN . #i18n.locale: "en"Copy the codeAfter this configuration, we were able to log in using our Kibana username and password.
All right, we’re all set. Again, we use the following command to deploy:
ansible-playbook -K -i inventory/hosts.yml playbooks/deploy-demo.yml
Copy the codeThe result of running the command above is:
$ pwd /Users/liuxg/ansible/elasticsearch $ ansible-playbook -K -i inventory/hosts.yml playbooks/deploy-demo.yml BECOME password: PLAY [all] ********************************************************************* TASK [Gathering Facts] * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * ok: [192.168.0.4] TASK [.. / roles/add - elastic - repo: Add ElasticSearch public signing key ******** OK: [192.168.0.4] TASK [../roles/ add-elastice-repo: Install apt - transport - HTTPS] * * * * * * * * * * * * * * * * * ok: [192.168.0.4] TASK [.. / roles/add - elastic - repo: Add ElasticSearch repo definitions] ********** OK: [192.168.0.4] TASK [../roles/add-elastic-repo: system update] ******************************* changed: [192.168.0.4] PLAY [elk] * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * TASK [Gathering Facts] * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * ok: [192.168.0.4] TASK [.. / roles/elasticsearch: Installing Elasticsearch] * * * * * * * * * * * * * * * * * * * * * * * ok: [192.168.0.4] TASK [.. / roles/Elasticsearch: Replace Default ElasticSearch. Yml] ************** changed: [192.168.0.4] TASK [../roles/ ElasticSearch: The service] * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * changed: [192.168.0.4] TASK [.. / roles/elasticsearch: The create elasticsearch keystore] * * * * * * * * * * * * * * * * * * ok: [192.168.0.4] TASK [.. / roles/elasticsearch: Set elasticSearch keystore permissions] ********* OK: [192.168.0.4] TASK [../roles/ elasticSearch: Check if elasticSearch keystore is setup] ******* OK: [192.168.0.4] TASK [../roles/ Elasticsearch: Create bootstrap password for elastic user] ***** changed: [192.168.0.4] TASK [../roles/ ElasticSearch: Restarting Elasticsearch] * * * * * * * * * * * * * * * * * * * * * * * changed: [192.168.0.4] TASK [.. / roles/Elasticsearch: The Update elastic user password] * * * * * * * * * * * * * * * * * * * ok: [192.168.0.4] TASK [.. / roles/elasticsearch: The Update Kibana user password] * * * * * * * * * * * * * * * * * * * * ok: [192.168.0.4] TASK [.. / roles/elasticsearch: Update logstash_system user password] *********** OK: [192.168.0.4] TASK [../roles/ elasticSearch: Update beats_system user password] ************** OK: [192.168.0.4] TASK [../roles/ ElasticSearch: Update APM system user password] **************** OK: [192.168.0.4] TASK [../roles/ ElasticSearch: Update remote_monitoring_user password] ******** OK: [192.168.0.4] TASK [../roles/kibana: Installing Kibana with apt] * * * * * * * * * * * * * * * * * * * * * * * * * * * * ok: [192.168.0.4] TASK [.. / roles/Kibana: Replacing Default Kibana. Yml with updated file] ******** changed: [192.168.0.4] TASK [../roles/ Kibana: Starting Kibana] *************************************** ok: [192.168.0.4] PLAY RECAP * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 192.168.0.4: ok=23 changed=6 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0Copy the codeIn real practice, if you want to practice more than once, follow these steps:
- Delete ElasticSearch install: sudo DPKG -r ElasticSearch
- Sudo apt-get –purge elasticSearch
- Delete directories rm -rf /etc/elasticSearch and /var/lib/elasticsearch
We can also delete Kibana in the same way:
- Delete Kibana: sudo DPKG -r ElasticSearch
- Sudo apt-get — Purge kibana
Then do the exercise again.
We run the following command on an Ubuntu OS machine:
curl -u elastic:goodwitch http://localhost:9200
Copy the codeThe command above shows:
We accessed Kibana on a MacOS machine:
Click on Elasticsearch’s super user elastic and log in:
From there, we successfully logged on to Kibana. This shows that our configuration is successful.
You can find the source at github.com/liu-xiao-gu…