Don’t know when or what the configuration file was last modified? Republish the project after changing the configuration file or manually trigger a service restart? Is the configuration file incorrect unexpectedly affecting online deployment? Are you struggling with these problems? 50+ online projects, hundreds of + profiles, we are often abused by these profiles, it is time to make a change! This article will take you through these problems and make them easy to operate over your coffee

Configuration center selection

The principle of selection: simple, easy to fall to the ground, no platform, no language, as little as possible dependence.

Compared with Disconf, Apollo and other programs, the final choice of Etcd+Confd program, basically in line with the above principles, and Etcd we deployed Kubernetes has been used, is a familiar road.

Configure the central architecture diagram

  • The configuration center adopts C/S mode. Etcd is used as the server to store data, and Confd is used as the client to fetch data from Etcd for update
  • To facilitate management, the WebUI is actually a web user interface (WebUI) of the Etcd service. It interacts with the Etcd service to access data
  • Confd pulls data from the Etcd cluster according to the configuration file, and then generates the final configuration file according to the template file and fills the data in a fixed location according to the preset format
  • After the configuration file is generated, it can be used togethercheck_cmdandreload_cmdCommand to check and reload the configuration file

Configuration Center Deployment

Etcd cluster

  • System environment

    • System: Debian 8
    • Etcd: v3.3.9

  • Server address

    • 192.168.107.101
    • 192.168.107.102
    • 192.168.107.103
All servers need to execute the following commands to install etCD and create directories

1. Download the ETCD installation package and decompress it

Wget # https://github.com/coreos/etcd/releases/download/v3.3.9/etcd-v3.3.9-linux-amd64.tar.gz
# tar - ZXVF etcd v3.3.9 - Linux - amd64. Tar. Gz
Copy the code

2. Copy the program to the /usr/bin directory for easy execution. Etcd is written for go and can be run directly.

# mv etcd - v3.3.9 - Linux - amd64 / etcd * / usr/bin /
Copy the code

3. Create the etcd configuration file directory /etc/etcd and the data storage directory /home/data/etcd

# mkdir /etc/etcd /home/data/etcd
Copy the code
The etcd configuration files of the three nodes are as follows

Node1 configuration

# cat /etc/etcd/etcd.conf 
name: 'node1'Data - dir: / home/data/etcd listen - peer - urls: http://192.168.107.101:2380 listen - the client - urls: http://192.168.107.101:2379, http://127.0.0.1:2379 initial cluster - the state:'new'
initial-cluster-token: 'etcd-cluster-conf'Initial advertise - the client - urls: http://192.168.107.101:2379 - advertise - peer - urls: http://192.168.107.101:2380 initial - cluster: Node1 = http://192.168.107.101:2380, 2 = http://192.168.107.102:2380, node3 = http://192.168.107.103:2380Copy the code

2 configuration

# cat /etc/etcd/etcd.conf 
name: 'node2'Data - dir: / home/data/etcd listen - peer - urls: http://192.168.107.102:2380 listen - the client - urls: http://192.168.107.102:2379, http://127.0.0.1:2379 initial cluster - the state:'new'
initial-cluster-token: 'etcd-cluster-conf'Initial advertise - the client - urls: http://192.168.107.102:2379 - advertise - peer - urls: http://192.168.107.102:2380 initial - cluster: Node1 = http://192.168.107.101:2380, 2 = http://192.168.107.102:2380, node3 = http://192.168.107.103:2380Copy the code

Node3 configuration

# cat /etc/etcd/etcd.conf 
name: 'node3'Data - dir: / home/data/etcd listen - peer - urls: http://192.168.107.103:2380 listen - the client - urls: http://192.168.107.103:2379, http://127.0.0.1:2379 initial cluster - the state:'new'
initial-cluster-token: 'etcd-cluster-conf'Initial advertise - the client - urls: http://192.168.107.103:2379 - advertise - peer - urls: http://192.168.107.103:2380 initial - cluster: Node1 = http://192.168.107.101:2380, 2 = http://192.168.107.102:2380, node3 = http://192.168.107.103:2380Copy the code
Start each node after the configuration is complete

You need to run it in the background. Screen is recommended

# /usr/bin/etcd --config-file /etc/etcd/etcd.conf 
Copy the code

After all three nodes are started, run the etcdctl member list command to view the cluster list and check the cluster status

# etcdctl member list732 ca490026f580d: name = node3 peerURLs isLeader = = http://192.168.107.103:2380 clientURLs = http://192.168.107.103:2379falseBc16d35c3ad1c5ee: name = 2 peerURLs isLeader = = http://192.168.107.102:2380 clientURLs = http://192.168.107.102:2379true
f7a043d3b65cdA4: name = node1 peerURLs isLeader = = http://192.168.107.101:2380 clientURLs = http://192.168.107.101:2379false
Copy the code

Confd

1. Download confd and save it to the /usr/bin/ directory for easy use

Wget # https://github.com/kelseyhightower/confd/releases/download/v0.16.0/confd-0.16.0-linux-amd64
# mv confd - 0.16.0 - Linux - amd64 / usr/bin/confd
# chmod +x /usr/bin/confd
Copy the code

2. Create the ConfD configuration file directory

# mkdir /etc/confd/{conf.d,templates}
Copy the code

3. Create a new resource file. The end of the.toml file has become a fixed format

# cat /etc/confd/conf.d/nginx.conf.toml 
[template]
src = "nginx.conf.tmpl"
dest = "/tmp/nginx.conf"

keys = [
   "/conf/project/env/nginx/nginx.conf",
]

check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/sbin/service nginx reload"
Copy the code

Nginx configuration file (nginx configuration file)

  • SRC: Specifies the location of the template file, which is the location of the nginx configuration file template TMPL

  • Dest: Specifies the absolute path to the configuration file to be generated or updated

  • Keys: the key used in the template file, which is the key used in the etCD corresponding to the project configuration file

  • Check_cmd: executes the check command after updating the configuration file. Here we check whether the nginx configuration file has syntax errors

  • Reload_cmd: After the check passes, you can run the command configured here. If the check in the previous step is ok, the reload command will be executed to reload the configuration file

  • Prefix: configure the prefix of the key. For example, if the key starts with /conf, add the prefix=”/conf” and omit /conf in the following keys

  • Owner: Configures the user that generates the configuration file

  • Mode: configures the permission to generate configuration files

4. Create a template file

# cat /etc/confd/templates/nginx.conf.tmpl 
{{getv "/conf/project/env/nginx/nginx.conf"}}
Copy the code
  • Confd template syntax there are many, not to repeat here, can be found on the official website
  • We store the entire configuration file as a value in etCD, so we only need a geTV command to get the value and fill it into the target file

Alignment test

With the ETCD cluster and ConfD service deployed, it’s time to test if they work together properly

1. Create a KV value on the Etcd server

# etcdctl set /conf/project/env/nginx/nginx.conf 'user www-data;> worker_processes 4; > > pid /var/run/nginx.pid; > error_log /home/logs/nginx/error.log warn; > > events { > use epoll; > worker_connections 51200; > } > > http { > default_type application/octet-stream; > > server { > listen 80; > server_name domain.com; > > root /home/project/webroot; > index index.shtml index.html; > >}}'
Copy the code
# check the contents of set key
# etcdctl get /conf/project/env/nginx/nginx.confuser www-data; worker_processes 4; pid /var/run/nginx.pid; error_log /home/logs/nginx/error.log warn; events { use epoll; worker_connections 51200; } http { default_type application/octet-stream; server { listen 80; server_name domain.com; root /home/project/webroot; index index.shtml index.html; }}Copy the code
  • Etcd API is divided into V2 and V3 versions. There is a big difference between the two versions. V3 has been optimized a lot, but we use V2 version for compatibility
  • The default version is V2. You can use environment variablesexport ETCDCTL_API=3To switch to v3, v2 passedetcdctl -vYou can view the API version. V3 passesetcdctl versionViewing the API Version

2. Start confd

# confd - watch - backend etcd - node = http://192.168.107.101:2379 - node = http://192.168.107.102:2379 - node = http://192.168.107.103:2379
2018-08-23T13:46:13+08:00 onlinegame.i.nease.net confd[17084]: INFO Backend set to etcd
2018-08-23T13:46:13+08:00 onlinegame.i.nease.net confd[17084]: INFO Starting confd
2018-08-23T13:46:13+08:00 onlinegame.i.nease.net confd[17084]: INFO Backend source(s) setTo http://192.168.107.101:2379, http://192.168.107.102:2379, http://192.168.107.103:2379 the T13:2018-08-23 + spake 08:00 onlinegame.i.nease.net confd [17084] : INFO Target config /tmp/nginx.conf out of sync 2018-08-23T13:46:13+08:00 onlinegame.i.nease.net confd[17084]: INFO Target config /tmp/nginx.conf has been updatedCopy the code

Parameter Description

  • – Watch: Enables the watch mode to monitor changes in the ETCD configuration center file. If there are changes, the etCD will be updated immediately. Without this option, the etCD client will not be updated

  • – Backend: indicates the backend type. The backend type supports etcd, ZooKeeper, Consul, Vault, redis, file, and Rancher. Confd also has some independent configurations for unavailable backend types

  • -node: address of the etcd node. If there are multiple nodes, write multiple -nodes. We have a cluster of three nodes, so write ‘-node’ three times.

  • -onetime: replaces the -watch parameter, which means run once and exit. You can use this parameter if you want to update the configuration file only once rather than in real time

  • -interval: specifies the interval for obtaining data from backend every seconds. This parameter is used to reduce the pressure on the ETcd server and automatically update the configuration file on the client

Conf file is correctly synchronized and updated. Check the/TMP /nginx.conf file to make sure the content is correct

WebUI Kerrigan

Can’t all configuration file updates be done from the command line? To facilitate management, I spent three days (literally three days) writing a web user interface (WebUI) named Kerrigan, which can implement a directory tree, view configuration online, modify configuration, view configuration update history and other practical functions

Configuration page, through which you can configure etCD connection information

Home page, list of projects on the left (project information synchronized with CMDB)

After clicking the item list, take out the directory structure in etCD according to the corresponding rules and present it as a tree

Click the configuration file. The current configuration file is displayed on the right

Click the edit button to edit the configuration, just like creating a new page, except that editing does not allow changing the path

Click the “History” button to jump to the history page of the configuration file, which shows the history of all changes to the configuration file

Write in the last

  1. This interface is so ugly! No way, front-end and back-end testing and wiring are all done by me, no design cells, look at it this way, and the most important thing is not function
  2. Why not use K8S configMap? We originally wanted to use K8S configMap as the configuration center, but not all projects run in K8S and changes to configMap require a container restart to take effect, so we didn’t use it
  3. Can anyone modify etCD? It doesn’t feel safe? In fact, we use the account password authentication, and only on the Intranet, limit IP, a bit safer, another solution is etCD SSL, but the client side to put the certificate is more troublesome
  4. How do I confirm that the Client configuration file is updated successfully? If you start the command once, you can determine whether the command is executed properly after you start it. If you start the command in Watch mode or interval, then. I just have to check. I don’t have a good way

If you find the article helpful, please share it and let more friends see it. If you’re not enjoying your reading, read the following:

  • DevOps operation automation tool system platform
  • Fortress WebSSH full function implementation tutorial
  • The principle and application of ELK system in production environment