0 Album Overview

Etcd is an important foundation component in the cloud native architecture and is incubated and hosted by CNCF. Etcd can be used not only as service registration and discovery, but also as middleware for key-value storage in microservices and Kubernates clusters.

“Thoroughly Understand ETCD series of articles” will be from the basic function of ETCD practice, API interface, implementation principle, source code analysis, as well as the implementation of the pit experience and other aspects of the specific introduction of ETCD. It is expected that there will be about 20 articles, I will continue to update every week, welcome to pay attention to.

1 EtCD cluster deployment

In a production environment, etCD is normally deployed in a cluster to ensure high availability of the entire cluster, avoiding a single point of failure. This section describes how to deploy an ETCD cluster. There are three mechanisms for bootstrapping etCD clusters:

  • static
  • Etcd dynamic discovery
  • DNS found

Statically starting an ETCD cluster requires that each member know about the other member in the cluster. In many cases, the IP addresses of cluster members may not be known in advance. In these cases, the ETCD cluster can be booted with the help of the discovery service.

Each of these approaches will be described below.

2 Start the ETCD cluster statically

Single machine installation

If you want to practice setting up etCD clusters on a single machine, you can use the Goreman tool.

Goreman is a multi-process management tool written in the Go language. It is a rewrite of the widely used Ruby Version of Foreman (foreman also implemented a Go version, forego, which is not as useful as Goreman).

We need to confirm the Go installation environment and execute directly:

go get github.com/mattn/goreman
Copy the code

The compiled file is stored in $GOPATH/bin. The $GOPATH/bin directory has been added to the system $PATH, so we can easily execute the goreman command. Here is the Procfile script. We start three etcds as follows:

HostName ip Client interaction port Peer communication port
infra1 127.0.0.1 12379 12380
infra2 127.0.0.1 22379 22380
infra3 127.0.0.1 32379 32380

The Procfile script is as follows:

etcd1: etcd --name infra1 --listen-client-urls http://127.0.0.1:12379 --advertise-client-urls http://127.0.0.1:12379 --listen-peer-urls http://127.0.0.1:12380 --initial-advertise-peer-urls http://127.0.0.1:12380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1 = http://127.0.0.1:12380, infra2 = http://127.0.0.1:22380, infra3 = http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr
etcd2: etcd --name infra2 --listen-client-urls http://127.0.0.1:22379 --advertise-client-urls http://127.0.0.1:22379 --listen-peer-urls http://127.0.0.1:22380 --initial-advertise-peer-urls http://127.0.0.1:22380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1 = http://127.0.0.1:12380, infra2 = http://127.0.0.1:22380, infra3 = http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr
etcd3: etcd --name infra3 --listen-client-urls http://127.0.0.1:32379 --advertise-client-urls http://127.0.0.1:32379 --listen-peer-urls http://127.0.0.1:32380 --initial-advertise-peer-urls http://127.0.0.1:32380 --initial-cluster-token etcd-cluster-1 --initial-cluster 'infra1 = http://127.0.0.1:12380, infra2 = http://127.0.0.1:22380, infra3 = http://127.0.0.1:32380' --initial-cluster-state new --enable-pprof --logger=zap --log-outputs=stderr
Copy the code

Description of configuration items:

  • –name: the name of the node in the etcd cluster, which can be distinguished and not repeated
  • Listen-peer-urls: listening urls that are used to communicate between nodes, multiple urls that will be used for data interaction within the cluster (such as elections, data synchronization, etc.)
  • — Initial-advertise-peer-urls: Recommended URL for communication between nodes, which will communicate with each other.
  • Listen-client-urls: Listening urls used for client communication, also listening for more than one.
  • –advertise-client-urls: Recommended client communication URL for etCD agents or ETCD members to communicate with ETCD nodes.
  • – initial cluster – token: Etcd-cluster-1 is the token value of a node. After the token value is set, the cluster will generate a unique ID and a unique ID for each node. When the same configuration file is used to start another cluster, the ETcd clusters will not affect each other as long as the token value is different.
  • Initial-cluster: a collection of all initial-advertise-peer-urls in a cluster.
  • — Initial-cluster-state: new: indicates the flag of a new cluster

Note In the preceding script, you need to configure the etcd command based on the actual local installation address. Let’s start the ETCD cluster.

goreman -f /opt/procfile start
Copy the code

Use the preceding command to start the ETCD cluster and view the members of the cluster.

$etcdctl --endpoints=http://localhost:22379 member list 8211f1d0f64f3269, started, infra1, http://127.0.0.1:12380, http://127.0.0.1:12379,false
91bc3c398fb3c146, started, infra2, http://127.0.0.1:22380, http://127.0.0.1:22379, falseFd422379fda50e48, started, infra3, http://127.0.0.1:32380, http://127.0.0.1:32379,false
Copy the code

We successfully set up the pseudo cluster in a single machine. It should be noted that when the cluster is started, we specify the members of the cluster statically. In the actual environment, the IP addresses of the cluster members may not be known in advance. This is where dynamic discovery is needed.

Docker starts the cluster

Etcd uses GCR. IO/ETCD-development/ETCD as the main accelerator of the container and Quay. IO/Coreos/ETCD as the auxiliary accelerator. Unfortunately, we can’t visit these two accelerators. If you can’t download them, you can use the address provided by the author:

Docker pull bitnami/etcd: 3.4.7Copy the code

Then re-tag the pulled image:

Docker image tag bitnami/etcd: 3.4.7 quay. IO/coreos/etcd: 3.4.7Copy the code

After the image is set up, we start the etCD cluster of 3 nodes with the following script:

REGISTRY=quay.io/coreos/etcd

# For each machineETCD_VERSION=3.4.7 TOKEN=my-etcd-token CLUSTER_STATE=new NAME_1=etcd-node-0 NAME_2= ETcd-node-1 NAME_3=etcd-node-2 HOST_1= 192.168.202.128 HOST_2= 192.168.202.129 HOST_3= 192.168.202.130 CLUSTER=${NAME_1}=http://${HOST_1}: 2380,${NAME_2}=http://${HOST_2}: 2380,${NAME_3}=http://${HOST_3}:2380
DATA_DIR=/var/lib/etcd

# For node 1
THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
docker run \
  -p 2379:2379 \
  -p 2380:2380 \
  --volume=${DATA_DIR}:/etcd-data \
  --name etcd ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For node 2
THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
docker run \
  -p 2379:2379 \
  -p 2380:2380 \
  --volume=${DATA_DIR}:/etcd-data \
  --name etcd ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

# For node 3
THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
docker run \
  -p 2379:2379 \
  -p 2380:2380 \
  --volume=${DATA_DIR}:/etcd-data \
  --name etcd ${REGISTRY}:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://0.0.0.0:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://0.0.0.0:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}
Copy the code

Note that the above script is deployed on three machines, each of which executes the corresponding script. You can specify the API version at runtime:

docker exec etcd /bin/sh -c "export ETCDCTL_API=3 && /usr/local/bin/etcdctl put foo bar"
Copy the code

Docker installation is relatively simple, and readers can customize some configurations as required.

3 Dynamically discover and start the ETCD cluster

As mentioned earlier, in a real world environment, the IP addresses of cluster members may not be known in advance. In this case, you need to boot the ETCD cluster using automatic discovery rather than specifying a static configuration, a process known as discovery. We start three ETcds, which are as follows:

HostName ip Client interaction port Peer communication port
etcd1 192.168.202.128 2379 2380
etcd2 192.168.202.129 2379 2380
etcd3 192.168.202.130 2379 2380

Principle of protocol

Discovery Service Protocol helps new ETCD members discover all other members during cluster boot using a shared URL.

The protocol uses the new discovery token to boot a unique ETCD cluster. A discovery token can only represent one ETCD cluster. Once the discovery protocol on this token is started, even if it fails midway, it cannot be used to boot another ETCD cluster.

Tip: Discovery Service Protocol is only used during cluster boot and cannot be used for runtime reconfiguration or cluster monitoring.

Discovery Protocol uses an internal ETCD cluster to coordinate the boot program for a new cluster. First, all new members interact with the discovery service and help generate the expected list of members. Each new member then boots its server with this list, which performs the same function as the — initial-Cluster flag, which sets the membership information for all clusters.

Obtain the discovery token

Generate a unique token that will identify the new cluster. In the following steps, it will be used to discover the unique prefix in the key space. An easy way to do this is to use uuidgen:

UUID=$(uuidgen)
Copy the code

Specify the cluster size

When obtaining the token, you must specify the cluster size. The discovery service uses this size to know when it has discovered all the members that will initially make up the cluster.

The curl -x PUT http://10.0.10.10:2379/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83/_config/size -d value=3
Copy the code

We need to put the url http://10.0.10.10:2379/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83 as parameters, discovery to start Etcd.

Nodes automatically using the http://10.0.10.10:2379/v2/keys/discovery/6c007a14875d53d9bf0ef5a6fc0257c817f0fb83 directory etcd registration and discovery service.

Public Discovery Service

When there is no etCD cluster available locally, the etCD official website provides an ETCD storage address accessible to the public network. We can get the directory of the etcd service by using the following command as the –discovery parameter.

The common discovery service discovery.etcd.io works in the same way, but with a layer of decorations that can extract ugly urls, automatically generate UUID, and provide protection against excessive requests. Common discovery services still use etCD clusters as data stores on them.

$ curl http://discovery.etcd.io/new? size=3 http://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573deCopy the code

Start a cluster in dynamic discovery mode

In etcd discovery mode, run the following command to start etcd:

# etcd1 start$/ opt/etcd/bin/etcd - name etcd1 - initial - advertise - peer - \ urls http://192.168.202.128:2380 - listen - peer - urls http://192.168.202.128:2380 \ - data - dir/opt/etcd/data \ - listen - the client - urls http://192.168.202.128:2379, http://127.0.0.1:2379 \ \ - advertise - the client - urls http://192.168.202.128:2379 - discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de# etcd2 start/ opt/etcd/bin/etcd - name etcd2 - initial - advertise - peer - \ urls http://192.168.202.129:2380 - listen - peer - urls http://192.168.202.129:2380 \ - data - dir/opt/etcd/data \ - listen - the client - urls http://192.168.202.129:2379, http://127.0.0.1:2379 \ \ - advertise - the client - urls http://192.168.202.129:2379 - discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573de# etcd3 start/ opt/etcd/bin/etcd - name etcd3 - initial - advertise - peer - \ urls http://192.168.202.130:2380 - listen - peer - urls http://192.168.202.130:2380 \ - data - dir/opt/etcd/data \ - listen - the client - urls http://192.168.202.130:2379, http://127.0.0.1:2379 \ \ - advertise - the client - urls http://192.168.202.130:2379 - discovery https://discovery.etcd.io/3e86b59982e49066c5d813af1c2e2579cbf573deCopy the code

It is important to note that after we complete the initialization of the cluster, this information becomes useless. To add a node, run the etcdctl command. For security, each time a new ETCD cluster is started, the new Discovery token is registered. In addition, if more than the specified number of nodes are started during initialization, the excess nodes are automatically converted to Proxy mode ETCD.

results

Once the cluster is up and verified, let’s look at the members of the cluster:

$ /opt/etcd/bin/etcdctl member list
# The result is as follows40 e2ac06ca1674a7, started, etcd3, http://192.168.202.130:2380, http://192.168.202.130:2379,falseC532c5cedfe84d3c, started, etcd1, http://192.168.202.128:2380, http://192.168.202.128:2379,falseDb75d3022049742a, started, etcd2, http://192.168.202.129:2380, http://192.168.202.129:2379,false
Copy the code

The results are in line with expectations. Let’s look at the health status of the node:

$ /opt/etcd/bin/etcdctl  --endpoints=http://192.168.202.128:2379, http://192.168.202.129:2379, http://192.168.202.130:2379, ""  endpoint  health
# The result is as followshttp://192.168.202.128:2379 is healthy: successfully committed proposal: Took = 3.157068 ms http://192.168.202.130:2379 is healthy: successfully committed proposal: Took = 3.300984 ms http://192.168.202.129:2379 is healthy: successfully committed proposal: took = 3.263923 msCopy the code

4 DNS self-discovery mode

Etcd also supports booting with DNS SRV records. In fact, DNS SRV records are used to achieve continuous rotation query. DNS SRV is a type of resource record supported by DNS databases. It records the mapping between a computer and the service information provided by it.

4.1 installed Dnsmasq

Dnsmasq is used to create the DNS service. Dnsmasq Provides DNS cache, DHCP, and Tftp services. As a domain name resolution server, Dnsmasq can cache DNS requests to speed up connections to visited urls. Dnsmasq is lightweight and easy to configure. It is suitable for individual users or networks with less than 50 hosts. It also comes with a PXE server.

When receiving a DNS request, Dnsmasq first looks for the /etc/hosts file and then the external DNS defined in /etc/resolv.conf. Configure Dnsmasq as the DNS cache server and add local Intranet resolution to the /etc/hosts file. In this way, the hosts file will be queried preferentially by Intranet servers. In this way, the /etc/hosts file will be shared with all Intranet servers. So as to solve the problem of Intranet machine mutual recognition. Instead of editing hosts files on a machine-by-machine basis or adding Bind DNS records, you can edit only one hosts file.

Centos 7 based on the author’s host, first install Dnsmasq:

yum install dnsmasq
Copy the code

Once installed, do the configuration, all in one file /etc/dnsmasq.conf. We can also write configuration files of any name in /etc/dnsmasq.d.

The IP address of the upstream server is specified

Resolv-file Configures an additional upstream DNS server for Dnsmasq. If this function is disabled, nameserver in /etc/resolv.conf of the Linux host is used.

$ vim /etc/dnsmasq.conf

# Add the following content:Resolv - file = / etc/resolv dnsmasq. Conf, srv-host=_etcd-server._tcp.blueskykong.com, etcd1.blueskykong.com, 2380,0,100 Srv-host=_etcd-server._tcp.blueskykong.com, etcd2.blueskykong.com, 2380,0,100 Srv-host=_etcd-server._tcp.blueskykong.com, etcd3.blueskykong.com, 2380,0,100Copy the code

The corresponding domain name records in dnsmasq.conf are configured for the three servers involved, corresponding to ETCD1, ETCD2, and ETCD3 respectively.

Adds a forwarding DNS address to the specified file
$vim /etc/resolv.dnsmasq.conf nameserver 8.8.8.8 nameserver 8.8.4.4Copy the code

These two free DNS services, you should be familiar with. You can perform the configuration based on the actual local network.

Dnsmasq parsing is enabled locally
$vim /etc/resolv.conf nameserver 127.0.0.1Copy the code

It makes sense to parse Dnsmasq configuration locally.

Adding parsing records

Configure the associated A record for each domain name to point to the machine IP corresponding to the etCD core node. You can add resolution records using the default hosts file, custom hosts file, or custom CONF file. Here we use the simpler first method.

$ vim /etc/hosts

# Add the following content parsingEtcd1.blueskykong.com 192.168.202.129 etcd2.blueskyKong.com 192.168.202.130 etcd3.blueskykong.comCopy the code

Start the service

service dnsmasq start
Copy the code

Once started, we verify:

  • Query SRV records on the DNS server. The result is as follows:

    $dig @192.168.202.128 +noall +answer SRV _etcd-server._tcp.blueskykong.com _etcd-server._tcp.blueskykong.com. 0 IN SRV 0 100 2380 etcd2.blueskykong.com. _etcd-server._tcp.blueskykong.com. 0 IN SRV 0 100 2380 etcd1.blueskykong.com. _etcd-server._tcp.blueskykong.com. 0 IN SRV 0 100 2380 etcd3.blueskykong.com.Copy the code

  • Query domain name resolution results

    $dig @192.168.202.128 +noall +answer etcd1.blueskykong.com etcd2.blueskykong.com etcd3.blueskykong.com Etcd1.blueskykong.com. 0 IN A 192.168.202.128 etcd2.blueskyKong.com. 0 IN 192.168.202.129 etcd3.blueskyKong.com A 192.168.202.130Copy the code

At this point, we have successfully installed Dnsmasq. Let’s start the ETCD cluster based on DNS discovery.

Start the cluster

With the above two-step DNS configuration, you can start the ETCD cluster using DNS. You need to delete the ETCD_INITIAL_CLUSTER configuration (for static service discovery) and specify the DNS SRV domain name (ETCD_DISCOVERY_SRV). Set the URL parameter of DNS resolution to -discovery-srv, where the etcd1 node start command is as follows:

$ /opt/etcd/bin/etcd --name etcd1 \ --discovery-srv blueskykong.com \ --initial-advertise-peer-urls http://etcd1.blueskykong.com:2380 \ --initial-cluster-token etcd-cluster-1 \ --data-dir /opt/etcd/data \ --initial-cluster-state new \ --advertise-client-urls http://etcd1.blueskykong.com:2379 \ --listen-client-urls http://0.0.0.0:2379 \ - listen - peer - urls http://0.0.0.0:2380Copy the code

Etcd cluster members can broadcast using domain names or IP addresses, and DNS records are resolved during startup. — The parse address in initial-advertise-peer-urls must match the parse address in the SRV target. The ETCD member reads the parsed address to find out if it belongs to the cluster defined in the SRV record.

We verify the correctness of starting a cluster based on DNS discovery by looking at the list of cluster members:

$ /opt/etcd/bin/etcdctl member list

# The result is as follows:40 e2ac06ca1674a7, started, etcd3, http://192.168.202.130:2380, http://etcd3.blueskykong.com:2379,falseC532c5cedfe84d3c, started, etcd1, http://192.168.202.128:2380, http://etcd1.blueskykong.com:2379,falseDb75d3022049742a, started, etcd2, http://192.168.202.129:2380, http://etcd2.blueskykong.com:2379,false
Copy the code

As you can see, the resulting output ETCD cluster has three members, as expected. Let’s continue to verify the status of cluster nodes using IP addresses.

$ /opt/etcd/bin/etcdctl  --endpoints=http://192.168.202.128:2379, http://192.168.202.129:2379, http://192.168.202.130:2379, ""  endpoint  health

# The result is as follows:http://192.168.202.129:2379 is healthy: successfully committed proposal: Took = 2.933555 ms http://192.168.202.128:2379 is healthy: successfully committed proposal: Took = 7.252799 ms http://192.168.202.130:2379 is healthy: successfully committed proposal: took = 7.415843 msCopy the code

More etCD cluster operations, readers can try their own, I do not expand one by one here.

5 subtotal

This article is a supplement on the basis of the single etCD installation in the last article. It mainly introduces various installation and startup modes of ETCD cluster: static monomer, static Docker, dynamic discovery and DNS discovery startup modes. So many installation positions, are for our actual use, the next article we will specifically enter the use of etCDCTL explained.

Subscribe to the latest articles, welcome to follow my official account

Recommended reading

  1. Comparison between ETCD and other K-V components such as Zookeeper and Consul
  2. Thoroughly understand ETCD series article (1) : first know ETCD
  3. Thoroughly understand etCD series article (2) : ETCD installation posture

reference

etcd docs