Today let’s try deploying a Redis cluster in K8S to learn more about the details and features of K8S.

Note: Background knowledge and details related to Redis-cluster are not described here

I. Problem analysis

In essence, deploying a Redis cluster on K8S is not much different from deploying a normal application, but there are a few things to be aware of:

Redis is a stateful application

This is the problem we need to pay attention to when deploying redis cluster. When we deploy Redis in the form of POD in K8S, the data cached in each pod is different, and the IP of pod can change at any time. At this time, many problems will occur if common deployment and Service are used to deploy redis-cluster, so StatefulSet + Headless service is needed to solve the problem

Data persistence

Although Redis is based on memory, it still needs to rely on disk for persistent data, so that the cached data can be recovered when the service is restarted. In a cluster, we need to use shared file system + PV (persistent volume) to make all pods in the cluster share the same persistent store

Second, concept introduction

Before we get started, let’s explain a few concepts and principles in detail.

A Headless Service is a Service that does not specify a Cluster IP address. In k8s DNS mapping, the result of a Headless Service is not a Cluster IP address. It’s the IP list of all the pods it’s associated with

StatefulSet StatefulSet is a resource in K8S specifically designed for stateful application Deployment. Generally, it can be considered as a variant of Deployment/RC and has the following features:

Each Pod managed by StatefulSet has a unique document/network identity and is generated numerically, rather than having a random name and IP as in Deployment (e.g., The StatefulSet name is redis, So the pod name is redis-0, redis-1…

StatefulSet ReplicaSet’s start and stop sequence is strictly controlled. Operation of the NTH POD must wait for the first N-1 to complete

Pods in StatefulSet are stored in stable persistence and the corresponding PVS are not destroyed when pods are deleted

In addition, StatefulSet must work with Headless Service. It adds another layer to the DNS mapping provided by Headless Service, resulting in a per-POD domain name mapping in the following format:

$(podname).$(headless service name)

With this mapping, you can configure clusters using domain names instead of IP addresses to manage stateful application clusters

Three,

With StatefulSet and Headless Services, the cluster deployment plan is designed as follows (figure from the reference article) :

The configuration procedure is as follows: Configure the shared file system NFS create PV and PVC Create ConfigMap Create Headless Service Create StatefulSet Initialize the REDis cluster

The actual operation

To simplify the complexity, the data is mounted using ordinary volumes instead of PV and PVC.

1. Create the ConfigMap configuration file

appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
cluster-node-timeout 5000
dir /var/lib/redis
port 6379
Copy the code

Kubectl create configmap redis-conf –from-file=redis.conf to create configMap

2. Create HeadlessService

apiVersion: v1
kind: Service
metadata:
  name: redis-service
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    port: 6379
  clusterIP: None
  selector:
    app: redis
    appCluster: redis-cluster
Copy the code

3. Create StatefulSet

apiVersion: apps/v1beta1 kind: StatefulSet metadata: name: redis-app spec: serviceName: "redis-service" replicas: 6 template: metadata: labels: app: redis appCluster: redis-cluster spec: terminationGracePeriodSeconds: 20 affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - redis topologyKey: kubernetes.io/hostname containers: - name: Redis image: "registry.cn-qingdao.aliyuncs.com/gold-faas/gold-redis:1.0 command:" - "redis - server" args. - "/etc/redis/redis.conf" - "--protected-mode" - "no" resources: requests: cpu: "100m" memory: "100Mi" ports: - name: redis containerPort: 6379 protocol: "TCP" - name: cluster containerPort: 16379 protocol: "TCP" volumeMounts: - name: "redis-conf" mountPath: "/etc/redis" - name: "redis-data" mountPath: "/var/lib/redis" volumes: - name: "redis-conf" configMap: name: "redis-conf" items: - key: "redis.conf" path: "redis.conf" - name: "redis-data" emptyDir: {}Copy the code

Initialize the Redis cluster

After StatefulSet is created, you can see that the six pods have started, but the entire Redis cluster has not been initialized yet and you need to use the official provided Redis-Trib tool.

We can certainly run the corresponding tools on any Redis node to initialize the entire cluster, but this is not appropriate. We want to keep the responsibility of each node as single as possible, so it is best to have a separate POD to run the management tools for the entire cluster.

Redis-trib is an official redDIS-cluster management tool, which can achieve the creation and updating of redIS clusters. In the early version of Redis, It works as a Ruby script in the source package redis-trib.rb (you can also pull python on PIP, but I failed to run it) and is now officially integrated into redis-CLI (I used 5.0.3).

To initialize the cluster, start by creating an Ubuntu pod on K8S that will serve as the management node:

kubectl run -i --tty redis-cluster-manager --image=ubuntu --restart=Never /bin/bash
Copy the code

Install some tools, including wget,dnsutils, and then download and install Redis:

Wget http://download.redis.io/releases/redis-5.0.3.tar.gz tar - XVZF redis - 5.0.3. Tar. Gz CD redis - 5.0.3. Tar. Gz && makeCopy the code

After compiling, redis-cli will be placed in the SRC directory. Put it in /usr/local/bin to facilitate subsequent operations

To obtain the host IP addresses of the six nodes, you can use nsLookup in connection with StatefulSet’s domain name rules. For example, to find the IP address of the pod redis-app-0, run the following command:

root@redis-cluster-manager:/# nslookup redis-app-0.redis-service Server: 10.96.0.10 Address: 10.96.0.10#53 Name: Redis - app - 0. Redis - service. Gold. SVC. Cluster. The local Address: 172.17.0.10Copy the code

172.17.0.10 is the corresponding IP address. For this deployment we use 0, 1, and 2 as Master nodes; 3, 4, and 5 as Slave nodes, run the following command to initialize the Master node of the cluster:

Redis -cli --cluster create 172.17.0.10:6379 172.17.0.11:6379 172.17.0.12:6379Copy the code

Then attach corresponding Slave nodes to each of them. The cluster-master-ID is given in the previous step:

Redis-cli --cluster add-node 172.17.0.13:6379 172.17.0.10:6379 --cluster-slave --cluster-master-id Adf443a4d33c4db2c0d4669d61915ae6faa96b46 redis - cli - cluster add -node 172.17.0.14:6379 172.17.0.11:6379 - cluster - slave - cluster - master - id 6 e5adcb56a871a3d78343a38fcdec67be7ae98f8 redis - cli - cluster add -node 172.17.0.16:6379 172.17.0.12:6379 - cluster - slave - cluster - master - id c061e37c5052c22f056fff2a014a9f63c3f47ca0Copy the code

After cluster initialization, enter a node at random to check the cluster information:

Redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli: redis-cli:

5. Create Service

Now any node in the Redis cluster can be accessed directly, but in order to provide access to the other services in the cluster, we need to create a service for service discovery and load balancing (note that the service is not the same as the headless service we created earlier).

Yaml files are as follows:

apiVersion: v1
kind: Service
metadata:
  name: gold-redis
  labels:
    app: redis
spec:
  ports:
  - name: redis-port
    protocol: "TCP"
    port: 6379
    targetPort: 6379
  selector:
    app: redis
    appCluster: redis-cluster
Copy the code

Do a test after deployment:

Nice. All the work is done here