1. Environment preparation

Apply for three ECS virtual servers on huawei cloud for temporary test. You are advised to apply for three ECS virtual servers on demand. Select the VPC network segment 192.168.0.0/16, and deploy the Kubernetes cluster v1.23.1 on these three nodes. Why three nodes? To prepare the experimental CEPH as back-end distributed storage, at least three nodes are required, and a raw disk is required to be used as the CEPH data storage disk. Therefore, each ECS is configured with an additional 100 GB empty EVS block storage on Huawei Cloud. The details are as follows: Choose Flannel as the underlying network plug-in.

role OS The node name storage IP docker version kubelet version kubeadm version kubectl version network
master Centos7.9 master 40 GB +100 GB (data disk) 192.168.0.11 Docker 20.10.8 V1.23.1 V1.23.1 V1.23.1 flannel
master Centos7.9 node1 40 GB +100 GB (data disk) 192.168.0.23 Docker 20.10.8 V1.23.1 V1.23.1 V1.23.1 flannel
master Centos7.9 node2 40 GB +100 GB (data disk) 192.168.0.51 Docker 20.10.8 V1.23.1 V1.23.1 V1.23.1 flannel

Table 1: Environment information, you can swipe left and right to view all information.

Note: Kubernetes removes docker dependencies by default from version V.1.20. If you install Docker and Containerd on your host, you will use Docker as a container engine first. If docker is not installed on the host and only Containerd is installed, containerd will be used to run the engine as a container. In order to reduce learning cost, docker installation is selected here.

2. Configure security groups

No matter huawei cloud, Tencent Cloud, Ali Cloud, AWS, Azure will choose the security group by default when configuring and generating virtual machines, so as to carry out simple protection on the virtual machine network, that is, the security group limits which ports are released and which ports are accessible. During the installation process, some components of K8s provide network services through Service and POD. Yes The corresponding port must be enabled. Otherwise, the service may fail to be deployed.

Of course, if you directly use physical servers or VMware VMS, do not need to consider. The following network ports need to be enabled by default:

2.1 Inbound rules

Checking ports on the master node:

Protocol Direction Port Range Purpose
TCP Inbound 6443 Kube-apiserver
TCP Inbound 2379-2380. Etcd API
TCP Inbound 10250 Kubelet API
TCP Inbound 10251 Kube-scheduler
TCP Inbound 10252 Kube-controller-manager

Checking ports on node1 and node2:

Protocol Direction Port Range Purpose
TCP Inbound 10250 Kubelet api
TCP Inbound 30000-32767. NodePort Service

2.2 Outbound rules

Agreement rules port source strategy
ALL ALL 0.0.0.0/0 allow

3. Configure basic information

Prepare basic software for all nodes.

3.1 Modifying Host Information

Set the host name on each of the three hosts to take effect after restart
hostnamectl set-hostname master
reboot

hostnamectl set-hostname node1
reboot

hostnamectl set-hostname node2
reboot

# Synchronize time on all three hosts
systemctl restart chronyd

Configure hosts addresses for DNS resolution on the three hosts
cat >> /etc/hosts << EOF 192.168.0.11 Master 192.168.0.23 node1 192.168.0.51 node2 EOF

Node1 and node2 can be accessed directly on the master node by copying the key to the other two nodes
# If you want to access nodes on node1 and node2, execute the following statement to generate and copy the keys, respectively
ssh-keygen -t rsa
ssh-copy-id root@node1
ssh-copy-id root@node2

# disable firewall and iptables
systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl stop iptables.service
systemctl disable iptables.service

Close # SELinux
setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

Close # swap
swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab

# configure kernel parameters:
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
Copy the code

3.2 Modifying the Yum Source

sudo mkdir /etc/yum.repos.d/bak && mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak wget -O /etc/yum.repos.d/CentOS-Base.repo https://repo.huaweicloud.com/repository/conf/CentOS-7-reg.repo sudo yum clean all sudo  yum makecache fastCopy the code

3.3 Installing basic Software

Install autocomplete software and base dependency packages
sudo yum install -y bash-completion
source /etc/profile.d/bash_completion.sh
sudo yum remove docker docker-common docker-selinux docker-engine
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
Copy the code

4. Install the docker

Install Docker on all nodes and configure image acceleration.

4.1 Installing Docker software

# Add the Docker yum source
wget -O /etc/yum.repos.d/docker-ce.repo https://repo.huaweicloud.com/docker-ce/linux/centos/docker-ce.repo
sudo sed -i 's+download.docker.com+repo.huaweicloud.com/docker-ce+' /etc/yum.repos.d/docker-ce.repo
sudo yum makecache fast

# to install the latest version of the Docker 20.10.8, before the installation can use yum list Docker - ce - showduplicates | sort - r view the Docker yum source list
sudo yum install -y  docker-ce
sudo systemctl enable docker 
sudo systemctl start docker
sudo systemctl status docker
docker --version
Copy the code

4.2 Docker Image acceleration and Cgroup Setting

/etc/docker/daemon.json file
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://e2660ea6dc2b4a16a3ae382f8d227beb.mirror.swr.myhuaweicloud.com"]."exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

[" native-cgroupDriver =systemd"] [" native-cgroupdriver =systemd"] [" native-cgroupdriver =systemd"] [" native-cgroupdriver =systemd"

# restart docker
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl status docker
Copy the code

5. Install the Kubernets cluster

Install Kubernetes on the Master node and add Node1 and Node2 to the cluster.

5.1 installation Kubeadm

Install kubeadm, Kubelet, kubectl tools on each of the three nodes.

# Add Ali Kubernetes source on all nodes
cat >> /etc/yum.repos.d/kubernetes.repo  <<EOF [kubernetes] name=Kubernetes Repository baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg EOF

sudo yum clean all
sudo yum makecache fast

# check whether package 1.23.1 is in the yum source
yum list kubelet --showduplicates | sort -r
yum list kubectl --showduplicates | sort -r
yum list kubeadm --showduplicates | sort -r

Kubectl, kubelet will be installed automatically
sudo yum install -y kubeadm
sudo systemctl enable kubelet
sudo systemctl start kubelet

kubeadm version
kubectl version
kubelet --version

#kubectl command completion
cd
echo "source <(kubectl completion bash)" >> ~/.bash_profile
source .bash_profile 

# master view the desired image
sudo kubeadm config images list
------------------------------------------------
The following image is required for the queryK8s. GCR. IO/kube - apiserver: v1.23.1 k8s. GCR. IO/kube - controller - manager: v1.23.1 k8s. GCR. IO/kube - the scheduler: v1.23.1 K8s. GCR. IO/kube - proxy: v1.23.1 k8s. GCR. IO/pause: 3.6 k8s. GCR. IO/etcd: 3.5.1 track of - 0 k8s. GCR. IO/coredns/coredns: v1.8.6 -------------------------------------------------# Since kubeadm relies on the image of k8s.gcr. IO from abroad, the solution is to download the image of kubeadm from abroad and re-tag itDocker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.23.1 docker pull Registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.23.1 docker pull Registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.23.1 docker pull Registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.23.1 docker pull Registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 docker pull Registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.1-0 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.6# Change the tag back to k8s.gcr. IODocker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.23.1 k8s. GCR. IO/kube - apiserver: v1.23.1 Docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.23.1 K8s. GCR. IO/kube - controller - manager: v1.23.1 docker tag Registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.23.1 k8s. GCR. IO/kube - the scheduler: v1.23.1 docker tag Registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.23.1 k8s. GCR. IO/kube - proxy: v1.23.1 docker tag Registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 k8s. GCR. IO/pause: 3.6 docker tag Registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.1-0 k8s. GCR. IO/etcd: 3.5.1 track of 0 docker tag Registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.8.6 k8s. GCR. IO/coredns/coredns: v1.8.6Copy the code

5.2 Initializing the Master Node

IP address 10.244.0.0/16, IP address 10.244.0.0/16, IP address 10.244.0.0/16, IP address 10.244.0.0/16Kubeadm init \ --kubernetes-version=1.23.1 \ --apiserver-advertise-address=192.168.0.11 \ --service-cidr=10.1.0.0/16 \ - pod - network - cidr = 10.244.0.0/16Copy the code

The initial installation process is printed as follows:

[init] Using Kubernetes version: v1.23.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.lo                                          cal master] and IPs [10.1.0.1 192.168.0.46]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.0.46 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.0.46 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 5.502543 seconds [uploa-config] Storing the configuration usedin ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "Kubelet - config - 1.23" in namespace kube-system with the configuration for the kubelets in the cluster
NOTE: The "Kubelet - config - 1.23" naming of the kubelet ConfigMap is deprecated. Once the UnversionedKubeletConfigMap feature gate graduates to Beta the default name will become just "kubelet-config". Kubeadm upgrade will handle this transition transparently.
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/master(deprecated) node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: b2n16t.n6filxh3vc6byr7c
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.46:6443 --token b2n16t.n6filxh3vc6byr7c \
        --discovery-token-ca-cert-hash sha256:f4d103707658df3fa7a8dc95a59719f362cd42edb40c8ebc5ae19d53655813d1
Copy the code

Copy the configuration to the. Kube directory as prompted

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Copy the code

Flannel is an application network plug-in

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
Copy the code

5.3 Adding a Node to a Cluster

Execute kubectl join on both nodes to join the cluster.

[root@node1 ~]# kubeadm join 192.168.0.46:6443 --token b2n16t.n6filxh3vc6byr7c \
>         --discovery-token-ca-cert-hash sha256:f4d103707658df3fa7a8dc95a59719f362cd42edb40c8ebc5ae19d53655813d1
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

[root@node1 ~]#
Copy the code

If you want to execute kubectl on the node, copy the master configuration to $HOME/. Kube on the node.

Kubectl starts reading the certificate configuration directory by default
[root@node1 ~]# mkdir -p $HOME/.kube

Copy the configuration on the master node to node1
[root@master ~]# scp .kube/config root@node1:/root/.kube/

You can view the status of nodes, components, pods, etc
[root@node1 ~]# kubectl get nodesNAME STATUS ROLES AGE VERSION Master Ready Control-plane,master 43M v1.23.1 node1 Ready < None > 39m v1.23.1 node2 Ready < None > 39m v1.23.1 [root@node1 ~]# kubectl get ns
NAME              STATUS   AGE
default           Active   43m
kube-node-lease   Active   43m
kube-public       Active   43m
kube-system       Active   43m
[root@node1 ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated inV1.19 + NAME STATUS MESSAGE ERROR Scheduler Healthy OK controller-manager Healthy OK ETcD-0 Healthy {"health":"true"."reason":""}
[root@node1 ~]# kubectl get pods -nkube-system
NAME                             READY   STATUS    RESTARTS   AGE
coredns-64897985d-bs6b9          1/1     Running   0          43m
coredns-64897985d-s2kml          1/1     Running   0          43m
etcd-master                      1/1     Running   0          44m
kube-apiserver-master            1/1     Running   0          44m
kube-controller-manager-master   1/1     Running   0          44m
kube-flannel-ds-8jpd4            1/1     Running   0          39m
kube-flannel-ds-jlfzx            1/1     Running   0          39m
kube-flannel-ds-jztwk            1/1     Running   0          41m
kube-proxy-5lnr9                 1/1     Running   0          39m
kube-proxy-thghs                 1/1     Running   0          43m
kube-proxy-w7rhv                 1/1     Running   0          39m
kube-scheduler-master            1/1     Running   0          44m
[root@node1 ~]#

Copy the code

5.4 Installing ceph Storage

We installed ceph storage via RoOK, and CEPh requires at least three nodes with at least one raw disk per node. When we applied for ECS, we added a 100G EVS block storage disk.

[root@master ~]# lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 00 0 disk ├ ─vda1 253:1 00 0 part/VDB 253:1 00 00 0 disk [root@master ~]# lsblk -fNAME FSTYPE LABEL UUID MOUNTPOINT vda ├ ─vda1 ext4 b64C5c5D-9f6B-4754-9e1e-eAEF91437f7A/VDBCopy the code

For convenience, go to Github and download the entire RoOK project.

yum install -y git
git clone https://github.com/rook/rook.git
Copy the code

Ensure that the system kernel supports RBD

[root@master ~]# uname -r3.10.0-1160.15.2. El7. X86_64 / root @ master ~# modprobe rbd
[root@master ~]# lsmod |grep rbd
rbd                   102400  0
libceph               413696  1 rbd
Copy the code

Since we only deployed 3 nodes, and cepH requires a minimum of 3 nodes, POD is not allowed to be deployed on the master node by default. In order to ensure the normal deployment of POD in CEPH, we removed the stain on the master node in advance and allowed pod to be deployed on the master node.

[root@master1 ~]# kubectl taint nodes --all node-role.kubernetes.io/master-
Copy the code

Start deploying RoOK

cd /root/rook/deploy/examples
kubectl apply -f crds.yaml -f common.yaml 
kubectl apply -f operator.yaml   # if img can't be downloaded, you can pull it to local in advance
kubectl apply -f cluster.yaml 
kubectl get pods -n rook-ceph -o wide
Copy the code

According to the prompts, there will be many images that cannot be pulled, and they will be gradually obtained on Aliyun, which needs to be executed on each node. Several images will be recorded:

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/rook/ceph:master docker pull Registry.cn-hangzhou.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.3.0 docker pull Registry.cn-hangzhou.aliyuncs.com/google_containers/csi-provisioner:v3.0.0 docker pull quay. IO/cephcsi/cephcsi: v3.4.0 Docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-attacher:v3.3.0 docker pull Registry.cn-hangzhou.aliyuncs.com/google_containers/csi-snapshotter:v4.2.0 docker pull Registry.cn-hangzhou.aliyuncs.com/google_containers/csi-resizer:v1.3.0 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.3.0 K8s. GCR. IO/sig - storage/csi - node - driver - the registrar: v2.3.0 docker tag Registry.cn-hangzhou.aliyuncs.com/google_containers/csi-provisioner:v3.0.0 k8s. GCR. IO/sig - storage/csi - provisioner: v3.0.0 Docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-attacher:v3.3.0 K8s. GCR. IO/sig - storage/csi - attacher: v3.3.0 docker tag Registry.cn-hangzhou.aliyuncs.com/google_containers/csi-snapshotter:v4.2.0 k8s. GCR. IO/sig - storage/csi - snapshotter: v4.2.0 Docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-resizer:v1.3.0 K8s. GCR. IO/sig - storage/csi - resizer: v1.3.0Copy the code

After the mirroring problem is resolved, check the POD status of the namespace:

[root@master examples]# kubectl get pods -nrook-cephNAME READY STATUS RESTARTS AGE csi-cephfsplugin-ct842 3/3 Running 0 25m csi-cephfsplugin-cvb7f 3/3 Running 0 25m csi-cephfsplugin-j5gbm 3/3 Running 0 25m csi-cephfsplugin-provisioner-5c8b6d6f4-hhvjq 6/6 Running 0 25m csi-cephfsplugin-provisioner-5c8b6d6f4-kr4n5 6/6 Running 0 25m csi-rbdplugin-fcbk9 3/3 Running 0 25m csi-rbdplugin-fpv8t  3/3 Running 0 25m csi-rbdplugin-provisioner-8564cfd44-jkqrq 6/6 Running 0 25m csi-rbdplugin-provisioner-8564cfd44-q8srg  6/6 Running 0 25m csi-rbdplugin-qtgvt 3/3 Running 0 25m rook-ceph-crashcollector-master-7bcf565ddc-4mvmk 1/1 Running 0 20m rook-ceph-crashcollector-node1-7bfc99f96d-2jw4w 1/1 Running 0 20m rook-ceph-crashcollector-node2-678f85bdf-qw2gq 1/1  Running 0 20m rook-ceph-mgr-a-574b6956fd-fzt5q 1/1 Running 0 20m rook-ceph-mon-a-668b48987f-g5zfw 1/1 Running 0 25m rook-ceph-mon-b-54996b7487-6qscc 1/1 Running 0 24m rook-ceph-mon-c-6cc5bd5c85-wsrn9 1/1 Running 0 22m rook-ceph-operator-75dd789779-8kq7z 1/1 Running 0 30m rook-ceph-osd-0-849c84cc87-bzpf9 1/1 Running 0 20m rook-ceph-osd-1-77cfc975bb-hbdnn 1/1 Running 0 20m rook-ceph-osd-2-5c7d59d74d-g67fz 1/1 Running 0 20m rook-ceph-osd-prepare-master-98nld 0/1 Completed 0 20m rook-ceph-osd-prepare-node1-nvqvg 0/1 Completed 0 20m rook-ceph-osd-prepare-node2-x6cnk 0/1 Completed 0 20m [root@master examples]# kubectl get service -n rook-cephNAME TYPE cluster-ip external-ip PORT(S) AGE CSI-cephfSplugin-metrics ClusterIP 10.1.101.105 < None > 8080/TCP,8081/TCP 26m cSI-rbdplugin-metrics ClusterIP 10.1.238.71 <none> 8080/TCP,8081/TCP 26M rook-ceph-Mgr ClusterIP 10.1.98.179 <none> 9283/TCP 21M rook-ceph-Mgr-Dashboard ClusterIP 10.1.251.161 < None > 8443/TCP 21M rook-ceph-mon-a ClusterIP 10.1.0.149 <none> 6789/TCP,3300/TCP 26m rook-ceph-mon-b ClusterIP 10.1.42.253 <none> 6789/TCP,3300/TCP 25m rook-ceph-mon-c ClusterIP 10.1.99.90 < None > 6789/TCP,3300/TCP 24MCopy the code

<1>. The normal status of the three nodes rook-ceph-osD-prepare is Completed <2>. If one of the nodes is Running or lacks a Rook-ceph-OSD node, check the time, firewall, and memory usage of the abnormal node. <3>. Deploy Toolbox

The dashboard above is cluster IP for internal cluster access. If you want external access, you can deploy the NodePort type dashboard. Fortunately, the RoOK project is already written and can be used directly.

[root@master examples]# cd /root/rook/deploy/examples
[root@master examples]#
[root@master examples]# kubectl apply -f dashboard-external-https.yaml
service/rook-ceph-mgr-dashboard-external-https created
[root@master examples]# kubectl get service -n rook-cephNAME TYPE cluster-ip external-ip PORT(S) AGE CSI-cephfSplugin-metrics ClusterIP 10.1.101.105 < None > 8080/TCP,8081/TCP 31m cSI-rbdplugin-metrics ClusterIP 10.1.238.71 <none> 8080/TCP,8081/TCP 31m rook-ceph-Mgr ClusterIP 10.1.98.179 <none> 9283/TCP 26m rook-ceph-Mgr-Dashboard ClusterIP 10.1.251.161 < None > 8443/TCP 26m rook-ceph-Mgr-dashboard-external-https NodePort 10.1.182.240 < None > 8443:30301/TCP 35s rook-ceph-mon-a ClusterIP 10.1.0.149 < None > 6789/TCP,3300/TCP 31m Rook-ceph-mon -b ClusterIP 10.1.42.253 <none> 6789/TCP,3300/TCP 30m rook-ceph-mon-c ClusterIP 10.1.99.90 <none> 6789/TCP,3300/TCP 28m [root@master examples]#
Copy the code

An additional NodePort service of port 30301 is available. Access the service using the IP address of any node: https://Node-EIP1:30301, and enter the user name and password. The default dashboard user name is admin. You can run the following command to obtain the password:

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
Copy the code

Deployment of Ceph Toolbox: The Ceph cluster is enabled by default and Ceph authentication is enabled. In this way, you cannot log into the Pod where Ceph components are located to obtain the cluster status and execute CLI commands. In this case, you need to deploy Ceph Toolbox.

kubectl apply -f toolbox.yaml
# Check whether it is normal
kubectl -n rook-ceph get pods -o wide | grep ceph-tools
Ceph CLI command:
kubectl -n rook-ceph exec -it rook-ceph-tools-76c7d559b6-8w7bk bash
# Check cluster status
ceph status
Copy the code

Rook provides RBD services. Rook can provide the following three types of storage:

  • Block: Create block storage to be consumed by a pod
  • Object: Create an object store that is accessible inside or outside the Kubernetes cluster
  • Shared File System: Create a file system to be shared across multiple pods

Before Provisioning block storage, the StorageClass and storage pool need to be created. K8S needs both types of resources to interact with Rook and to allocate persistent volumes (PVS).

To provide the RBD block device service in a Kubernetes cluster, the following steps are required:

1) Create rdD-provisioner Pod

  • Create the Storageclass corresponding to the RBD
  • Create a PVC using the STORAGeclass corresponding to the RBD
  • Create pods using RBD PVC
  • After creating the Ceph Cluster from RoOK, RoOK itself provides the RDD-Provisioner service, so we don’t need to deploy its provisioner anymore.
  • Yaml vim StorageClass. Yaml contains a storage pool named replicapool. The storageClass named rook-ceph-block runs the YAML file
[root@master ~]#cd rook/deploy/examples/csi/rbd
[root@master rbd]# kubectl apply -f storageclass.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created
Copy the code

2) View the created storageclass:

[root@master rbd]# kubectl get storageclass
NAME              PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block   rook-ceph.rbd.csi.ceph.com   Delete          Immediate           true                   2m36s
Copy the code

3) Log in to ceph Dashboard to view the created storage pool: Using storage, using the official wordpress service example, create a classic wordpress and mysql application to use the block volumes provided by Rook. Both applications will use the block volumes provided by Rook. Yaml file configuration, mainly look at the definition of PVC and mount volume, using wordpress.yaml and mysql.yaml as examples:

[root@master ~]# cd rook/deploy/examples/
[root@master examples]# kubectl apply -f wordpress.yaml -f mysql.yaml
service/wordpress created
persistentvolumeclaim/wp-pv-claim created
deployment.apps/wordpress created
service/wordpress-mysql created
persistentvolumeclaim/mysql-pv-claim created
deployment.apps/wordpress-mysql created
[root@master examples]# kubectl get deployments.apps
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
wordpress         0/1     1            0           28s
wordpress-mysql   0/1     1            0           28s
Copy the code

Both applications create a block storage volume and mount it to their respective PODS. Check the declared PVC and PV:

[root@master examples]# kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim   Bound    pvc-cdfbbd11-a22e-4f72-96cd-064e228eb730   20Gi       RWO            rook-ceph-block   83s
wp-pv-claim      Bound    pvc-b09ce46e-d00e-4b7d-8303-748bbb7d0944   20Gi       RWO            rook-ceph-block   83s
[root@master examples]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS      REASON   AGE
pvc-b09ce46e-d00e-4b7d-8303-748bbb7d0944   20Gi       RWO            Delete           Bound    default/wp-pv-claim      rook-ceph-block            86s
pvc-cdfbbd11-a22e-4f72-96cd-064e228eb730   20Gi       RWO            Delete           Bound    default/mysql-pv-claim   rook-ceph-block            86s
[root@master examples]#
Copy the code

When the PVCS containing the StorageClass field are submitted, Kubernetes will create PVS based on the PVCS. This is done using Dynamic Provisioning to dynamically create PVS. The PV supports Static Static requests and dynamic creation.

Log in to the Ceph Dashboard to view the images created

5.5 Installing the Dashboard Visual Panel

Get the dashboard source code from Github

Wget HTTP: / / https://raw.githubusercontent.com/kubernetes/dashboard/v2.4.0/aio/deploy/recommended.yamlCopy the code

For testing purposes, we change the Service type to NodePort. Note that YAML adds a new type of type=NodePort in the Service section below:

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 443
      targetPort: 8443
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard
Copy the code

There is no default field type: NodePort. The service type is Cluster IP.

Then deploy the new version of Dashboard directly:

[root@master ~]# kubectl apply -f recommended.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
[root@master ~]# kubectl get ns
NAME                   STATUS   AGE
default                Active   50m
kube-node-lease        Active   50m
kube-public            Active   50m
kube-system            Active   50m
kubernetes-dashboard   Active   11s
rook-ceph              Active   46m
[root@master ~]# kubectl get svc -nkubernetes-dashboardNAME TYPE cluster-ip external-ip PORT(S) AGE dashboardmetrics-scraper ClusterIP 10.1.213.171 < None > 8000/TCP 32s Kubernetes-dashboard NodePort 10.1.221.14 < None > 443:31712/TCP 32sCopy the code

NodePort is 31712, which can be accessed by combining any Node IP address. https://NodeIP:31712. Because huawei extranet cannot directly access the internal IP address of the VPC, you need to use an external EIP address to access the IP address of an internal Node.

You need to use Token or Kubeconfig to log in.

kubectl create serviceaccount  dashboard-admin -n kube-system
kubectl create clusterrolebinding  dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')
Copy the code

Here is an example of Token:

[root@master ~]# kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')Name: dashboard-admin-token-thf6q Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name: dashboard-admin kubernetes.io/service-account.uid: d6ea3599-19c6-48a9-aa3b-2ec7ce265a24 Type: kubernetes.io/service-account-token Data ==== token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImJQRzl4aF9wMFdRbWE2blp0b1JvN2dVNWhkRkdZVzRpMndLMnhJbks5S00ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3Nlc nZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZ WFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tdGhmNnEiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtY WNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZDZlYTM1O TktMTljNi00OGE5LWFhM2ItMmVjN2NlMjY1YTI0Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9. PlaEmz10kVjQf1zxUSNfiGytP0Ha6hCLuk2fBFM08owjEaFcIWHdRVRsHL6RO0w0i81YG0Gh6x3zJffy_ojhi_M-bCaPSVubPFrZz-CYO7Uia4fYv1P8f5c6 I2X1e_-K2DzCYUlJvI3nzZy-jrFMIz_W19k63rRbxeNrqkdBJpsheWmaT_g8fjAzjtCDEnYUGDDPTVOtEvuhaSC_yci42f7eqTtlR2_QK1Bg2Id0GIEtEXT3 xBgaofWuyjJVEex1mc4LImsdzpVFMtmPum9vEoZzxq1EONhOWxaaFIaadstfM-id9vDNlvZ5O2szk5xVtdgryFi72ICX7x5EpPyOqw ca.crt: 1099 bytes namespace: 11 bytesCopy the code

You can use the above token to log in directly, and you can also use the config file to log in to Dashboard.

Generate kubeconFig file

DASH_TOCKEN=$(kubectl get secret -n kube-system dashboard-admin-token-thf6q -o jsonpath={.data.token}|base64 -d)
Dashboard-admin-token-thf6q is the token name generated aboveKubectl config set-cluster kubernetes --server=192.168.0.11:6443 --kubeconfig=/root/dashbord-admin.confWhere server address is api-server address

kubectl config set-credentials dashboard-admin --token=$DASH_TOCKEN --kubeconfig=/root/dashbord-admin.conf
kubectl config set-context dashboard-admin@kubernetes --cluster=kubernetes --user=dashboard-admin --kubeconfig=/root/dashbord-admin.conf
kubectl config use-context dashboard-admin@kubernetes --kubeconfig=/root/dashbord-admin.conf
Copy the code

The generated dashbord-admin.conf file can be used to log in to the Dashboard.

This article first public account: Mei Xuhong, welcome to pay attention to, update regularly.

The full text.