Author: Yun Min

Rook is introduced

Rook is an open source cloud-native storage choreographer that provides a platform, framework, and support for a variety of storage solutions to integrate natively with cloud-native environments.

Rook transforms distributed storage systems into self-managed, self-expanding, and self-healing storage services. It automates storage administrators’ tasks such as deployment, boot, configuration, configuration, expansion, upgrade, migration, disaster recovery, monitoring, and resource management.

In a nutshell, Rook is a set of Kubernetes operators that have complete control over the deployment, management, and automatic recovery of multiple data storage solutions such as Ceph, EdgeFS, Minio, Cassandra.

By far the most stable storage supported by Rook is still Ceph, and this article will show you how to use Rook to create and maintain Ceph clusters as persistent storage for Kubernetes.

Environment to prepare

The K8s environment can be deployed by installing KubeSphere, and I used a high availability solution.

Install KubeSphere on the public cloud reference: Multi-node installation

⚠️ note: kube-node(5,6,7) has two data disks.

Kube-master1 Ready Master 118D v1.17.9 Kube-master2 Ready Master 118d v1.17.9 Kube-Master3 Ready Master 118d v1.17.9 kube-master3 Ready Master 118d v1.17.9 Kube-node1 Ready worker 118d v1.17.9 kube-node2 Ready worker 118d v1.17.9 kube-node3 Ready worker 111d v1.17.9 Kube-node4 Ready worker 111d v1.17.9 kube-node5 Ready worker 11d v1.17.9 kube-node6 Ready worker 11d v1.17.9 kube-node7 11 d v1.17.9 Ready workerCopy the code

Before installing LVM2, ensure that LVM2 is installed on all Nodes. Otherwise, an error may occur.

Deploy and install Rook and Ceph clusters

1. Clone the Rook repository to a local directory

$ git cloneB - release - 1.4 - https://github.com/rook/rook.gitCopy the code

2. Switch the directory

$ cd /root/ceph/rook/cluster/examples/kubernetes/ceph
Copy the code

3. Deploy Rook and create CRD resources

$ kubectl create -f common.yaml -f operator.yaml
# description:
# 1.comm.yaml is mainly about permission control and CRD resource definition
Yaml is deloyment of rook-ceph-operator
Copy the code

4. Create a Ceph cluster

$kubectl create -f cluster.yaml # Ceph cluster automatically identifies new unformatted free disks on node nodes by default, and automatically initializes OSD for these disks (at least 3 nodes are required, each node has at least one free disk).Copy the code

5. Check the POD status

$ kubectl get pod -n rook-ceph -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES Csi-cephfsplugin-5fw92 3/3 Running 6 12d 192.168.0.31 kube-node7 <none> < None > Csi-cephfsplugin-78plf 3/3 Running 0 12d 192.168.0.134 kube-node1 <none> <none> Csi-cephfsplugin-bkdl8 3/3 Running 3 12d 192.168.0.195 kube -node2 <none> <none> Csi -cephfsplugin-provisioner- 77f457bcb9-6w4CV 6/6 Running 0 12d 10.233.77.95 kube-node4 <none> <none> Csi -cephfsplugin-provisioner-77f457bcb9-q7vxh 6/6 Running 0 12d 10.233.76.156 kube-node3 <none> <none> Csi-cephfsplugin-rqb4d 3/3 Running 0 12d 192.168.0.183 kube-node4 <none> <none> Csi-cephfsplugin-vmrfj 3/3 Running 0 12d 192.168.0.183 kube-node4 <none> <none> Csi-cephfsplugin-vmrfj 3/3 Running 0 12d 192.168.0.91 kube-node3 <none> <none> Csi-cephfsplugin-wglsw 3/3 Running 3 12d 192.168.0.116 kube-node3 <none> <none> Csi-rbdplugin-4m8hv 3/3 Running 0 12d 192.168.0.91 kube-node3 <none> <none> csi-rbdplugin-7wt45 3/3 Running 3 12d 192.168.0.91 kube-node3 <none> <none> Csi-rbdplugin-7wt45 3/3 Running 3 12d 192.168.0.195 kube - node5 < none > < none > csi - rbdplugin - bn5pn 3/3 Running 3 d 12 192.168.0.116 kube - node6 < none > < none > Csi-rbdplugin - hwL4b 3/3 Running 6 12d 192.168.0.31 kube-node7 <none> <none> csi-rbdplugin-provisioner- 7897F5855-7m95p 6/6 Running 0 12d 10.233.77.94 kube-node4 <none> <none> csi-rbdplugin-provisioner-7897f5855-btwt5 6/6 Running 0 12d 10.233.77.94 kube-node4 <none> <none> csi-rbdplugin-provisioner-7897f5855-btwt5 6/6 Running 0 12d 10.233.76.155 kube-node3 <none> <none> csi-rbdplugin-qvksp 3/3 Running 0 12d 192.168.0.183 kube-node4 <none> <none> Csi-rbdplugin-rr296 3/3 Running 0 12d 192.168.0.134 kube-node1 <none> <none> Rook-ceph-crashcollector-kube-node1-64cf6f49fb-bx8lz 1/1 Running 0 12d 10.233.101.46 kube-node1 <none> <none> Rook-ceph-crashcollector-kube 1/1 Running 0 12d kube <none> <none> rook-ceph-crashcollector-kube <none> Rook-ceph-crashcollector-kube-node4-78549d6d7f-9zz5q 1/1 Running 0 8d 10.233.77.226 kube-node4 <none> <none> Rook-ceph-crashcollector-kube-node5-5db855746-b8zp6 1/1 Running 1 11d 10.233.81.239 kube-node5 <none> <none> Rook-ceph-crashcollector-kube-node6-78b796769-8qh45 1/1 Running 0 8d 10.233.66.252 kube-node6 <none> <none> Rook-ceph-crashcollector -kube-node7-78c97898fd -k85L4 1/1 Running 1 8d 10.233.111.33 kube-node7 <none> <none> Rook -ceph-mds-myfs-a-86bdb684b6-4pbj7 1/1 Running 0 8d 10.233.77.225 kube-node4 <none> <none> Rook -ceph-mds-myfs-b-6697d66b7d- JGNKW 1/1 Running 0 8d 10.233.66.250 kube-node6 <none> <none> Rook -ceph-mgr-a-658db99d5b-jbrzh 1/1 Running 0 12d 10.233.76.162 kube-node3 <none> <none> Rook-ceph-a-5cbf5947d8 - VVFGF 1/1 Running 1 12d 10.233.101.44 kube-node1 <none> <none> Rook-ceph-mon-b-6495c96d9d-b82st 1/1 Running 0 12d 10.233.76.144 kube-node3 <none> <none> Rook -ceph-mon-d-dc4c6f4f9-rdfpg 1/1 Running 1 12d 10.233.66.219 kube-node6 <none> <none> Rook -ceph-operator-56fc54bb77-9rswg 1/1 Running 0 12d 10.233.76.138 kube-node3 <none> <none> Rook-ceph-osd-0-777979f6b4-jxtg9 1/1 Running 1 11d 10.233.81.237 kube-node5 <none> <none> Rook-ceph-osd-10-589487764d-8bmpd 1/1 Running 0 8d 10.233.111.59 kube-node7 <none> <none> Rook-ceph-osd-11-5b7dd4c7bc-m4nqz 1/1 Running 0 8d 10.233.111.60 kube-node7 <none> <none> Rook-ceph-osd-2-54cbf4d9d8-qn4z7 1/1 Running 1 10d 10.233.66.222 kube-node6 <none> <none> rook-ceph-osD-6-c94cd566-ndgzd 1/1 Running 1 10d 10.233.81.238 kube-node5 <none> < None > rook-ceph-osD-7-d8cdc94fd-v2LM8 1/1 Running 0 9d 10.233.66.223 Kube-node6 <none> <none> rook-ceph-osd-prepare-kube-node1-4bdch 0/1 Completed 0 66m 10.233.101.91 kube-node1 <none> <none> rook-ceph-osD-prepare-kube-node3-bg4wk 0/1 Completed 0 66m 10.233.76.252 kube-node3 <none> <none> Rook -ceph-osd-prepare-kube-node4-r9dk4 0/1 Completed 0 66m 10.233.77.107 kube-node4 <none> <none> Rook -ceph-osd-prepare-kube-node5- RBVCN 0/1 Completed 0 66m 10.233.81.73 kube-node5 <none> <none> Rook -ceph-osd-prepare-kube-node5- RCNGG 0/1 Completed 5 10d 10.233.81.98 kube-node5 <none> <none> Rook-ceph-osd-prepare-kube-node6-jc8cm 0/1 Completed 0 66m 10.233.66.109 kube-node6 <none> < None > Rook -ceph-osd-prepare-kube-node6- QSXRP 0/1 Completed 0 11d 10.233.66.109 kube-node6 <none> <none> Rook-ceph-osd-prepare-kube-node7-5c52p 0/1 Completed 5 8d 10.233.111.58 kube-node7 <none> <none> Rook-ceph-osd-prepare-kube-node7-h5d6c 0/1 Completed 0 66m 10.233.111.110 kube-node7 < None > < None > Rook -ceph-osd-prepare-kube-node7-tzvp5 0/1 Completed 0 11d 10.233.111.102 kube-node7 <none> <none> Rook-ceph-osd-prepare-kube-node7-wd6dt 0/1 Completed 7 8d 10.233.111.56 kube-node7 <none> <none> Rook-ceph-tools-64fc489556-5clvj 1/1 Running 0 12d 10.233.77.118 kube-node4 <none> <none> rook-discover-6kbvg 1/1 Running 0 12d 10.233.101.42 kube-node1 <none> <none> rook-discover-7dr44 1/1 Running 2 12d 10.233.66.220 kube-node6 <none> <none> rook-discover-dqr82 1/1 Running 0 12d 10.233.77.74 kube-node4 <none> <none> rook-discover- GQPPP 1/1 Running 0 12d 10.233.76.139 kube-node3 <none> <none> rook-discover- HDKXF 1/1 Running 1 12d 10.233.81.236 kube-node5 <none> <none> rook-discover- PZHSW 1/1 Running 3 12d 10.233.111.36 kube-node7 <none> <none>Copy the code

The preceding shows the POD status of all components. The POD starting with rook, ceph, osD-prepare automatically senses that a new hard disk is mounted to the cluster. When a new hard disk is mounted to the cluster, the OSD process is automatically triggered.

6. Configure the Ceph cluster Dashboard

Ceph Dashboard is a built-in Web-based management and monitoring application that is part of the open source Ceph distribution. The Dashboard allows you to obtain various basic status information about the Ceph cluster.

The SVC address of the Ceph-Dashboard installed by default is the Service clusterIP and cannot be accessed externally. You need to create service

$ kubectl apply -f dashboard-external-http.yaml
Copy the code
apiVersion: v1
kind: Service
metadata:
  name: rook-ceph-mgr-dashboard-external-https
  namespace: rook-ceph # namespace:cluster
  labels:
    app: rook-ceph-mgr
    rook_cluster: rook-ceph # namespace:cluster
spec:
  ports:
    - name: dashboard
      port: 7000
      protocol: TCP
      targetPort: 7000
  selector:
    app: rook-ceph-mgr
    rook_cluster: rook-ceph
  sessionAffinity: None
  type: NodePort
Copy the code

Note: Because 8443 is an HTTPS access port that requires a certificate, this tutorial shows that only 7000 is configured for HTTP access port

7. Check the SVC status

$ kubectl get svc -n rook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE csi-cephfsplugin-metrics ClusterIP 10.233.3.172 <none> 8080/TCP,8081/TCP 12d CSI-rbdplugin-metrics ClusterIP 10.233.43.23 <none> 8080/TCP,8081/TCP 12d Rook-ceph-mgr ClusterIP 10.233.63.85 < None > 9283/TCP 12d Rook-ceph-Mgr - Dashboard ClusterIP 10.233.20.159 < None > 7000/TCP 12d rook-ceph-mgr-dashboard-external-https NodePort 10.233.56.73 < None > 700:31357 /TCP 12d rook-ceph-mon-a ClusterIP 10.233.30.222 < None > 6789/TCP,3300/TCP 12d Rook -ceph-mon-b ClusterIP 10.233.55.25 < None > 6789/TCP,3300/TCP 12d Rook -ceph-mon-d ClusterIP 10.233.0.206 < None > 6789/TCP,3300/TCP 12dCopy the code

8. Verify access to the Dashboard

Open KubeSphere platform to open extranet service

Access mode:

http://{master1-ip:31357}
Copy the code

How to obtain a user name:

$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}"|base64 --decode && echo
Copy the code

Note: When a dashboard displays HEALTH_WARN warnings, log in to the seelog to view the causes, such as the OSD node is down or the number of PGS is insufficient

9. Deploy the RoOK toolkit

The Rook toolkit is a container containing common tools for Rook debugging and testing

$ kubectl apply -f toolbox.yaml
Copy the code

Check the Ceph cluster status in the toolkit

$ kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- bash
Copy the code
$ ceph -s cluster: id: 1457045a-4926-411f-8be8-c7a958351a38 health: HEALTH_WARN mon a is low on available space 2 osds down Degraded data redundancy: All objects degraded (crashed), 60 PGS Undersized (Crashed), 50 PGS undersized (crashed) 3 Daemons have recently crashed Services: Mon 3 daemons, quorum a,b,d (age 9d) mgr: a(active, since 4h) mds: myfs:1 {0=myfs-b=up:active} 1 up:standby-replay osd: 12 osds: 6 up (since 8d), 8in(since 8d); 9 remapped pgs data: pools: 5 pools, 129 pgs objects: 53 objects, 37 MiB usage: 6.8 GiB Used, 293 GiB / 300 GiB Avail PGS: D = 25/159 Objects Misplaced (15.723%) 5/159 objects Misplaced (3.145%) 69 active+clean 35 active+undersized 16 active+undersized+degraded 9 active+clean+remappedCopy the code

Toolbox related query commands

ceph status
ceph osd status
ceph df
rados df
Copy the code

Deploy StorageClass

1. Introduction to RBD block storage

Ceph can provide object storage RADOSGW, block storage RBD, and file system storage Ceph FS. RBD is short for RADOS Block Device. RBD Block storage is the most stable and commonly used storage type. RBD block devices like disks can be mounted. RBD block devices provide features such as snapshot, multi-copy, clone, and consistency. Data is stored on multiple OSD nodes in a Ceph cluster in striped mode.

2. Create StorageClass

[root@kube-master1 rbd]# kubectl apply -f storageclass.yaml
Copy the code

3. View the StorageClass deployment status

4. Create a PVC

$ kubectl apply -f pvc.yaml
Copy the code
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbd-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: rook-ceph-block
~
Copy the code

5. Create a POD with PVC

$ kubectl apply -f pod.yaml
Copy the code
apiVersion: v1
kind: Pod
metadata:
  name: csirbd-demo-pod
spec:
  containers:
    - name: web-server
      image: nginx
      volumeMounts:
        - name: mypvc
          mountPath: /var/lib/www/html
  volumes:
    - name: mypvc
      persistentVolumeClaim:
        claimName: rbd-pvc
        readOnly: false

Copy the code

6. Check the POD, PVC, and PV status

conclusion

For the first time contact roOK +Ceph deployment experience students need to understand more content, encountered more pits. Hope that through the above deployment process records can help you.

1. The Ceph cluster displays a message indicating that no disk is available for OSD

A: There are several cases. Check whether the mounted data disk has been used before. Although formatted, the previous RAID information still exists. You can use the following script to clean up after formatting in mount.

#! /usr/bin/env bash
DISK="/dev/vdc"  # Modify your drive information as needed

# Zap the disk to a fresh, usable state (zap-all is important, b/c MBR has to be clean)

# You will have to run this step for all disks.
sgdisk --zap-all $DISK

# Clean hdds with dd
dd if=/dev/zero of="$DISK" bs=1M count=100 oflag=direct,dsync

# Clean disks such as ssd with blkdiscard instead of dd
blkdiscard $DISK

# These steps only have to be run once on each node
# If rook sets up osds using ceph-volume, teardown leaves some devices mapped that lock the disks.
ls /dev/mapper/ceph-* | xargs -I% -- dmsetup remove %

# ceph-volume setup can leave ceph-<UUID> directories in /dev and /dev/mapper (unnecessary clutter)
rm -rf /dev/ceph-*
rm -rf /dev/mapper/ceph--*

# Inform the OS of partition table changes
partprobe $DISK
~
Copy the code

2. What storage types does Ceph support?

A: RDB block storage, CEPHFS file storage, S3 object storage, etc

3. How to check all kinds of pits in deployment?

A: It is strongly recommended to check relevant documents on roOK and CEPH official website for troubleshooting

  • rook.github.io/docs/rook/
  • docs.ceph.com/en/pacific/

4. Failed to access the Dashboard

A: If it is KubeSphere or K8s built by public cloud, please allow nodeport in the security group

This article is published by OpenWrite!