Reporter: each alibaba cloud original reader friend everybody is good, met with everybody again. Today is our old friend “Ali cloud container service ACK release” last visit to explore the mystery of life series column, in the previous interview, it brought us a wonderful explanation, interested friends welcome to review. Since its launch in December last year, the Container Services ACK release has received a lot of attention, support and downloads. What do you think about this?

ACK Distro: Yes, I’m lucky to have received over 400 downloads since its launch three months ago. I have communicated with you about technology in different ways. Thank you for your attention and I hope you can enjoy better container service experience.

Reporter: Well, let’s get down to business. We’ve learned that Sealer helps you quickly build and deploy, and hybridnet helps you to build hybrid cloud unified network planes.

ACK Distro: We all know that stateful applications in the cloud native context need a storage solution to persist data. Compared with distributed storage, local storage is superior in cost, ease of use, maintainability and IO performance. Therefore, today I will explain to you alibaba’s open source local storage management system, open-Local, and how I use it to play with container local storage. Local storage as a low-cost delivery of Kubernetes cluster, there are still many problems:

• Kubernetes lacks storage resource awareness: As a “non-standard” resource, local storage is much less supported in Kubernetes than standard resources (CPU, memory, etc.). Using local storage requires certain labor costs, such as limiting Pod scheduling by marking nodes, manually managing disks of different models, manually mounting specified disks to containers by Hostpath, etc. At the same time, there are some private software delivery problems, such as binding the wrong host path so that the failure can not be found in time, these seriously affect Kubernetes delivery efficiency and the stability of the application runtime;

• Lack of isolation capability of local storage space: Improper mounting of an application to a host directory (such as mounting to the root path of the host) causes host faults, such as no response during container operation, TRIGGERING Pod expulsion, and INTERACTION between I/O between PODS due to full disk of application data;

• Kubernetes does not support local storage for stateful applications: Node retention cannot be achieved through Hostpath, resulting in application data loss after Pod drift; The use of semi-automatic static Local PV can ensure that the node is maintained, but can not achieve automatic, still need human participation (such as creating folder path, marking nodes, etc.); Some advanced storage capabilities (such as snapshots) are unavailable.

Open-local avoids these problems to the greatest extent and gives everyone a better experience. Using local storage on Kubernetes is just as easy as using centralized storage.

Architecture composition of open-local

Reporter: Can you tell us more about the architectural components of open-local?

ACK Distro: Of course, open-local consists of four components:

1. The scheduler – extender: As an extension component of KuBE-Scheduler, Extender extends the native scheduler’s perception of local storage resources to realize scheduling decisions on information including disk capacity, multi-disk perception, disk media (SSD or HDD), and achieve mixed scheduling of storage resources.

Csi-plugin: local disk management capability that complies with the Container Storage Interface (CSI) standard, including the ability to create, delete, and expand Storage volumes, create and delete snapshots, and expose metrics of Storage volumes.

3. Agent: Each node running in the cluster initializes storage devices according to the configuration list and reports information about local storage devices in the cluster for scheduler-Extender decision scheduling.

4. Controller: Obtains the initial storage configuration of the cluster and delivers the detailed resource configuration list to the Agent running on each node.

Open-local contains two CRDS:

  1. NodeLocalStorage: open-local Reports storage device information on each node through the NodeLocalStorage resource. This resource is created by the controller and its status is updated by the Agent of each node. The CRD is a globally scoped resource.
  2. NodeLocalStorageInitConfig: open – the local controller through NodeLocalStorageInitConfig resources to create each NodeLocalStorage resources. NodeLocalStorageInitConfig resources contained in the global default node configuration and specific node configuration, if the nodes of the node label content expression is using a specific node configuration, or use the default configuration.

Its architecture diagram can be seen below:

Open-local application scenario

Reporter: So what kind of requirements scenarios do people use open-local for?

ACK Distro: I’ve put together a few use cases that you can fit into your own.

  1. The application expects data volumes to be capable of capacity isolation to avoid problems such as full system disks filled with logs.
  2. Applications require a large amount of local storage resources and rely on nodes, such as Hbase, ETCD, ZooKeeper, and ElasticSearch.
  3. The number of local disks in a cluster is large. Therefore, the scheduler is required to implement automatic deployment of stateful applications.
  4. You can back up instantaneous data for database applications by storing snapshots.

How to use open-local in ACK Distro

Reporter: Now comes the old question, how do you reflect the advantages of open-local? Or how can you use open-local to achieve best practices?

ACK Distro: I’ll break it down for you

1. Initialize the Settings

First of all to ensure that the environment has been installed the LVM tools, installed in the deployment of I will be the default open – local, edit NodeLocalStorageInitConfig resources, the initial configuration for storage.

# kubectl edit nlsc open-local
Copy the code

To use open-local, volumegroups (VGS) must exist in the environment. If VGS already exist in your environment and have free space, you can add them to the whitelist. If no VG exists in the environment, you need to provide a block device name for open-local to create VG.

apiVersion: csi.aliyun.com/v1alpha1
kind: NodeLocalStorageInitConfig
metadata:
  name: open-local
spec:
  globalConfig: NodeLocalStorage is a global default node configuration that is populated into its Spec when it is initialized to create NodeLocalStorage
    listConfig:
      vgs:
        include: # VolumeGroup Whitelist that supports regular expressions
        - open-local-pool-[0-9]+
        - your-vg-name If VGS exist in the environment, you can whitelist them and manage them by open-local
    resourceToBeInited:
      vgs:
      - devices:
        - /dev/vdc  # If there is no VG in the environment, the user needs to provide a block device
        name: open-local-pool-0 Initialize the block device /dev/vdc to VG named open-local-pool-0
Copy the code

NodeLocalStorageInitConfig resource editor has been completed, the controller and agent will update all nodes NodeLocalStorage resources.

2. The storage volume is dynamically supplied

Open-local By default, some storage class templates are deployed in the cluster. I use open-local-lVM, open-local-lVM-xfs, and open-local-lVM-io -throttling as examples.

# kubectl get sc
NAME                           PROVISIONER            RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
open-local-lvm                 local.csi.aliyun.com   Delete          WaitForFirstConsumer   true                   8d
open-local-lvm-xfs             local.csi.aliyun.com        Delete          WaitForFirstConsumer   true                   6h56m
open-local-lvm-io-throttling   local.csi.aliyun.com   Delete          WaitForFirstConsumer   true
Copy the code

Create a Statefulset that uses the open-local-lVM storage class template. In this case, the file system of the storage volume is ext4. If the open-local-lvM-xfs storage template is specified, the storage volume file system is XFS.

# kubectl apply -f https://raw.githubusercontent.com/alibaba/open-local/main/example/lvm/sts-nginx.yaml
Copy the code

Check the Pod/PVC/PV status. The storage volume is created successfully:

# kubectl get pod
NAME          READY   STATUS    RESTARTS   AGE
nginx-lvm-0   1/1     Running   0          3m5s
# kubectl get pvc
NAME               STATUS   VOLUME                                       CAPACITY   ACCESS MODES   STORAGECLASS     AGE
html-nginx-lvm-0   Bound    local-52f1bab4-d39b-4cde-abad-6c5963b47761   5Gi        RWO            open-local-lvm   104s
# kubectl get pv
NAME                                         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS    AGE
local-52f1bab4-d39b-4cde-abad-6c5963b47761   5Gi        RWO            Delete           Bound    default/html-nginx-lvm-0   open-local-lvm  2m4s
kubectl describe pvc html-nginx-lvm-0
Copy the code

3. Expand the storage volume

Edit corresponding PVC spec. Resources. Requests. Storage field, will the storage size of PVC statement from 5 gi expanded to 20 gi.

# kubectl patch pvc html-nginx-lvm-0 -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
Copy the code

Check the STATUS of PVC/PV:

# kubectl get pvc
NAME                    STATUS   VOLUME                                       CAPACITY   ACCESS MODES   STORAGECLASS     AGE
html-nginx-lvm-0        Bound    local-52f1bab4-d39b-4cde-abad-6c5963b47761   20Gi       RWO            open-local-lvm   7h4m
# kubectl get pv
NAME                                         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                           STORAGECLASS     REASON   AGE
local-52f1bab4-d39b-4cde-abad-6c5963b47761   20Gi       RWO            Delete           Bound    default/html-nginx-lvm-0        open-local-lvm            7h4m
Copy the code

4. Storage volume snapshot

Open-local provides the following snapshot classes:

# kubectl get volumesnapshotclass
NAME             DRIVER                DELETIONPOLICY   AGE
open-local-lvm   local.csi.aliyun.com   Delete           20m
Copy the code

Create VolumeSnapshot resource:

# kubectl apply -f https://raw.githubusercontent.com/alibaba/open-local/main/example/lvm/snapshot.yaml
volumesnapshot.snapshot.storage.k8s.io/new-snapshot-test created
# kubectl get volumesnapshot
NAME                READYTOUSE   SOURCEPVC          SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS    SNAPSHOTCONTENT                                    CREATIONTIME   AGE
new-snapshot-test   true         html-nginx-lvm-0                           1863          open-local-lvm   snapcontent-815def28-8979-408e-86de-1e408033de65   19s            19s
# kubectl get volumesnapshotcontent
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER                VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT      AGE
snapcontent-815def28-8979-408e-86de-1e408033de65   true         1863          Delete           local.csi.aliyun.com   open-local-lvm        new-snapshot-test   48s
Copy the code

Create a Pod with the same volume data as the previous snapshot point:

# kubectl apply -f https://raw.githubusercontent.com/alibaba/open-local/main/example/lvm/sts-nginx-snap.yaml
service/nginx-lvm-snap created
statefulset.apps/nginx-lvm-snap created
# kubectl get po -l app=nginx-lvm-snap
NAME               READY   STATUS    RESTARTS   AGE
nginx-lvm-snap-0   1/1     Running   0          46s
# kubectl get pvc -l app=nginx-lvm-snap
NAME                    STATUS   VOLUME                                       CAPACITY   ACCESS MODES   STORAGECLASS     AGE
html-nginx-lvm-snap-0   Bound    local-1c69455d-c50b-422d-a5c0-2eb5c7d0d21b   4Gi        RWO            open-local-lvm   2m11s
Copy the code

5. Native block devices

Open-local Storage volumes can be mounted to containers in the form of block devices (in this example, block devices are in the /dev/sdd path of containers) :

# kubectl apply -f https://raw.githubusercontent.com/alibaba/open-local/main/example/lvm/sts-block.yaml
Copy the code

Check Pod/PVC/PV status:

# kubectl get pod
NAME                READY   STATUS    RESTARTS   AGE
nginx-lvm-block-0   1/1     Running   0          25s
# kubectl get pvc
NAME                     STATUS   VOLUME                                       CAPACITY   ACCESS MODES   STORAGECLASS     AGE
html-nginx-lvm-block-0   Bound    local-b048c19a-fe0b-455d-9f25-b23fdef03d8c   5Gi        RWO            open-local-lvm   36s
# kubectl describe pvc html-nginx-lvm-block-0
Name:          html-nginx-lvm-block-0
Namespace:     default
StorageClass:  open-local-lvm
...
Access Modes:  RWO
VolumeMode:    Block Mount the load container as a block device
Mounted By:    nginx-lvm-block-0
...
Copy the code

6. IO current limit

Open-local Supports I/O traffic limiting for the PV. The following storage class templates support I/O traffic limiting: apiVersion: storage.k8s. IO /v1 kind: StorageClass Metadata: name: open-local-lvm-io-throttling provisioner: local.csi.aliyun.com parameters: csi.storage.k8s.io/fstype: ext4 volumeType:"LVM"
  bps: "1048576" Read/write throughput is limited to 1024KiB/s
  iops: "1024"   The IOPS limit is around 1024
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
Copy the code

Create a Statefulset that stores the class template using open-local-lVM-io-throttling.

# kubectl apply -f https://raw.githubusercontent.com/alibaba/open-local/main/example/lvm/sts-io-throttling.yaml
Copy the code

After the Pod is in the Running state, it enters the Pod container:

# kubectl exec -it test-io-throttling-0 sh
Copy the code

In this case, the storage volume is mounted to /dev/sdd as a native block device. Run the fio command:

# fio -name=test -filename=/dev/sdd -ioengine=psync -direct=1 -iodepth=1 -thread -bs=16k -rw=readwrite -numjobs=32 -size=1G -runtime=60 -time_based -group_reporting
Copy the code

The read/write throughput limit is around 1024KiB/s, as shown below:

. Run status group 0 (alljobs): READ: Bw =1024KiB/s (1049kB/s), 1024KiB/ s-1024kib /s (1049kB/ s-1049KB /s), IO =60.4MiB (63.3MB), run=60406-60406msec WRITE: Bw =993KiB/s (1017kB/s), 993KiB/ s-993kib /s (1017kB/ s-1017KB /s), IO = 58.6MIB (61.4MB), RUN =60406-60406msec Disk stats (read/write): dm-1: Ios =3869/3749, merge=0/0, Ticks =4848/17833, in_queue=22681, util=6.68%, aggrios=3112/3221, aggrmerge=774/631, Aggrticks = 3921/13598, aggrin_queue = 17396, aggrutil = 6.75% VDB: Ios =3112/3221, merge= 744/631, ticks=3921/13598, in_queue=17396, util=6.75%Copy the code

7. Temporary volume

Open-local Supports creating a temporary volume for Pod. The life cycle of a temporary volume is the same as that of Pod. That is, a temporary volume is also deleted after Pod is deleted. This is the open-local version of emptydir.

# kubectl apply -f ./example/lvm/ephemeral.yaml
Copy the code

The results are as follows:

# kubectl describe po file-server
Name:         file-server
Namespace:    default
......
Containers:
  file-server:
    ......
    Mounts:
      /srv from webroot (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dns4c (ro)
Volumes:
  webroot:   # This is CSI temporary volume
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            local.csi.aliyun.com
    FSType:
    ReadOnly:          false
    VolumeAttributes:      size=2Gi
                           vgName=open-local-pool-0
  default-token-dns4c:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dns4c
    Optional:    false
Copy the code

8. Monitor the market

Open-local provides the large-sized monitoring function. You can use Grafana to view local storage information of a cluster, including storage devices and volumes. As shown below:

ACK Distro: In a word, open-local can reduce labor costs in operation and improve the stability of cluster operation. In terms of functions, it maximizes the advantages of local storage, so that users can not only experience the high performance of the local site, but also enrich the application scenarios with various advanced storage features, so that the majority of developers can experience the bonus brought by cloud native, and realize the key step of cloud native deployment of applications, especially stateful applications.

Reporter: Thank you, ACK Distro, for explaining Distro to us. We’ve learnt a lot about him and his Distro friends over these three visits, and I hope they can help you as you read these Distro articles.

ACK Distro: Yes, Distro and I welcome harassment in the GitHub community and community!

A link to the

[1] Open source repository address: github.com/alibaba/ope… [2] ACK Distro’s official website: www.aliyun.com/product/ali… [3]ACK Distro official GitHub: github.com/AliyunConta… [4] let innovation within reach, ali cloud container services ACK distributions of open and free download: mp.weixin.qq.com/s/Lc2afj91s… [5] the first bomb depth interview: mp.weixin.qq.com/s/wB7AS52vE… [6] second depth interview: mp.weixin.qq.com/s/O095yS5xP…