The introduction

With the increasing popularity of cloud native technology today, running stateless applications on Kubernetes has been very mature, smooth expansion ability is also very strong, but for stateful applications, data need persistent storage, which still has a lot of room for improvement, facing many challenges.

Challenges of cloud native storage

Above is a CNCF survey of “challenges encountered in using/deploying containers.” According to the results of the report, challenges encountered by izumo native storage can be summarized in the following aspects:

  • Ease of use: Complex storage service deployment, operation and maintenance, low degree of cloud primitive, lack of integration with mainstream orchestration platforms
  • High performance: A large number of applications have I/O access, resulting in high IOPS requirements and low latency
  • High availability: The cloud native storage has been applied to the production environment and must be highly reliable and available without single points of failure
  • Agility: RAPID PV creation, destruction, smooth expansion/contraction, RAPID PV migration with Pod migration, etc

Common cloud native storage solutions

Rook-ceph: Rook-Ceph is an Operator that provides Ceph cluster management capabilities to perform its responsibilities using the capabilities provided by the underlying cloud native container management, scheduling, and choreography platform.

OpenEBS: The OpenEBS storage controller itself runs in a container. OpenEBS Volume consists of one or more containers that operate as microservices.

advantage

1. Integration with cloud native choreography system, with good container data volume access ability; 2. Fully open source, relatively active community, rich network resources and use materials, easy to start;

disadvantage

Rook – Ceph inadequate:

  • Poor performance: Poor I/O performance, throughput, and latency, which is difficult to apply to high-performance service scenarios.
  • High maintenance costs: although deployment and entry are simple, there are many components, complex architecture and difficult troubleshooting. Once problems occur in operation, it is very difficult to solve, requiring a strong technical team to ensure;

Openebs-hostpath insufficient: No high availability function, single point of failure; Openebs-zfs-localpv (openebs-zfs-localPV, openebs-zfs-localPV, openebs-zfs-localPV, openebs-zfs-localPV)

Therefore, it is mostly used in the internal test environment of the enterprise, but rarely used to persist critical application data and deploy it to the production environment.

Why Is NeonIO good for cloud native storage

NeonIO profile

NeonIO is an enterprise-class distributed block storage system that supports containerized deployment, Kubernetes platform provides Dynamic Provisioning for Persistent volumes, supports Clone, Snapshot, Restore, Resize, and other functions. NeonIO looks like this:

NeonIO includes the following service components:

  • Zk/ETCD: provides services such as cluster discovery, distributed coordination, and master selection
  • Mysql: provides metadata storage services, such as PV storage volume metadata
  • Center: provides logical management services, such as creating PV volumes and snapshots
  • Monitor: Provides monitoring services that expose collection monitoring metrics to Promethus
  • Store: a storage service that processes I/O functions of applications
  • Portal: provides UI services
  • CSI: provides CSI standard IO access services

Here’s why NeonIO is a good fit for cloud native storage:

Ease of use

  1. Component containerization: service component, CSI, Portal containerization

  2. CSI support: provides standard I/O access capabilities, and supports static and dynamic PV creation

  3. UI interface, convenient operation and maintenance:

    • Storage operation and maintenance interface, alarm and monitoring visual management;
    • Performance monitoring based on PV granularity, such as IOPS and throughput, can quickly locate hotspot PVS.
    • There are Qos based on PV granularity, which can guarantee high priority quality of service for users.

  1. Highly integrated with cloud native:

    • Promethus is supported. Acquisition indicators of NeonIO are exposed to Promethus and Grafana for graphical display through ServiceMonitor
    • At the same time, Promethus can be connected to the UI to display other cloud native monitoring indicators, such as Node-Exporter’s DISK I/O load and bandwidth
    • Platform-based operation and maintenance, storage expansion, upgrade, disaster recovery operation and maintenance operations, only need Kubernetes some commands can be achieved, do not need to master too much storage related operation and maintenance knowledge
    • Service discovery, distributed coordination support ETCD, metadata management, using CRD

  2. One-click deployment: : helm install neonio./neonio –namespace kube-system

  1. Simple and flexible deployment: Compared to Rook-Ceph:
function NeonIO Rook-Ceph
Planning Node Deployment Label the node By modifying cluster.yaml, you need to configure the node IP to configure those services
Quick Start There are 4 steps:

1. Check to ensure there is equipment available for Neonio;

2. Check whether QBD has been installed.

3. Add helm repo.

Helm install neonio./neonio –namespace kube-system		 A total of five steps

1. Check to ensure there is equipment available for CEPH;

2. Check whether the RBD has been installed.

3.apt-get install -y lvm2

Git Clone –single-branch –branch master

Github.com/rook/rook.g…

5.cd

rook/cluster/examples/kubernetes/ceph

kubectl create -f crds.yaml -f common.yaml -f operator.yaml

kubectl create -f cluster.yaml
Single machine all – in – one helm install neonio ./neonio –namespace kube-system –set sc.rep_count=1 –set center.servers=1 — cd

rook/cluster/examples/kubernetes/ceph

kubectl create -f crds.yaml -f common.yaml -f operator.yaml

kubectl create -f cluster-test.yaml

Yaml, which is different from cluster-test.yaml, cannot be shared
RDMA/TCP helm install neonio ./neonio –namespace kube-system –set store.type=RDMA Ceph itself supports RDMA, rook-CEPh does not
Management and storage networks are separated or shared helm install neonio ./neonio –namespace kube-system –set store.port=eth0 –set rep_port.port=eth1 Ceph itself pubIC, cluster network port separation public, Rook-CEPh adaptation complex

A high performance

Performance single PV IOPS 100K, sub-millisecond delay.

  1. Distributed storage architecture with full flash

    • The I/O performance increases linearly with the number of nodes in a cluster
    • Storage media support NVME SSDS
    • RDMA support: Nodes are connected via high-speed RDMA technology

  2. Extremely short I/O paths: Discard the file system and develop the metadata management system to make the I/O paths extremely short

  1. The HostNetwork mode is used

Benefits:

  • Store CSI Pod uses HostNetwork, directly using physical network, reducing network layers
  • The management network, front-end network, and data synchronization network are separated to avoid network competition.

High availability

  1. Service component reliability and availability

    • The management service uses 3 copies of Pod by default. The number of copies can be configured. It is recommended to use 3/5 copies
    • Use the probe to detect whether the Pod service is available and alive. When the Pod service department is detected, the component service can be removed. When the Pod dies, restart the Pod to restart the service

  2. Data reliability and availability

  • The Volume fragment is Shard
  • Each Shard selects a storage location independently
  • Three copies of each Shard are stored on different physical nodes
  • When writing, write three copies synchronously, strong consistency
  • Read only from the master copy
  • The number of copies can be set by volume

agility

  1. Pod cross-node reconstruction efficiency: 16s for mount/unload of 2000PV
  2. Batch PV creation capability: creates 2000 PVS in 5 minutes

NeonIO performance

Teststand: NeonIO hyper-converged All-in-one cluster (3 Nodes, 192.168.101.174-192.168.101.176)

Note: All tests use NVMe SSDs. Volume size = 1tib. Performance Tool: github.com/leeliu/dben…

The yellow graph is NeonIO. The first graph is IOPS and the second graph is milliseconds. NeonIO has a clear advantage in IOPS and latency regardless of whether it is a single copy or three copies.

NeonIO application scenarios

  1. Devops scenario: Rapid batch creation and destruction of PVS (five minutes) for 2000 PVS
  2. Database scenario: the WEB back-end database, such as MySQL, provides stable persistent storage with high IOPS and low latency
  3. Big data application analysis scenario: provides large capacity (PV can be expanded to 100TB)

This article is published by OpenWrite!