1. The background

Ceph is a decentralized distributed storage system that provides high performance, reliability and scalability. The Ceph project has its roots in Sage’s PhD work (the first results were published in 2004) and has since been contributed to the open source community under the LGPL. After several years of development, cloud computing is now supported by many cloud computing vendors (OpenStack, CloudStack, OpenNebula, Hadoop) and is widely used.

2. Introduction

Ceph is a reliable distributed storage system with automatic rebalancing and recovery. Ceph can be divided into object storage, block devices, and file system services according to different scenarios.

The main advantage of Ceph is distributed storage. When each data is stored, the location of the data is calculated and the data distribution is balanced as far as possible. There is no traditional single point of failure and it can be horizontally expanded.

A Ceph storage cluster requires at least one Ceph Monitor and two OSD daemons. To run the Ceph file system client, a Metadata Server is required.

3. The instructions

Whether it is to provide Ceph object storage or Ceph block devices for cloud platforms, or to deploy a Ceph file system, the deployment of all Ceph storage clusters starts with Ceph nodes, networks, and Ceph storage clusters. A Ceph storage cluster requires at least one Ceph Monitor and two OSD daemons. To run the Ceph file system client, a Metadata Server is required.

  • Ceph OSDsCeph OSD daemons store data, copy, restore, backfill, and rebalance data, and provide monitoring information for Ceph Monitors by checking the heartbeats of other OSD daemons. If a Ceph storage cluster has two copies, at least two OSD daemons are required for the cluster to have two copiesactive+cleanStatus (Ceph has 3 copies by default and can be adjusted).
  • Monitors: The Ceph Monitor maintains charts that show cluster status, including Monitor, OSD, Home Group (PG), and CRUSH charts. Ceph keeps historical information (called epoch) about each state change that occurred on Monitors, OSD, and PG.
  • MDSs: Ceph metadata server (MDS) stores metadata for Ceph file system. The metadata server enables POSIX file system users to perform such operations without overloading Ceph storage clustersls,findAnd other basic commands.

Ceph stores client data as objects in a storage pool. Using the CRUSH algorithm, Ceph can calculate which home group (PG) should hold the specified Object, and then further calculate which OSD daemon holds the home group. The CRUSH algorithm enables Ceph storage clusters to scale, rebalance, and repair dynamically.

Characteristics of 4.

Ceph is suitable for small file storage across clusters with the following features:

  • A high performance

    The Client communicates with the Server directly without proxy or forwarding.

    The Client does not need to be responsible for the replication of copies. The Primary node is responsible for the replication, which can effectively reduce the consumption of clien network.

    CRUSH algorithm, balanced data distribution, high parallelism, support thousands of storage nodes, support TB and PB level data.

  • High availability

    Multiple data copies, support fault domain separation, strong data consistency;

    No single point of failure, good fault tolerance, effectively support a variety of fault scenarios;

    Support all fault detection and automatic recovery, can achieve automatic management;

    Parallel recovery greatly reduces the data recovery price and improves data reliability.

  • High scalability

    Highly parallel design, there is no single central control component, all load can be dynamically divided to each server.

    Decentralized, flexible, linear growth with the increase of nodes.

  • The scene is rich

    Supports three storage port types: block storage, file storage, and object storage. At the same time support custom interface, C++ for the bottom implementation, compatible with a variety of languages.

    • Block storage: Maps disk space to hosts, which is suitable for docker containers and VM disk storage allocation. Log storage, file storage.

    • File storage: To solve the block storage failure, set up FTP and NFS servers for directory storage and log storage.

    • Object storage: Large-capacity hard disks. Installed with storage management software, object storage provides read/write capabilities, high-speed read/write capabilities of block storage, and file storage sharing. Suitable for picture storage or video storage.

5. Compare distributed storage systems

Comparison shows that TFS FASTDFS MooseFS GlusterFS CEPH
Development of language C++ C C C C++
Data storage mode block File/Trunk block File/piece Object/file/block
Online expansion support support support support support
Redundancy backup support support support support support
A single point of failure There are There is no There are There is no There is no
Ease of use Installation is complicated and official documents are few Easy to install and relatively active community Easy to install, official documents Simple installation, professional official documents Simple installation, professional official documents
Applicable scenario Small files across clusters Small and medium files in a single cluster Single cluster of large and medium files Cross-cluster cloud storage Large, medium and small files in a single cluster

This article was created and shared by Mirson. For further communication, please add to QQ group 19310171 or visit www.softart.cn