This is the 12th day of my participation in Gwen Challenge

background

Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop Hadoop

Distributed storage system

Routing algorithms for different distributed storage systems

  1. Both Swift and Cassandra use consistent hashing algorithm to do routing

The read/write speed is relatively fast, but if adding or deleting machines will cause data movement, the consistent hashing algorithm and the virtual node mechanism can greatly reduce the data movement

  1. HBase and HDFS use the central route to locate data

Adding or removing nodes does not cause data movement, but for data balancing, load balancing is often performed in production, which also causes data block movement

My understanding of central routing algorithm:

  1. NameNode maintains a network topology, a cluster of computers with a tree-like network topology. For example, a cluster may consist of multiple Data centers with multiple racks for computing purposes.
  2. Then according to the topology to get the need to route to which nodes

Compare data write methods

  1. GlusterFS and Swift write to disk per file unit for both large and small files, so performance can be relatively poor for small files because large random writes cost disk performance
  2. Cassandra is characterized by distributed database (NOSQL). For files, it is mainly suitable for storing small files. Three levels of write are adopted on the storage side: first, log is written to commitlog; The data is then written to memory, and when certain conditions (data size, time, number of keys) are reached, the data is flushed to disk. This is a common way of processing small files in the industry: For example, Hbase, BookKeeper (Yahoo), KeyStack (Facebook), and TFS (Taobao) all use this method to convert random write to sequential write, providing small file write performance
  3. Glusterfs, Swift, and HDFS do not provide small file solutions, so small file support is poor. In addition, HDFS has a high number of write file communication times and strong consistency, which also leads to poor small file performance. It is worth mentioning that HDFS and Mysql append operation logs to a file, which also improves write performance by sequential writing

The NWR theory

NWR is a policy for controlling consistency levels in distributed storage systems. In Amazon’s Dynamo cloud storage system, NWR is used to control consistency. Each letter has the following meanings:

  • N: number of replicas in the same data
  • W: is the number of copies that must be successfully updated when updating a data object
  • R: Number of replicas to be read to read a data

In distributed systems, single points of data are not allowed. That is, if the number of replicas in the line is 1, it is very dangerous, because if the Replica is wrong again, permanent data errors may occur

Consistency handling of read and write data

GlusterFS

  1. When writing: Final consistency is used; Write files directly to disk
  2. Read time: Hash locates all copies of the data; Check whether data needs to be restored before reading data. Read the copy using the RR algorithm

Swift

  1. Write: weak consistency, final consistency; Write files directly to disk
  2. Read: NRW (W+R>N)

Cassandra

  1. Write: Strong consistency and final consistency; The files are written to the disk as logs and to the memory at the same time. When the data reaches a certain condition flush to the disk (same as hbase)
  2. Read: Quorum+NRW (W+R>N); When reading, check whether the data needs to be repaired

Data deletion comparison

GlusterFS deletes files directly, while Swift and Cassandra use tag first and then delete periodically, which is better in terms of system response time, as well as Haystack, TFS and others. HDFS also uses a similar flag first. You can also choose to put it into the recycle bin. If you do not put it into the recycle bin, it will be deleted as soon as possible. In fact, for small files, you can also choose to delete them in the time period when the system is not busy (because small files themselves occupy less space