HDFS architecture, operations, and management

1. HDFS Architecture

The following figure shows the HDFS architecture:

Namespace: Contains the hierarchy of file systems. Journaling: Protects consistency of data written to the file system. Changes to the file system are persisted to disk’s **fsimage_

and Edits_

** files. Blocak Map: maps file name and data block ID. DataNode: stores HDFS file data. Usually, multiple disks are used to improve I/O throughput. SecondaryNameNode: Reduce the CheckPoint operation of the primary NameNode; Provide redundancy in case the primary NameNode hangs;

Write data to HDFS & read data

The following figure shows how a file is written to HDFS:

Process:

  • Step 1: HDFS Client sends a request to NameNode to write a file like HDFS.
  • Step 2: The NameNode verifies that the requested path and file name are valid. If yes, the NameNode sends a lease to the Client.
  • Step 3: Clien verifies the default HDFSdata block sizeRequest block IDs, a list of Datanodes corresponding to each block. Applications can request different block sizes.
  • Step 4: NameNode sends the corresponding information to the Client
  • Step 5-11: Write the data and checkSum to the DataNodes and send the response to the Client.
  • Step 12: When all data blocks are written to disk, the Client notifies NameNode that the write operation is complete.

The following figure shows how to read files in HDFS:

Location of copy and block

When a disk, system, or network fault occurs, the HDFS automatically and clearly copies data to different Datanodes. A file is split into one or more data blocks. The default blocksize is 128 MB, which is determined by dfs.blocksize in hdfs-site. XML. Any Client can override the default block size. If the file is larger than the default block size, it will be divided into multiple blocks (for example, a file of 1 GB will be divided into eight blocks). If the file is smaller than the default block size, only the required space is consumed (e.g., a 1-byte file consumes only one byte of space, plus a checksum file). Generally, HDFS requires blocks of the same file to be the same size. Since HDP2.3, HDFS supports different block sizes in the same file. This improvement can be used in a particular case: combining several small files into one large file. For example, this is done as part of the YARN log aggregation feature.

The following is the copy block storage format:

DataNode and NameNode storage

DataNode stores HDFS data in Disk, and HDFS data blocks are presented as files in DataNode local file system. Hdfs-site. XML: dfs.datanode.data.dir Determines the parent directory for storing HDFS data blocks. This property can contain a parent directory separated by commas, enabling a Datanode to use multiple physical disk data blocks for storage. (e.g. /hadoop/ HDFS /data1, /hadoop/ HDFS /data2)

Blocks of data can be scattered across multiple directories. When the number of data blocks in a directory exceeds 64, a new statistics directory is created to ensure HDFS read performance. Each block has a unique ID, which is used by NameNode to locate and read data. Each data block is stored as a file in the local file system of DataNode. The file name is based on the ID number. To ensure data integrity, each data block has one or more checksums. Each 512-byte block has a checkSum. This sum is determined by dfs.bytes-per-checksum in hdFS-site. XML and cannot be modified. Each time a block of data is read, the checksum is recalculated for comparison checksum.

The DataNode stores

The NameNode storage

NameNodes persist HDFS storage state information to disk. The value recorded in The DFs.namenode.name. dir property in the hdfs-site.xml file determines the parent directory of the persisted information. This property can contain a comma-separated list of multiple directories. For example, one directory could be located on a local disk while a second directory could map to a directory mounted from an NFS server. This would provide redundancy for this critical information. Configuring NameNode High Availability (HA) is a Popular method to achieve NameNode redundancy. NameNode HA is described in a another lesson. The number of past edits Files to retain is controlled by the DFS. The namenode. Num. Extra. The edits. The retained property in HDFS - site. XML. The number of past fsimage checkpoint files to retain is controlled by the dfs.namenode.num.checkpoints.retained property in hdfs-site.xml.