Why develop an HDFS file system in a Hadoop project? Better support for distributed computing.

Storage model Architecture Design Role Function Metadata persistence security mode Copy placement policy Read/write process security policy

Storage model: Files are linearly divided into byte blocks with offset and ID. Each file can have different block sizes except for the last block in a file. The size of each block is adjusted according to the I/O characteristics of the hardware. Replication is critical to reliability and performance. File upload Allows you to specify the block size and the number of copies. After upload, you can only modify the number of copies

Architecture design: HDFS is a Master/Slaves architecture consisting of a NameNode and datanodes. The NameNode stores and manages file metadata and maintains a hierarchical file directory tree. The DataNode stores file data (blocks). The Datanodes maintain heartbeat communication with NameNode and report the block information they hold. The Client and NameNode interact file metadata and datanodes interact file block data

Role functions: NameNode stores file metadata, directory structure, and mapping file blocks based on the memory. The solution requires persistence to ensure data reliability. Provides a copy placement policy Keep heartbeat with NameNode to report block list status

Metadata persistence: Any changes to the file system metadata are recorded by the NameNode using a transaction log called an EditLog. Use the FsImage to store all metadata state in memory. Use the local disk to store the EditLog and FsImage. However, the recovery speed of FsImage is slow, and there is a risk of volume expansion. FsImage has a fast recovery speed, and the size of the data is similar to memory data, but it cannot be saved in real time, and the data will be lost. To ensure a closer FsImage and smaller EditLog size

Data persistence: log file: record real-time add/delete/modify operations mkdir/ABC Append Text file integrity better load restore data: < for example, I NN, memory 4G running 10 years log 5 years recovery, memory overflow mirror, snapshot, dump, DB, serialization interval (hours, days, 10 minutes, 1 minute, 5 seconds), The recovery speed is faster than that of log files. Some data may be lost due to the interval. HDFS: EditLog: Logs are small in size and few records are recorded. The EditsLog is now 10 points FI:9 points + EL 1 for 9 points to 10 points increments. Load FI 2. Load EL 3. Memory will get full data before shutdown ！！！！ Q: so: how does the FI time point scroll?? NN 8 point overwrite, 9 point overwrite... NN: When you first start up, you only write FI once. Suppose that the EL records logs between 8:00 and 9:00. You only need to update the logs between 8:00 and 9:00 to the FI at 8:00. TXT byte: 9 bytes int A =999999999 4 bytes < binary fileCopy the code

Safe mode: When the HDFS is built, it is formatted. The formatting operation generates an empty FsImage. When the NameNode starts, It reads the EditLog and the FsImage from the hard disk and applies all the transactions in the EditLog to the FsImage in memory and saves the new version of the FsImage from memory to the local disk and then deletes the old EditLog, Because the old EditLog transactions have been applied to the FsImage, the NameNode starts and enters a special state called safe mode. In safe mode, the NameNode does not replicate data blocks. The NameNode receives heartbeat signals and block status reports from all Datanodes Whenever NameNode checks to confirm that the number of copies of a data block reaches this minimum, After a certain percentage (configurable) of the data block is confirmed to be safe by NameNode (plus an additional 30 seconds of waiting time), the data block is safely replicated. The NameNode pushes the safe mode state. It then determines which data blocks have not reached the specified number of copies and copies them to other Datanodes

HDFS SNN: SecondaryNameNode (SNN) SecondaryNameNode (SNN) SecondaryNameNode (SNN) SecondaryNameNode (SNN) SecondaryNameNode (SNN) SecondaryNameNode (SNN) SecondaryNameNode (SNN) SecondaryNameNode (SNN) Period set the edits log size based on the configuration file. Fs.checkpoint. Size the maximum value of the edits file is 64MB by default

The first copy is placed at the DN of the uploaded file. If the node is submitted outside the cluster, select a node with less disk capacity and CPU busy at random. Second replica: placed on a node on a rack different from the first replica. Third replica: nodes on the same rack as the second replica. More copies: random nodes

HDFS write process: NN determines whether the metadata is valid. NN triggers a copy placing policy and returns an ordered list of DN. Client and DN establish a Pipeline connection. Fill the Client with chunk (512B) +chucksum(4B) and place the packet in the dataqueue. In this process, the upstream node simultaneously sends the next packet. In this process, the upstream node simultaneously sends the next packet. In life, it is analogous to the pipeline of a factory. HDFS uses this transmission mode, the number of copies is transparent to the client. When the block transfer is complete, the DN reports to the NN separately, while the client continues to transfer the next block. Therefore, the client transmission and the block report are also parallel

HDFS reading process: in order to reduce the overall bandwidth consumption and read latency, HDFS tries to let readers read a copy of its nearest If the reader has a copy in the same frame, then read this copy If an HDFS cluster spans multiple data centers, client also will be the first to read a copy of the data center semantic: Download a file: Client and NN interact with each other to obtain the file metadata fileBlockLocation NN will return to the client in order of the distance policy to attempt to download the block and verify the data integrity semantics: Downloading a file is actually obtaining all the block metadata of the file. In this case, obtaining some blocks of the subset should be established. HDFS supports the client to provide the offset of the file and customize the DN of the connected blocks