01 Let’s learn big data together

Hello everyone, today we continue to explain the big data HDFS knowledge point. Lao Liu tried to explain to everyone in plain English, so that everyone can try to solve the confusion of HDFS after reading. If you think Lao Liu’s writing is good, give Lao Liu a thumb-up! At the same time, hope to get everyone’s criticism and advice.

02 HDFS Knowledge

Point 10: Heartbeat mechanism of HDFS mechanism

According to this picture, Mr. Liu will talk about the heartbeat mechanism workflow. It can be clearly seen from the picture that the Master is the core. First, the Master will start first and open an IPC server there. Then, after the slave node is started, the slave node will go to its master, so it finds the master and registers the connection with the master. Moreover, the slave node will send a heartbeat to the master node every 3 seconds, carrying status information and reporting its own situation. If the slave node wants to control the slave node, it will keep track of the slave node’s situation. Finally, the master node sends instructions to the slave node through the return value of the heartbeat.

So what’s the use of this heartbeat mechanism?

NameNode periodically receives heartbeat signals and block status reports from each DataNode in the cluster. Receiving heartbeat signals indicates that the DataNode is working properly. The block status report contains a list of all data blocks on the DataNode.

2. After DataNode is started, it registers with NameNode and periodically reports all block lists to NameNode. Sends a heartbeat to NameNode every 3 seconds and returns the command that NameNode sends to the DataNode. If NameNode does not receive a DataNode heartbeat for more than 10 minutes, the node is considered unavailable.

When the Hadoop cluster starts up, it goes into safe mode (99.9%) and uses heartbeat mode.

Here involves the safe mode, Liu has seen several institutions of HDFS courseware, most of them with a safe mode. But if we are communicating with others, you say the safe mode, people ask you what the safe mode is, will you be embarrassed. So a lot of knowledge points since mentioned, at least to say a concept.

What is safe mode?

Safe mode is a protection mechanism of Hadoop to ensure the security of data blocks in a cluster. In this state, the file system only accepts data read requests, but does not accept change operations such as deletion or modification.

When a Hadoop cluster starts up, it goes into safe mode and checks the integrity of data blocks. Assuming we set the number of copies to 5, there should be 5 copies on the DataNode. However, if there are only three datanodes at present and 3/5=0.6 is less than 99.9%, the system will automatically copy copies to other Datanodes. If there are eight in the system and we only set five, the extra three will be removed. Under normal circumstances, safe mode will run for a period of time and then automatically exit.

Point 11: HDFS data read process

What is the basic flow?

Let’s look at the diagram and say the flow, in the HDFS reading process, first think about, we want to read data, whether to know where the data is? In which small module is it stored? How can I get it?

So our client will call the open method of the file system, and the file system will call the open method of the NameNode remotely through RPC to get the block location information back.

When the block location information is returned, NameNode calls the read method of FSDataInputStream, which contacts its nearest DataNode.

We should also pay attention to this, this recently refers to the network topology to do the sorting, the client is close to the first, but Liu is not sure, you can go to Baidu to see.

Next, if the first DataNode fails to connect, the client will automatically contact the next DataNode. If the check value of the block data fails, the client needs to report to the NameNode and automatically contact the next DataNode.

Data read tolerance

What if the communication between the Client and the DataNode is interrupted during block reading?

The Client connects to another DataNode that stores the block, reads the data, records the DataNode in question, and does not read data from it.

What if a Client reads a block and finds a problem with the block data?

Check (CRC32-); check (CRC32-); check (CRC32-); If it is not equal, which node block is faulty, another node reads, tells NameNode which node block is faulty, and copies a copy of data from other nodes to that node.

What if the Client reads incomplete data?

If not, a different node reads the faulty block, tells NameNode which block is faulty, and copies a copy of the data from other nodes to that node.

Point 12: HDFS data writing process

What is the basic flow?

In the same way, I want to write data into the cluster, where the data will be stored, how to store, through what path, all have to figure out!

1. When the client invokes the create method of the distributed system, the file system also invokes the Create method of the NameNode through RPC. In this case, the NameNode performs the following operations: (1) Check whether it is running properly, (2) check whether the file to be created exists, (3) Check whether the Client has the permission to create files, and (4) Record changes to the HDFS status in the Edits log

2. The NameNode responds that the client is ready to upload.

3. The client starts to upload the first block based on the block size set by itself. The default value is 0-128 MB.

4. The client requests to establish a transmission channel based on the returned DataNode. The client initiates a channel establishment request to the nearest DataNode, and then the DataNode successively sends a channel establishment request to the next node in the channel (closest to the current DN). Each node sends a response, and the channel is successfully established.

5. Every 64K data read by the client is encapsulated into a packet(it is a packet, which is the basic unit of transmission), and the packet is sent to the next node in the channel. After receiving the packet, the node in the channel drops the disk for storage and sends the packet to the next node in the channel. Send an ACK acknowledgement message to the client.

6. After the data transfer of a block is complete, the channel is closed. DataNode reports a message to NameNode indicating that a block has been received

7. The first block is transferred, the second block is transferred, and the above steps are repeated until the last block is transferred, the NameNode responds to the client, and the client closes the output stream.

But what if there is an exception to the write process?

Steps 1 through 4 are the same as before, straight up.

5. Each 64K data read by the client is encapsulated into a packet, and the packet successfully encapsulated is put into a queue, which is called dataQuene(packet to be sent).

6. When sending, send the packets in dataQuene in sequence first, and then put them into Ackquene (the queue being sent). After receiving packet, each node sends an ACK message to the client.

If a packet has received ack messages returned by all DN after being sent, the packet will be deleted in Ackquene.

If a packet times out after being sent and the ACK message returned by DN is received, the transmission is aborted and the packet in Ackquene is rolled back to dataQuene. Re-establish channels and eliminate bad DN nodes. Once the setup is complete, transfer continues. As long as a DN node receives data and THE DN reports that the NN has received the block, the NN considers that the current block has been successfully transmitted.

Point 13: Hadoop HA

Hadoop HA this point, Lao Liu is going to use in the ZooKeeper module, we can look forward to it.

Point 14: Hadoop federation

For a large cluster with a large number of files, memory will become a bottleneck that limits the horizontal development of the system, and federation was created to overcome this bottleneck. This talk is relatively simple, Lao Liu also does not understand this very much.

Point 15: Why isn’t HDFS good for small files?

This is a question from Lao Liu’s own study materials. Write it out and let us think about it.

Here are some of the answers I found online:

First, NameNode stores metadata of the file system. Each file, directory, and block contains about 150 bytes of metadata. Therefore, the limit of the number of files is also determined by the memory size of NameNode.

Give another view of my own summary:

HDFS is designed to store large files. The size of a block of data is about 150 bytes, and storing a small file takes up 150 bytes of memory. If you store a large number of small files, you can quickly run out of memory, but the amount of data stored in the entire cluster is so small that HDFS is meaningless. Just like you play a similar skill points game, all added in the very useless skills above, and others play when there is no powerful big move is the same!

How to solve the problem of storing a large number of small files?

In Sequence File scheme, the core is to organize small files with the File name key and the File content value. A large number of small files can be programmed into Sequence File, and then processed by data flow, or processed by MapReduce. The advantages of this method are as follows: (1) Sequence files can be partitioned. MapReduce can cut files into blocks and operate each block independently. (2) Sequence files support compression. In most cases, it is best to compress them in blocks. Because a block contains multiple records, the compression efficiency is higher by using the similarity between records.

03 summary

All right! Old Liu will talk about big data HDFS module in plain English, there will be some not very good place, old Liu will slowly correct. If you don’t understand the place, you can leave a message directly, or contact the public number: hardworking Old Liu.

Finally, something, contact the public number: hard old Liu; Nothing. Just learning big data with Lao Liu.

If you think Lao Liu’s writing is ok, please give Lao Liu a thumbs up!