Liu is a second-year graduate student who is about to find a job. On the one hand, he wrote his blog to review and summarize the knowledge points of big data development, and on the other hand, he hopes to help his fellow students who learn programming by themselves. Because Old Liu is self-taught big data development, there will certainly be some deficiencies in the blog, but also hope that everyone can criticize and correct, let us progress together!

Today, I will introduce the HBase architecture design. Why is the HBase architecture design so great? The content of this article will not be very long, all is the essence of Lao Liu’s summary, we can not miss!

                                                   

1 background

We need to know two questions in advance. The answers to these two questions also answer why the HBase architecture design is so great!

First, as a distributed database, how does HBase ensure low-latency random reads and writes in massive data?

The second question is how does Hbase ensure data security in a distributed architecture?

2. Answer to the first question

To ensure low-latency random read and write in massive data, HBase has really put a lot of thought and design into it. Let Lao Liu give you a good talk.

2.1 Memory + Disks

This design exists in many frameworks and has the advantage of creating a trade-off between data security and data manipulation efficiency.

2.2 Memory Data Good data structure

HBase ensures efficient data operations and orderly data writing in the memory. What does this sentence mean? What makes data manipulation more efficient?

That is, in order to ensure the efficient operation of inserting and deleting data, the data is in order, so we use the data structure of ConcurentSkipListMap!

2.3 Range Partitioning + Sort

What method does mass data do query generally adopt?

The most violent way is violent scanning, but we generally do not use, the truth is understood! Therefore, we generally adopt the method of sorting first, and then range partition + segment scan. That massive data in order to make the data in order, in the process of using insert data, how to do the data in order?

According to the figure above, Lao Liu will explain the reason. A comes first, C comes next, and FINALLY B comes, but how can we put B between A and C?

First put the data in memory, a and c is adjacent, b come now, to ensure that ABC such order, actually very easy to do, we can use an array or a list, move the c went back, and then inserted the b come in, but memory is limited, you need every once in a while, the data in the memory are written to the disk file, the files in the disk is orderly, This can be convenient after fast positioning!

2.4 Jumper Topo structure

What does that mean? This is HBase addressing.

HBase addressing modes are earlier than or later than version 0.96.

Before hBase-0.96, HBase has two special tables: -root – table and -root table. META. Table where -root – is stored in ZooKeeper and -root – itself is stored in ZooKeeper. META. Table RegionInfo information. The process of addressing is as follows:

  1. Client requests ZooKeeper to obtain the RegionServer address of -ROOT-.

  2. Client request -ROOT- RS address, obtained. META. Address of the table. The Client caches -root – information for the next quick access.

  3. The Client request. META. RegionServer address of the table. The Client obtains the address of the RegionServer where data is accessed. The META. Related information is cached down for quick next access.

  4. Client Requests the address of the RegionServer where data resides to obtain the corresponding data.

We can see that the user needs 3 requests to know the real location of the user table, which leads to some performance degradation. The main reason for using a 3-tier design prior to 0.96 was to consider how large the metadata might need to be.

The addressing flow after 0.96 is as follows:

  1. Client requests ZooKeeper to obtain the file. META. Address of the RegionServer where META resides.

  2. The Client request. META. RegionServer Obtains the address of the RegionServer where data is accessed. The META. Related information is cached down for quick next access.

  3. Client requests the RegionServer where the data resides to obtain the required data.

The reason for removing -root – is that after hbase-1. x, the maximum size of each region is 10 GB by default, whereas before, it was 128 MB. Now, the two-layer structure can meet the requirements of clusters and improve the performance.

Region address: User table, meta table, root table, Region address: user table, meta table, root table

If you have a table that is too large to be stored on a single server, you need to split the table into several parts (say, four parts). If the table had been sorted before it was split, the split data would be divided into segments, each of which would contain all the data in that segment.

The client now queries the data and scans the data. Which server does it scan? We are not sure, it is possible that every server has to be scanned, which makes it very inefficient!

We can first find an intermediary to confirm which segment of the data we want to find, create a meta table meta Region location of the real data, first find meta table. When the metadata table becomes very large and is partitioned into RegionServer, a ROOT table is provided to locate the Meta table.

This creates a hierarchical relationship from the ROOT table to the Meta table to the specific RegionServer.

The formation of the hop table Topo structure reduces the number of scans, which used to require N or 4 scans, but now changes to 1 scan. The performance is improved, and a lot of data can be managed.

How to understand that you can manage a lot of data?

Before hbase1.0, the default size of each Region is 128 MB, and each metadata is 1KB. Then, the Region can store more than 1000 metadata, and the three-layer structure can store tens of millions of records. After hbase1.0, the default size of each Region is 10 gb, and the two-tier structure can store enough data to meet cluster requirements.

2.5 Read Cache + Write Cache

Memory is divided into read cache and write cache. Frequently queried data is stored in the read cache to improve efficiency. How can write cache scan files at the highest rate? Is to use the way of memory + disk, put the data in memory sort first, and then write the data to the disk, so that the files in the disk is in order, and then the disk file binary search, high efficiency.

3 answer to the second question

What does HBase do to ensure data security in distributed architecture?

3.1 Memory + Disks

HBase uses memory and disk storage to balance data security and data operation efficiency.

3.2 a WAL mechanism

WAL stands for Write Ahead Log, similar to MySQL’s binlog, used for disaster recovery. To prevent data loss due to RegionServer process exceptions after data is written to the cache, HBase writes data to the HLog(WAL) in sequence before writing data to the cache. If RegionServer breaks down or other faults occur, the HLog can be played back to recover data to prevent data loss. The most important aspect of HBase fault recovery is how to recover lost data through HLog playback.

4 summarizes

Well, the architecture design of HBase is almost talked about, Lao Liu mainly told us why the architecture design of HBase is so great. Although his current level may not be as good as yours, Liu still hopes to become better and help more self-taught programming partners.

If you have any questions, please contact our official account: hardworking Old Liu. If feel helped you, might as well like attention to support a wave!