Abstract: In the 9th China Database Technology Conference, Alibaba technical expert Meng Qingyi explained the practice and evolution of Alibaba HBase data pipeline facilities. This chapter describes data import scenarios, HBase Bulkload, HImporter system, data export scenarios, and the HExporter system. Please click PPT to download the live video.

Data Import Scenario

Business staff

A business advisor is a data product that serves businesses, helps them make decisions and operate. For example, if you open a store on Taobao or Tmall, the business staff will provide daily traffic, conversion rate, customer portraits and the industry to compare where these data belong. Businesses can make decisions based on traffic analysis, activity analysis and industry analysis. The database writes data to HBase in real-time based on logs, clicks, and visits. Some data is written to the offline system and is periodically cleaned and calculated before being written to HBase for services to query HBase.

Ant risk control

Any transaction payment on ant will invoke risk control, which is mainly to see whether the transaction belongs to the same device, whether it is a place of frequent transaction, and the store information of the transaction. It must complete the risk within 100ms to 200ms. Risk control is based on comprehensive consideration of long-term historical information, recent historical information and real-time information. User input can real-time write HBase, at the same time, the real-time information increment will be imported into the offline system, offline system regularly for calculation, data calculation results of the data as a history or recent history to write back to HBase, a payment may be one hundred times the risk of control, and need to return in hundred milliseconds.

Problems to be resolved in data import

In 2013, when WE just started to do data import, we faced more problems of functional requirements. Now, we need to consider the problems of periodic scheduling of import, multiple heterogeneous data sources, high import efficiency and data consistency under multiple clusters. The first two problems are better solved by platforms. HBase data import focuses on import efficiency and data consistency in multiple clusters.

What is Bulkload? What does Bulkload do?



Bulkload uses a new structure, LSM Tree, to write and update data, as shown in the figure above. When Application Code is used to write data, data is written to MemStore. MemStore is a hoptable in HBase. It can be regarded as an ordered list and data is continuously inserted into it. When a certain amount of data is reached, flush is started to encode and compress the data, and HFile is written. HFile is a file structure composed of index blocks and data blocks. It is read-only and cannot be modified after HFile is generated. When users read data, they will search for it from three hfiles and a MemStore. The optimization of this structure is to turn random writes into ordered writes. Bulkload can add thousands of data pieces to HBase in milliseconds. Therefore, the advantages of Bulkload are as follows:

  • High throughput
  • Don’t need…
  • Avoid small compaction
  • Support for offline builds



The import structure of Bulkload is shown in the preceding figure. Data comes from data stores. Data is partitioned and sorted according to HBase partition rules. Then Partition Data is generated. You need to write an HBase and insert it into the Partition Data. The synchronization center will transfer a job, and there will be many tasks inside the job. Each task independently reads the file and writes it to HDFS to form an HFile file. When all files are written, the synchronization center configures the Bulkload command to HBase to load all Hfiles in one time. The method of multi-cluster import was used before, but multi-cluster import has many disadvantages as follows:

  • It is difficult to ensure the simultaneous completion of multiple tasks, resulting in data inconsistency in a certain time window
  • The running environment after scheduling is inconsistent. Procedure
  • The network latency is inconsistent
  • Disadvantages of cluster deployment opacity to services:
  • Multiple tasks need to be configured
  • Reconfigure tasks in cluster migration To ensure data consistency, logical cluster import is adopted.



The logical cluster process is shown in the preceding figure. Partition merging is performed first, and dual data streams are processed. The streams are written to Active and Backup respectively. Because Bulkload is millisecond level, consistency can be achieved. The differences between multitasking and logical clustering are as follows:

  • Multi-task mode: requires repeated configuration, is opaque, difficult to ensure consistency, partition sorting needs to be performed twice, encoding compression twice.
  • Logical cluster mode: configure once, migration is not aware, consistency reaches millisecond level, partition sorting is performed once, but the number of partitions increases, encoding compression once. As the business grows larger, this import will encounter new online issues such as scalability, resource utilization, r&d efficiency, monitoring, and so on.

What is the HImporter system?

HImporter is an intermediate layer used to assist data synchronization, and it will sink all HFile build, load logic down to HImporter.



HImporter’s location is shown above.

The advantage of HImporter

  • Distributed horizontal scaling, different tasks of the same job can be scheduled to different worker nodes of HImporter
  • Improved resource utilization and reduced CPU intensive operations such as compression to HImporter
  • Fast iteration, HImporter’s operations and operations are independent of the synchronization center
  • Independent monitoring, HImporter can monitor according to his own needs

HImporter functional iteration

Functional iterations include table attribute awareness, ensuring localization rates, supporting lightweight computing, and secure isolation. Among them, table attribute perception is the perception feature, and ensure that the feature will not change, mainly including, mixed storage, new compression coding, table level copy number; Ensure the localization rate is to write Hfile to the server where the partition is located, ensure the localization rate and storage characteristics, for some RT sensitive business effect is obvious; Support light calculation is to support MD5, string concatenation and other functions; Security isolation prevents HDFS addresses from being exposed and supports task-level retry.

Data Export Scenario

Novice alliance



The scene of the Rookie League is shown above. A logistics detail is transmitted to HBase, and HBase is transmitted to the data warehouse, which generates reports and trains routing algorithms.

Taobao customer service



A refund application scenario of Taobao customer service is shown in the figure above. This is a reverse link, which writes refund, dispute, refund logistics, and refund fees to HBase in real time. HBase will write them to queue in real time. The Blink flow system will consume queue and generate some data, which will be returned to HBase.

Issues to be resolved with incremental data export

The problems to be solved in incremental data export mainly include T+1 processing characteristics of offline data, throughput, real-time performance, and active/standby traffic switchover. The early solution was to periodically list all the logs from HDFS, and then sort the logs to produce an ordered time stream. Take the shortest synchronization time in the work as the final synchronization time. This scheme has many problems such as great pressure on NN nodes, inability to deal with master/standby switchover and low log hotspot processing ability.

HExporter system



HExporter1.0 as shown in the figure above, HExporter1.0 has the following advantages: primary and secondary traffic switching does not affect data export, it can identify data sources and filter non-original data; The independent synchronous time stream ensures that data is dumped to the data warehouse in an orderly time partition. The HBase Replication framework can reduce development workload and reuse HBase monitoring, operation and maintenance (O&M) systems.

HExporter1.0 optimization mainly includes the following five points:

  • Reduce data transmission on the topology network, and avoid sending duplicate data to the standby repository.
  • Remote auxiliary digester, idle machine to help digest hot spots;
  • Avoid sending small packets. HExporter can handle packets after receiving them for a period of time.
  • Synchronous channel configuration isolation, real-time consumption link and offline consumption link can use different configurations;
  • Data compression before sending.

    The problem of HExporter1.0 is that business write traffic peaks and offline synchronization delays occur. The HBase upgrade speed is slow. Then we have the following solutions: offline synchronized resources can be isolated from online resources, and off-line large pools can be used to cut peak load and fill valley; Workers with a close friend are stateless, and if all logic is at my friend, upgrades and expansions are easy and quick. Then came HExporter2.0.

HExporter2.0 is shown in the figure above

conclusion

An intermediate layer is added to import and export ALiHBase data channels. The core values of the intermediate layer are easy to expand, high reliability, fast iteration, and stable. Because using distributed horizontal scaling is easier to scale; Automatic identification active/standby switchover is adopted to ensure reliable HBase access. Using architecture decoupling, fast iteration makes the iteration speed faster; Because stateless, node peers are more stable.

The original link