HBase supports 100 TB data and millisecond response under millions of QPS, and is suitable for online risk control and recommendation scenarios in the context of big data. Alibaba Cloud HBase provides risk control and recommendation services for a number of financial, advertising, and media services, continuously optimizing the services in the direction of high availability, low latency, and low cost. The current single cluster promises 99.9% availability, SSD models can achieve request burrs of 99.5% < 50ms, and the newly launched in-table hot and cold separation accelerates hot data access performance, reduces cold data storage costs, and is transparent to users on the API. To further meet customers’ demands for availability and low latency, alibaba cloud HBase provides a high availability solution supported by redundancy, concurrency, isolation, and degradation based on the active/standby architecture. The goal is to provide 99.99% availability and 99.9% requests smaller than 50ms. The first phase of the HA project has been completed. This article describes the active/standby architecture and one-click switchover capability.

HBase cluster availability

An HBase system consists of Zookeeper, HDFS, and HBase. Zookeeper is a self-healing system that implements redundancy in data and computing. A single node failure can be recovered in seconds. HBase relies on Zookeeper lightly, and no major problems occur. HDFS is a distributed file system in master-slave mode. The Namenode node is deployed in HA mode, and the failure of a single Namenode can be recovered in seconds. When both NNS are down, the recovery time depends on the playback log size. At the minute level. Multiple data copies are redundant. Services are not interrupted when a Single Datanode goes down. HBase is also a distributed system in master-slave mode. The HMaster does not participate in reading and writing paths. Therefore, services are not interrupted when the HMaster is down. HBase also has a single point called Meta service. It randomly selects a RegionServer node as the host. If the Meta service is unavailable, all request routes become invalid. Since the client has a route cache, the meta service outage does not immediately cause unavailability. HBase data fragments use Range Partition. Each Region is serviced by a RegionServer. That is, the HBase computing service is 0 redundant. The HBase Failover mechanism can heal itself. The recovery time of a Single RegionServer is in minutes and that of multiple RegionServers ranges from 10 minutes to 1 hour.



HBase ha deployment in a single cluster

Component Role High Availability mechanism Impact Zookeeper data and computing redundancy Single point of breakdown second level service HDFSNamenode Two-node HA Deployment Single point of breakdown second level recovery Two-node breakdown recovery time At the minute level Datanode data and computing redundancy Single-point breakdown uninterrupted service HBaseHMaster Two-node HA deployment does not affect data read and write Meta service Single-point, self-healing global unavailable Region Single-point, self-healing local unavailable, single-point minute recovery, multi-point recovery time at the hour level

The HBase self-healing cannot be controlled due to a Bug in the kernel or running environment

Case1 WAL data is damaged due to the RESTART of the ECS in the HBase built by a financial company. HBase Failover cannot be performed and the cluster cannot be used for self-healing. Case2 Meta data in the HBase cluster built by a video company is inconsistent and cannot be restored. Part of the table is not available to self-healing case3 IO pressure often trigger a media company space leak Bug, leading to insufficient space cluster downtime cannot heal Above is the real case, the Bug is inevitable, unknown Bug processing test kernel team’s technical strength, need to have the ability to surgery, but the processing time is unpredictable.

Cloud HBase active/standby HA scheme

A high availability system with predictable fault recovery time. When a fault occurs, read and write capabilities can be restored at the minute level. The implementation scheme chooses the classic master and slave mode, and also supports the master and slave mode. The secondary cluster is deployed in different equipment rooms in the same city and is completely isolated from each other on the resource level. Different software versions can be deployed to maximize the independence of the two clusters. Based on the empirical data, the maximum recovery time of a single room is one hour, and the probability of another room failing within this hour is very low. There are two exceptions: the first is a city-level disaster, which requires a remote disaster recovery plan. In the second case, the fault occurs because the service traffic increases and the system is overloaded. In this case, the solution is to limit traffic or expand capacity. These two cases are not considered in this design. The system provides the one-click switchover capability, which does not require the client to restart. Each client automatically completes the connection switchover upon receiving the switchover signal. The client switchover is performed concurrently, and all clients can complete the switchover within 10 seconds.

The HA solution can be independently provided for self-built users. Self-built and cloud HBase can work in active/standby mode. If you already have an HBase cluster, you can create a redundant standby cluster. Table creation and data replication are automatically completed in the standby cluster. When a fault occurs, you can switch traffic to the standby database on the console and switch traffic back after the active cluster recovers. The cloud HBase HA service is an independent service, which can be regarded as a middleware based on HBase. The hybrid cloud is our design principle. The self-built HBase, EMR HBase, and cloud HBase in the cloud and VPC can be used in both active and standby mode. For HBase built by IDC or HBase in its cloud, users need to resolve network connectivity.

Easy to use. No code modification is required to convert the HBase client from the common mode to the high availability mode. Cloud HBase provides a plug-in integrated into the HBase client and provides a new Endpoint for HA. The Endpoint is simply IP +port, which replaces the Zookeeper address in single-cluster mode. Users can select core tables for high availability construction, while non-core tables remain a single cluster to reduce costs. You can create, modify, and delete highly available tables using the high availability client. The console ensures metadata consistency between the active and standby clusters. Currently, we support multiple versions such as 0.96, 0.98, 1.x, 2.x, and cloud HBase Enhanced edition.

0.961 x2. X cloud enhanced version 0.96 YESNONOYES1. XNOYESYESYES2. Enhanced version YESYESYESYES xNOYESYESYES cloud


Version Active/standby compatibility list

The console provides one-click switchover without restarting the client. AliHBaseConnector serves as a plug-in of the native HBase client. The plug-in encapsulates the active/standby switchover logic. When the client starts, it obtains the primary and secondary address information from BDS and establishes a highly available command channel. The switch command sent by the console reaches the client through this channel. The client automatically disconnects the old link, opens the standby link and retries the request.

To sum up, the cloud HBase active/standby HA architecture is as follows:



  • Data synchronization

The cloud HBase does not use the open source Replication solution. The main reasons are as follows: Each machine can only process its own logs. HBase Replication Failover is faulty, resulting in a long synchronization delay once the RS breaks down. HBase Replication and the kernel compete for resources. HBase Replication cannot be independently expanded. There are many configuration items and it is difficult to maintain HBase Replication in hybrid cloud mode

BDS Replication is an independent data Replication service that provides bidirectional data synchronization with a 100-millisecond delay. BDS Replication can be scaled up and down independently, requiring no disks and managing costs better.



  • Active/Standby Consistency

HBase in the cloud adopts asynchronous replication to provide version-based final consistency. A row written to the master keeps the same version number as the master when it is synchronized to the standby. Currently, only the final consistency switchover is supported for HBase high availability in the cloud. During the switchover, there may be a second inconsistency between the active and standby nodes. For risk control, recommendation, and portrait applications, the final consistency meets the requirements. If you use interfaces such as CheckAndPut and CheckAndDelete, consistency problems may occur. In this case, contact cloud HBase for technical consultation.

  • Configure the service

The configuration service stores the address information of the active and standby clusters and commands for notification switchover. The HA client accesses the service to load the IP addresses of the active and standby clusters. Switch commands executed through the console are also notified by the configuration service to all clients. The configuration service itself employs redundancy to ensure high availability of storage and messages.

  • monitoring

The HA service displays the status of active and standby clusters in real time. The active and standby roles cannot be changed during initial configuration. However, the active/ Standby status can be changed. The cluster in the Active state is the cluster accessed by the client. Displays the real-time delay of bidirectional synchronization links between active and standby clusters and monitors the number of client connections, enabling users to accurately judge the switchover progress.

Cloud HBase high availability mode

If you have used cloud HBase and want to add a standby database, perform the following steps: 1 Purchase a BDS cluster for active/standby management. 2 Purchase an empty cloud HBase cluster as the standby database. 3 Enter the instance ids of the active and standby clusters

If you use a self-built ECS cluster or EMR HBase and want to add a standby database, perform the following steps

1 Purchase a BDS cluster for active/standby management. 2 Purchase an empty HBase cluster as a standby database. 3 Configure domain name mapping and security groups (related services will be provided by one click)

The original link

This article is the original content of the cloud habitat community, shall not be reproduced without permission.