Business introduction
Duiba Group consists of Duiba Network and Tuiba Network. Duiba network is a user operation service platform dedicated to helping Internet enterprises improve operation efficiency, providing points shopping mall and media operation services. Tuah network is an interactive advertising platform. After years of exploration and practice, it has pioneered a new mobile advertising model and achieved a win-win situation for advertisers, media and users. In the advertising scenario of Tuiah, advertisers can get better advertising effects, media can get better traffic realization efficiency, and audiences can have better user experience. At present, Tuiah has served more than 15,000 media, and Ali Cloud hbase mainly serves the advertising business of Tuiah.
The overall business process of tui ah is as follows:
Overall Product Architecture
The advertising platform has a perfect infrastructure, which can effectively support business. The core data platform provides strong data support for all business of the company. The whole data platform can be roughly divided into three modules according to different processing services:
- Off-line statistics module: off-line statistics of data, reports and corresponding background data analysis
- Real-time statistics module: Real-time data is mainly used for docking algorithms and statistics of users’ real-time behaviors, such as exposure to different advertisements, clicking and other behaviors, which require fast computational response, so we adopt low-delay streaming computing
- Real-time OLAP analysis module: multidimensional real-time analysis, positioning is to provide minute granularity statistics, mainly used for arbitrary dimension and indicator statistics
HBase is in the “Push” scenario
HBase is mainly used for streaming data statistics and storing user portrait data, which is mainly stored in the real-time statistics module. In real-time statistics, users’ behavioral data will be counted according to different dimensions and indicators. For example, users’ exposure, clicks and participation in different advertisements will be recorded. Corresponding attributes of users will also be recorded, such as what kind of advertisements users are interested in, their age, gender, occupation and hobbies. All data is stored in HBase clusters.
Why to migrate from physical HBase to Ali Cloud HBase
In the beginning, we built HBase in the physical machine room. We chose ali Cloud HBase for the following reasons:
- Cloud HBase service basically requires no o&M. To reduce the pressure of o&M system tuning, alibaba Cloud hbase team provides professional O&M services.
- HBase infrastructure is of high importance. HBase as the underlying storage system, once appear, system failure, the cycle is long, high difficulty, in a short period of time is difficult to solve, directly affects the stability of the online system, in this respect, ali cloud HBase can provide strong technical support, ali cloud has the most powerful domestic core team, according to understand ali currently has three PMC, six committer, It has the most HBase committers in China.
- The cloud HBase service is good. If you have any questions about using Hbase, you can directly consult the Hbase students of Ali Cloud. They provide prompt response, thoughtful service and professional suggestions.
The whole process of migration
Based on the development of our services, the usage and problems of hbase in Alibaba Cloud are described in three stages
Stage 1: Assume the role of data mart and decompose business access pressure
At this stage, our data center was built in our OWN IDC room, using CDH Hadoop to build the cluster. All the components including Hive, JStorm and Druid were installed in the same cluster. JStorm used HBase provided by Hadoop for calculation and statistical data. After the calculation is complete, the finished data is written to HBase of Ali Cloud, and the service system accesses HBase of Ali Cloud to obtain the calculated data. The reasons for this are as follows:
- The business system uses the ECS server of Aliyun, which is connected to the IDC room through dedicated lines. The transmission across public networks occupies bandwidth, and the network quality cannot be guaranteed.
- Service systems are not advised to directly access the HBase cluster in the IDC equipment room because high concurrency may increase the load of the entire cluster and affect other services in the cluster.
In this phase, the HBase configuration is 8 GB with four cores and 100 GB with two nodes42 SSD: Synchronizes 20% of the service data to the online system. The data volume is about 200 GB. The query QPS is about 500, and the average query time is 2ms
Phase 2: Full migration: Replacing HBase on offline physical servers with cloud HBase
At this stage, we migrated IDC’s Hadoop cluster to Ali Cloud, and bought the HBase cluster of Ali Cloud to replace the HBase cluster in the original CDH. IDC room migration to Ali Cloud is mainly based on the following points to consider:
- In an IDC equipment room, all components are deployed on the same server. As a result, resources compete with each other and the running of each component affects each other. Therefore, resources used by components are isolated.
- We checked and found that the cost of IDC’s self-built computer room in 5 years was much more expensive than the server using Ali Cloud.
- After migrating to Aliyun, all our systems and services are in the same internal network environment, and the network quality is more guaranteed than the original public line.
In this phase, the hbase configuration is 32 GB with eight cores and 200 GB with four nodes
Stage 3: Optimization and transformation to ensure the ultimate read delay
Because HBase is based on the native mechanism of Java VIRTUAL machines, the jitter of HBase data read by the service system will reach 100-200ms due to GC. For the advertising recommendation system, an advertisement recommendation must be completed within 200ms. Such jitter is obviously unacceptable. We made the following modifications to the system:
- Delay control is added to services. When HBase data is read for more than 100ms, the service is disconnected, degraded, and advertisements are recommended randomly.
- Services are split and a new HBase cluster is purchased. Only services with high latency requirements can be used. Some services with high latency requirements are migrated. After migration, delay jitter is reduced from 2 ‰ to 6 ‰, and the delay is improved.
In addition, according to the introduction of ali HBase students, Ali Cloud will launch HBase 2.0, which has been optimized at the architecture level and will fundamentally solve the delay jitter caused by Java GC mechanism. I am looking forward to it very much.
conclusion
Overall, Ali Cloud HBase is excellent. We also thank Ali Cloud technology students for helping us solve the operation and peacekeeping performance optimization of the underlying system, ensuring the stability of the underlying system, enabling us to focus more on the service business and helping the business to develop faster.
The original link