The author | | yuan-zheng zhang source alibaba cloud native public number

Dubbo is a distributed microservice framework, and many companies have implemented distributed system architecture based on Dubbo in practice. With the open source reboot, we have not only seen the latest Roadmap for Dubbo 3.0 released, but also alibaba’s integration of Dubbo and internal HSF in its own e-commerce and Dubbo 3.0 on Singles’ Day. This article is the sharing of ICBC’s construction of financial micro-service architecture based on Dubbo, mainly narrated the coping strategies and achievements of service discovery. The practice of ICBC’s large-scale service monitoring and governance will be published later, as well as how to re-develop Dubbo from the perspective of enterprises. Welcome to follow.

Background and overview

Icbc’s traditional business system is based on JEE single architecture, which cannot meet the needs of the business in the face of the trend of online and diversified financial services. Therefore, since 2014, ICBC has chosen a business system to try servitization, verified, evaluated and compared several distributed service frameworks at that time, and finally chose Dubbo, which is relatively perfect and has been used by many companies in China. At the same time, ICBC also made enterprise customization for Dubbo, which helped this business system to complete the implementation of servitization, and received very good results after it was launched.

In 2015, ICBC began to expand the scope of service architecture. On the one hand, it helped the traditional business system to carry out architecture transformation, and on the other hand, it gradually precipitated the super-large service group similar to Zhongtai to support the combination and reuse of fast services of the business system. With the accumulation of experience, ICBC has also carried out iterative optimization and enterprise customization for Dubbo, and gradually built a perfect service ecosystem centering on services.

In 2019, THE microservice system of ICBC was officially upgraded as one of the key capabilities of the core banking system of ICBC Open Platform, helping ICBC’S IT architecture realize the real distributed transformation.

The composition of ICBC’s micro-service system is shown in the figure below:

In terms of infrastructure, both the service nodes of the business system and the work nodes of the micro-service platform have been deployed in the CLOUD platform of ICBC.
In terms of service registration discovery, in addition to the conventional service registry, metadata center is also deployed to realize service registration discovery by node.
In terms of service configuration, an external distributed configuration center is used to centrally manage and deliver dynamic parameters.
In terms of service monitoring, it realizes the unified collection and storage of various service operating indicators, and connects with the monitoring platform of the enterprise.
Service tracing is mainly used to trace the entire service links in real time, helping service systems quickly locate faults and accurately assess the impact range of faults.
Service gateway is to meet the demand of traditional business system to access services. On top of Dubbo service subscription and RPC capability, it realizes automatic discovery, automatic subscription and protocol conversion capability (HTTP protocol to RPC protocol) of new service and new version, realizing 7×24 hours uninterrupted operation.
The service management platform provides a one-stop management, monitoring and query platform for operation and maintenance personnel and development and testing personnel to improve the efficiency of daily service management.

The biggest challenge

After years of practice in ICBC, this paper summarizes the biggest challenges in the following two aspects:

In terms of performance capacity, the number of online services (the number of service interfaces in Dubbo’s concept) is now over 20,000, and the number of provider entries per registry (the total number of providers for each service) is over 700,000. It is estimated that the number of services that can support 100,000 levels and the number of provider entries at 5 million levels per registry will be needed in the future.
In terms of high availability, ICBC’s goal is that no node failure on the micro-service platform will affect online transactions. The business system of the bank operates 7×24 hours. Even within the version production time window, the production time of each business system is staggered. The platform nodes need to be upgraded, so how to avoid the impact on online transactions, especially the version update of the registry itself?

This paper will first share icbc’s coping strategies and results from the aspect of service discovery.

Service discovery and optimization

Introduction to 1.

In Dubbo, the registration, subscription and invocation of services is a standard paradigm, where the provider of the service registers the service when it initializes, and the service consumer subscribes to the service when it initializes and gets the full list of providers. At run time, when the service provider changes, the service consumer gets the latest list of providers. Point-to-point RPC calls between consumers and providers are made without a registry.

In the selection of registry, ICBC chose Zookeeper in 2014. Zookeeper is widely used in various scenarios and supports cluster deployment. Data consistency between nodes is ensured in CP mode.

Within Zookeeper, Dubbo sets up different nodes according to services. Each node has providers, consumers, configurations, and routers.

Providers temporary node: Records the list of service providers. When the provider goes offline, the child node is automatically deleted. Through the Watch mechanism of Zookeeper, consumers can know the change of the provider list in the first time.
Consumers temporary node: A list of recorded consumers, mainly used to query consumers during service governance.
Configurations Persistent node: Mainly stores service parameters that need to be adjusted for service governance.
Routers: Child nodes are persistent nodes and are used to configure dynamic routing policies for services.

In an online production environment, multiple clusters are deployed in the Zookeeper data center. Each cluster is configured with five election nodes and several Observer nodes. The Observer node is a new node type introduced in Zookeeper3.3.3. It does not participate in elections but only listens to voting results. Other capabilities are the same as those of the followers node. Observer nodes have the following benefits:

Distributary network pressure: With the increase of service nodes, if the clients are all connected to the election nodes, the election nodes need to consume a large amount of CPU to process network connections and requests. However, election nodes cannot be expanded at any level. The more election nodes there are, the longer the transaction voting process will be, which is detrimental to high concurrent write performance.
Reduce cross-city and cross-DC subscription traffic: When 100 consumers need to subscribe to the same service across town, the Observer can process the cross-city network traffic in a unified manner, avoiding strain on inter-city network bandwidth.
Client isolation: Several Observer nodes can be assigned to a specific application to ensure network traffic isolation.

2. Problem analysis

Icbc summarized the problems faced by Zookeeper as a service registration center according to the sad and tearful history of using online Zookeeper in recent years:

As the number of services and service provider nodes increases, the amount of data pushed by services will explode. For example, a service has 100 providers. When the provider is started, due to the CP feature of Zookeeper, every time a provider comes online, the consumer receives an event notification, reads the list of all current providers of the service from Zookeeper, and then flushes the local cache. In this scenario, each consumer theoretically received 100 event notifications and read the list of service providers from Zookeeper 100 times, 1+2+3+… +100, 5,050 provider data in total. This problem is particularly acute in the peak period of service system production, which tends to cause the network of the Zookeeper cluster to be full, resulting in extremely low service subscription efficiency, and further affecting the performance of service registration.
As the number of nodes written to Zookeeper increases, the Zookeeper snapshot file becomes larger. Each time the Snapshot is written to a disk, disk I/O overshoot occurs. Peak production, due to the high volume of transactions and the high frequency of snapshot files being written, posed a significant risk to the infrastructure. The larger the snapshot file is, the longer the recovery time of Zookeeper node is.
After the Zookeeper election node is re-elected, all Observer nodes need to synchronize full transactions from the new Leader node. If this phase takes too long, the session of the client connected to the Observer node may time out. All temporary nodes under the corresponding providers node are deleted, that is, from the point of view of the registry, these services are offline, and there is no provider exception on the consumer side. Then, these providers reconnect to Zookeeper and re-register the service. This phenomenon of mass service registration flipping in a short time often leads to more serious performance problems of service registration push.

To sum up, it can be concluded that Zookeeper is generally competent as a registry, but it needs to be further optimized in a larger service volume scenario.

3. Optimize the plan

Icbc’s most important optimization measures include the following aspects: subscription delay update, registration center to adopt multiple mode, upgrade to register by node, etc.

1) Subscribe to delayed updates

Icbc has optimized the Zookeeper client component ZKClient to make a small delay for consumers to obtain the provider list after receiving the event notification.

When ZkClient receives a childChange event, installWatch() uses EventThread to resume listening on the node, and getChildren() to read all the children of the node to get the list of providers. And flush the local service provider cache. This is the root of the “5,050 data” problem.

After zkclient receives the childchange() event, icbc waits for installWatch() to do what it was supposed to do. If the service provider changes during this wait, no ChildChange event is generated.

Zookeeper server data is strongly consistent. Consumers also receive event notifications, but they delay reading the provider list. When getChildren() is executed later, The read is the latest data on ZooKeeper, so there is no problem.

Internal pressure test results show that when service providers go online on a large scale, before optimization, each consumer receives a total amount of data of 4.22 million provider nodes, but after 1-second delay, the amount of data becomes 260,000, and the number of Childchange events and network traffic becomes about 5% of the original. After optimization, It can cope with the heavy service coming on and off during peak production periods.

2) Multiple mode

Icbc adopted and optimized the SPI implementation of Registrie-Multiple in the new version of Dubbo to optimize service subscription in multi-registry scenarios.

The original processing logic for service consumers in Dubbo is as follows: when there are multiple registries, the consumer filters the provider according to the invoker cache corresponding to the registry, and if it is not found in the cache corresponding to the first registry, it goes to the cache corresponding to the second registry. If there is an availability problem in the first registry at this time and the data pushed to consumers is missing or even empty, the screening process of consumers will be affected, such as there is no provider exception, call load imbalance, etc.

Multiple registries merge data pushed by multiple registries and then update the cache. Therefore, even if a single registry fails, the pushed data is incomplete or empty, as long as the data from any other registry is complete, the final merged data will not be affected.

In addition, the Multiple registry mechanism is also used in heterogeneous registry scenarios, where problems can be taken offline at any time. This process is completely transparent to service invocation of service nodes, which is suitable for grayscale pilot or emergency switching.

Furthermore, there are additional benefits. The consumer Reference object takes up a lot of JVM memory, and using the Multiple registry mode can save the consumer half of the invoker object overhead. Therefore, the multiple registry mode is highly recommended for multiple registry scenarios.

3) Register by node

Icbc reverse-ported the service discovery logic of Dubbo2.7 and Dubbo3.0, using the service register-discovery model of “register by node”. Here is the triangle of configuration center, metadata center, and registry:

The configuration center is used to store node level dynamic parameters and data of persistent nodes such as Configurations and routers written to Zookeeper.
Metadata center: Storage node metadata, namely, the mapping between the name of each service node (i.e., Applicaton-name) and the services it provides, as well as the class definition information of each service, such as the input and output parameter information of each method.
Registry: In this case, the registry only needs the relationship between the node name of the storage service provider and the actual IP port.

The change in this model has no effect on the service invocation of the consumer. On the consumer side, the service provider invoker cache compatible with stock mode is generated based on the relationship between service Node name and service in the metadata center and the relationship between service node name and actual IP port in the registry.

Pressure test results show that the amount of data in the registry can be reduced to 1.68% by node registration, which has no pressure on the online Zookeeper. The amount of services at the level of 100,000 and nodes at the level of 100,000 can be easily supported.

Future planning

In the future, ICBC also hopes to have the opportunity to go out and deeply participate in the community and contribute its good features in Dubbo, Zookeeper server and ZKClient. For example, in addition to the above optimization points, ICBC has also made refined identification of RPC results on Dubbo. PAAS adaption, port multi-protocol, self-isolation and other capabilities, and a registered circuit breaker mechanism has been added to Zookeeper. At the same time, the synchronization mechanism of Observer is being studied to avoid a series of problems caused by full data synchronization.

In addition, from the perspective of the development of micro-services, Mesh has been one of the hot spots at present. The pain point of ICBC is mainly in the upgrade of service SDK version. Istio is not up to par, and the life of MCP is uncertain. At present, there is a preliminary plan on how to smooth the transition of stock Dubbo service to MESH architecture, but there are still many technical difficulties to overcome.

Welcome Dubbo students who have practice to discuss the problems and experiences in large-scale scenarios, and work together to make Dubbo’s enterprise better!

Practice – Service Discovery of INDUSTRIAL and Commercial Bank of China in constructing financial micro-service Architecture based on Dubbo