Ele. me technical team spent more than one year to realize the overall remote multi-live service, which can flexibly schedule users between multiple remote computer rooms, and realize the goal of free capacity expansion and multi-computer room DISASTER recovery. This article introduces DAL and GZS, one of the five core base components of the project. For an overview of the project and implementation details of other components, please refer to other articles in this series.
GZS (Global Zone Service: Global Status Coordinator)
background
A core of multi-activity transformation is multi-activity traffic routing, which mainly includes three aspects:
- A request originated from outside the Ezone. For example, users place orders, merchants add dishes and so on. This part of the request is routed to the correct Ezone by the API Router. Please refer to https://zhuanlan.zhihu.com/p/32587960 for details
- A non-live business invokes a live business. When non-active services invoke active services, routes from the EZone to the correct EZone must be routed through the SOA Proxy.
- Request initiated by job. A typical scenario is job compensation, which involves batch fetching of qualified data for service operations. In this case, the service provider needs to filter out the data that the eZone needs to process. The job service of each Ezone processes only the service data that the eZone is responsible for.
Not only basic components depend on multiple routes, but certain services also need multiple routes. Therefore, a special component is required to manage multiple routes in a unified manner and coordinate the work of each module when switching between multiple routes.
Multiple route design
The active path consists of three parts: ezone, ShardingID, and ShardingKey. Flexible resource scheduling and failover can be realized by adjusting the routing relationship of the three layers.
- Ezone is a logical equipment room that can independently support key services. Generally speaking, it is located in a physical equipment room.
- Shardingid represents logical service groups divided by geographical location. Services of a SharDID are routed to the same EZone.
- Shardingkey represents the business-related ID that can be routed, including merchant ID, order ID and so on. In order to reduce the service intrusion caused by multi-active transformation, we adopt the policy of multi-active routing using our own service related IDS, and GZS maintains this routing relationship.
Here is a simple routing example
Generally speaking, the relationship between Ezone and Shardid is relatively fixed, while Shardingkey is in a constantly changing state. Shardingkey routing rules are of three types
- Computational: such as latitude and longitude. GZS calculates the shardid to which it belongs according to the geographic location fence to find the target Ezone. This type of Shardingkey has some CPU consumption.
- Mapping type: for example, city ID. Each city ID is basically bound with geographical location. GZS records a mapping table of city ID and SharDID to route. This type of Shardingkey has some memory consumption. Some shardkeys in the mapping type are increasing in total, such as the merchant ID. GZS must maintain routing relationships in real time to ensure second-level updates.
- Embedded: This type of Shardingkey is almost ideal. GZS can directly obtain Shardid from ShardingKey to do routing according to the predefined parsing method.
Design of multi-live Shard switching protocol
The principle of multi-live switchover is as follows:
- Availability priority: in case of a fault switch in the machine room, the system is available first, so that users can place orders for meals, tolerate data inconsistency within a limited period, and then repair afterwards. Each ezone has a full amount of service data. When one ezone fails, other ezones can take over users. Users’ orders placed in one ezone are copied to other Ezones in real time.
- Ensure that data is correct: When data is available, protect data to avoid errors. If the status of an order is found to be inconsistent between the two equipment rooms during switchover or fault, the system locks the order and prevents it from being changed to ensure data correctness.
Shard switching protocols are as follows:
- GZS changes the status of shard1 from “ACTIVE” to “BLOCK” and pushes routes to subscribers (DAL ApiRouter soaproxy e.g.)
- DAL blocks all operations on the corresponding shard in all machine rooms (return operation failed: 1036)
- The API Router blocks all operations on the corresponding shard (the corresponding shard is unavailable) (return fixed error code: 530)
- GZS Changes the status of SHARd1 from BLOCK to RESHARD, changes the ezone corresponding to Shard1 from ezoneA to ezoneB, and the reshard_at field indicates the current time
- DAL prevents old data from being updated on the corresponding SHard in the destination machine room (update created_at > reshard_AT)
- DAL holds block in other machine rooms
- Subscribers are routed according to the new shard-ezone mapping.
- GZS begins the countdown. The countdown ends or the DRC notification synchronization time reaches reshard_AT
- GZS Status changes from “RESHARD” to “ACTIVE”
- DAL unblock
Here is a simple example of switching shard 1’s order traffic from ezone1 to ezone2.
Designed and implemented
The design of GZS is a product of closely related business characteristics.
Why does the SDK use subscription push
Since multi-live query is a high-frequency operation for other multi-live components (DAL, APIRouter, soaProxy), remote query is obviously not up to the task. At the same time, shardingkey is not suitable for computing in GZS Server. Therefore, routing information needs to be pushed to the SDK to ensure efficient query.
Whether the routing subscription is full update or incremental update
GZS defines a set of incremental update protocols for all multi-active routing types. Incremental updates are used for mapped Shardingkeys and full updates are used directly for other types of Shardingkeys.
How do you think about consistency and availability
As mentioned above, multiple paths are three-tier structures with different requirements for consistency. For routes at the Sharding Id layer, there is a requirement for strong consistency, so we set a group of multi-active switching steps to ensure consistency. We sacrificed usability the whole time. For logical Sharding key, the consistent type of embedded type and computational type are bound to the consistent type of Sharding Id. The only thing we need to consider is the mapping Sharding Key, which is mainly designed based on the idea of BASE and does not emphasize real-time high consistency, but requires the final consistency to be achieved within a certain period of time. Take the store ID as an example. When a new store is added, GZS ensures the second-level synchronous mapping relationship, and the out-of-sync state is circumnavigated through services such as geo-location routing.
GZS topology
DAL (Data Access Layer)
background
DAL is a proxy database middleware that supports mysql protocol and has carried all the production mysql traffic of Ele. me before the start of Live Project. DAL is the last line of defense for data correctness in multiple live projects.
Live more DAL bottom
In the previous section, we mentioned the source of live traffic. Traffic from outside the Ezone can be aggregated through the API Router. As for the traffic from inside ezone, it is difficult to confirm whether the multi-live transformation is completed. Therefore, it is necessary to make a backstop in the DAL layer to prevent multiple live data from being corrupted.
DAL Live pocket bottom is mainly embodied in two parts
Global ezone
In the active-active service scenario, data is globally configurable and a large number of reads and a small number of writes are available. For this business, we provide a solution like Global Ezone.
This is a cross-room read/write separation mechanism. Query the database salve of the machine room, write all a Master, DB configuration transparent to the business side.
DAL makes it easy to point services across eZones to the same master. When a failover occurs, the master of the mysql database can be migrated from one Ezone to another by performing an active/standby DB switchover.
Vision of the future
At present, multi-live switching is manually operated, and every decision of multi-live switching always takes a lot of time. In the future, we should make the process of multi-live switching intelligent and automatic.
Literature references
https://yq.aliyun.com/articles/57715
https://coolshell.cn/articles/10910.html
https://en.wikipedia.org/wiki/CAP_theorem
https://zhuanlan.zhihu.com/p/32009822
https://zhuanlan.zhihu.com/p/32587960
The authors introduce
Lin Jing, graduated from Zhejiang University in 2011. She joined Ele. me in 2015 and is now an architect in ele. me Framework Tools Department.