Abstract: In the field of KV database, “strong consistency” is not only a technical term, but also an important requirement of business and operation.
This article is shared by Huawei cloud community “Huawei Cloud PB-level database GaussDB(for Redis) Revealed Issue 7: Gauss Redis and Strong consistency”, the original article is written by gauss Redis official blog.
Qingming festival has just passed, the May Day holiday is coming. Good spring, as wuyuan to see rape! Xiao Yun quickly opened the APP to brush out the remaining 2 tickets, quickly order! Alas, how did not grab again! On second thought, I can also understand: it takes at least 10 seconds from ticketing to placing an order. I’m afraid these two seats will have to be shared by all. Every holiday, millions of friends across the country at the same time check ticket booking, 12306 is how to ensure that the remaining tickets show accurate, tickets are not oversold?
Therefore, unable to restrain curiosity, the author undertook a thorough study. It turns out that behind the problem lies an extremely critical technology in the field of distributed database — data consistency assurance.
1. What is strong consistency?
Before we introduce the concept, let’s simulate a live football match.
Suppose I create an APP that uses the master-slave database shown above in the background. The score is written to the master node and queried by the node sharing user. During the contest, Alice exclaimed that the contest was over, and Bob refreshed the APP, but it showed that the contest was still going on! Bob experiences an obvious data inconsistency and silently gives the APP a bad review…
So what is the cause of the discrepancy?
In asynchronous replication, the master node returns without waiting for the slave node to write. The slave node cannot guarantee the update time due to network latency. Alice and Bob clearly query the same system at the same time and in the same place, but get the correct result before and after. In fact, this is the classic weak consistency.
In fact, in order to solve the single point of failure and improve the throughput performance, the same copy of data is replicated internally in distributed databases, and the redundant copies are scattered on different nodes. Simple asynchronous replication can only build a weakly consistent system, which is difficult to meet business requirements.
So what kind of consistency really counts? What are the categories? Let’s get to know this mysterious family!
1.1 Strong Consistency/Linear Consistency
Degree of reliability:
The highest standard of consistency and the highest difficulty to achieve. The core requirement is that once the write operation is complete, any subsequent client queries must return this new value. For example, once “write B” is complete, you must ensure that B is read. In the process of writing, it is considered that the jump of the value may occur at a certain moment, so it is possible to read a or B.
From a business perspective, the consistency experience is almost silky smooth! Because there is “as if” only one copy of the data inside it, every operation can be ordered atomically even if different nodes are accessed concurrently. Because of this, strongly consistent databases are often used in key places in business architectures.
Etcd is the elder statesman of the Strong Consensus club. It is based on the Raft consensus algorithm, which truly implements strong consistency and therefore plays an important role in Leader election, service discovery and other scenarios. GaussDB(for Redis), as a distributed cloud database, is the spokesperson of strong consistency after years of hard work.
1.2 SequentialConsistency
Degree of reliability:
Less than linear consistency, does not guarantee global timing of operations, but guarantees that each client operation can be executed sequentially. In the figure below, A writes x=10 first and x=20 later; Let’s write x is equal to 99, and then x is equal to 999. When C is read, sequential consistency ensures that 10 is read before 20 and 99 before 999.
Zookeeper is based on the ZAB protocol. All write operations are coordinated by the primary node to achieve sequence consistency.
1.3 CausalConsistency
Degree of reliability:
Further relax the requirement to preserve order only for cause-and-effect operations in concurrent access. Such as:
A writes 3, B reads it, multiplies it by 100 and updates it. In this scenario, since “A writes to 3” has an explicit causal relationship with “B writes to 300”, causal consistency guarantees that 300 is read later than 3.
Causal consistency is often used in blog comment systems, social software, etc. Naturally, the content of our response to a comment should not be shown before the comment itself.
1.4 EventualConsistency
Degree of reliability:
Stop writing and wait for a certain amount of time until all clients can read the same new data, but the time is not guaranteed. Many distributed databases meet final consistency, such as MySQL master/slave clusters.
However, this is a very weak guarantee. Since it is uncertain how long it will take for the system to converge internally, users may experience data inconsistency at any time before then. So final consistency has natural limitations and often confuses business logic.
1.5 Weak Consistency
Degree of reliability:
It is the most “cheeky” because it does not guarantee that data will be read in the future after being written! Weak consistency implementation has low technical threshold and few application scenarios. Strictly speaking, the pure open source Redis master-slave cluster falls into this category.
OK, members of the Consistency family have already met with you. Obviously, the more consistent the database system, the more business scenarios it can support. Some business students whisper, strong consistent technology cattle, but my business is simple, it doesn’t matter. In fact, the opposite is true:
Strong consistency is not only a technical problem, it is a business demand, operation and maintenance demand can not be ignored!
Let’s talk about those things in business that can only be done by strong agreement.
2. Strong consistency is a business necessity
2.1 Counter/current limiter
Counting services are a typical strongly consistent application scenario. In seckilling activities, e-commerce companies often set up Redis master-slave cluster to cache MySQL for the lower level. Because to resist large flow, need Redis counter function to limit the flow. So basically, we initialize counter=5000. Each subsequent business access executes the DECR command, blocking subsequent requests when counter returns to zero. In addition, it resets counter to 5000 at regular intervals to achieve a steady flow.
However, perfect assumptions are not enough!
Open source Redis adopts asynchronous replication. If the network is not smooth, the replication buffer accumulation of the master node often occurs. This will cause the counter from the node to be much larger. At this time, once the master node is down, switch to the slave node to continue to execute the DECR command, the pressure is easy to exceed the threshold, all fall into the lower vulnerable MySQL, may cause the system avalanche at any time!
Therefore, only truly strong consistency can provide reliable counters in current-limiting scenarios.
2.2 Leader election
When a large number of nodes are deployed and the availability requirements are high, the Leader election is often required. Etcd, as a strong consistent KV storage, can cover this scene perfectly. Etcd relies on two functions for Leader election:
1) TTL: Sets the validity period for the key. After the validity period expires, the key will be automatically deleted.
2) CAS: atomic operation on key. (This is only possible with strongly consistent databases)
The design of using ETCD to build the Leader election service is as follows:
1) Agree key, used for preemption during election. Its value is used to save the Leader node name.
2) The TTL is used to set the validity period for the key.
3) At startup: Each participating node tries cascreate key& set TTL. Only one node can be successfully executed under the strong consistency CAS mechanism of etCD cluster. The node becomes the Leader and writes the name to value; The remaining nodes become followers.
4) In operation: each node periodically TTL/2 tries to get key and compares value with its own name:
– If they are the same, they are the Leader. You only need to refresh the TTL of the key every TTL/2.
– If no, the value is Follower. Next, run the CAS create key& command every TTL/2 to set the TTL.
5) When the Leader node exits abnormally and the TTL cannot be refreshed, the key will soon expire. At this point, a new Leader will be generated among the other Follow.
From the principle, it can be seen that strong consistency is the foundation of Leader election. There are many similar “just need” business scenarios, and strong consistency is indispensable.
Well, that’s enough business. Let’s hear from Operations.
3. Strong consistency reduces the burden of operation and maintenance
3.1 The architecture of auxiliary components is complex and problems are difficult to locate
In the background architecture, MySQL primary/secondary hot backup is also a common deployment mode. Data is stored on the local disk. If the primary database fails, the synchronization mechanism of MySQL cannot ensure that the data is in the same state after the primary/secondary switchover. Hence the emergence of the “heavyweight” auxiliary component — Master High Availability (MHA). Let’s take a look at how it is deployed:
The MHA is responsible for helping the slave library keep up to date with the master library during failover, providing approximately consistent data. But this capability requires additional Manager nodes and Node services to be deployed on each MySQL Node. During failover, the Manager first replenishes the lagging data from the slave library and then restores user access by switching the VIP, a process that may take tens of seconds.
Such HA system deployment and maintenance are very complex. If failover is not performed smoothly or data loss occurs, o&M will face a difficult situation. In fact, operation and maintenance students do not want the hands of the system to run stably? Why rely on complex auxiliary components when the database itself provides strong consistency?
Read here, to strong consistent view, I believe that you readers have their own heart of a steelyard. Let’s highlight it again:
Strong consistency is not only a technical problem, it is a business demand, operation and maintenance demand can not be ignored!
From the point of view of product selection, the consistency guarantee provided by open source Redis is weak. While ETCD has strong consistency, it has poor single point write performance and fails to provide attractive data structures such as hash, sorted set, stream… Struggle!
At this point, the aspirant reader will say — I want it all!
GaussDB(forRedis) responds — ME, yes.
4. GaussDB(for Redis) consistent with Strong
Since the beginning of design, GaussDB(forRedis) (Gauss Redis for short) has positioned itself as a “strong consistent KV database”, so it completely abandons the asynchronous replication mechanism of open source Redis. With the advanced “memory separation” architecture of Huawei cloud GaussDB series, full data is sunk into a strongly consistent storage layer (DFV Pool), which exceeds the limit of traditional open source products in core technology.
Let’s get to know the toughness of Gauss Redis:
- The user-purchased instances as a whole provide strongly consistent KV storage.
User services access Gauss Redis in a Proxy cluster without considering the complex internal logic. Multi-point concurrent access instances, read and write operations to meet the strong consistency, no longer need to worry about the open source Redis asynchronous replication inconsistent hidden danger.
- The computing layer intelligently processes data fragmentation and dynamic failover, and sinks all data to the shared storage pool.
The CFGSVR cluster centrally manages ShardServer nodes and automatically shards massive data. In addition, second takeover can be implemented in fault scenarios to strictly prevent data inconsistency in any intermediate state.
- The storage layer implements high performance distributed data persistence through RDMA high-speed network, with three copy redundancy ensuring strong consistency and zero loss.
A DFV Pool is a distributed storage system with strong consistency and high performance. This is huawei-developed company-level Data Lake, which can stably support various full-stack Data services. Gauss Redis breaks through the “small pattern” memory architecture of open source Redis and sinks the full amount of data. Based on the strong consistency guarantee capability of DFV Pool, it brings a broader space for the expansion of user business.
5. Conclusion
Imagine a situation where a critical database is “not working,” and the business layer is busy adding complex, error-prone consistency assurance logic to the system. At the same time, operation and maintenance should always worry about the data backwardness caused by faults…… Does such a system really smell good?
Professional things to professional team to do!
GaussDB(forRedis), the flagship of Huawei Cloud NoSQL channel, has been focusing on data consistency design since its early development. With the advanced GaussDB DFV Pool, GaussDB(for Redis) consistently provides users with truly robust and consistent massive KV storage solutions.
6. The appendix
Author: Huawei Cloud Gauss Redis team
For more technical articles, follow the official Gauss Redis blog:
Bbs.huaweicloud.com/community/u…
Read more:
1. Huawei Cloud PB-level database GaussDB(for Redis) Revealed Phase 1: Separation of Redis and Memory
Bbs.huaweicloud.com/blogs/24562…
2. Strong Consistency Models
Aphyr.com/posts/313-s…
3. Redis Data Consistency Analysis
Baobing. Making. IO / 2017/12/23 /…
4. How to solve the problem of Inconsistency between Master and slave data of Redis
Segmentfault.com/a/119000001…
5. Don’t Settle for Eventual Consistency
Queue.acm.org/detail.cfm?…
6. The Disaster of Microservices – The Final Consensus
Xargin.com/disaster-of…
7. ETCD Application Scenario
Tonydeng. Making. IO / 2015/10/19 /…
Click to follow, the first time to learn about Huawei cloud fresh technology ~