In this article I want to talk to you about the architectural evolution of Redis.

Now Redis is becoming more and more popular, almost be used in a lot of projects, I do not know when you use Redis, have you ever thought, Redis exactly how stable, high performance to provide services?

You can also try answering these questions:

• My scenario with Redis is very simple, what’s wrong with using only the standalone Redis?
• My Redis is down. What if my data is lost? How can I ensure that my business applications are not affected?
• Why do you need a master-slave cluster? What are its advantages?
• What is a sharded cluster? Do I really need a sharded cluster?
• .

If you already know something about Redis, you’ve probably heard of data persistence, master-slave replication, and sentry. What are the differences and connections between them?

If you have any doubts, in this article I will take you step by step to build a stable, high-performance Redis cluster from 0 to 1 and from 1 to N.

In the process, you can learn what optimizations Redis has taken to achieve stability and high performance, and why.

By mastering these principles, you will be able to use Redis with ease.

This article has a lot of dry material. I hope you can read it patiently.

Start with the simplest: Redis standalone

First, let’s start with the simplest scenario.

If you have a business application and need to introduce Redis to improve the performance of the application, you can choose to deploy a standalone version of Redis, like this:

The architecture is very simple, your business application can use Redis as a cache, query data from MySQL, write it to Redis, and then read it from Redis. Because Redis data is stored in memory, this is very fast.

If your business is not very large, this architecture model will probably meet your needs. Isn’t that easy?

Over time, as your business grows and more data is stored in Redis, your business applications become more and more dependent on Redis.

However, suddenly one day, your Redis goes down for some reason, and all your business traffic will be sent to the backend MySQL, which will cause your MySQL pressure to increase dramatically, or even overwhelm MySQL.

What should you do?

I’m guessing your solution is to restart Redis so it can continue to work.

However, since all the data in Redis is in memory, even if you restart Redis now, the previous data is also lost. Although Redis can work normally after restart, but because there is no data in Redis, the business traffic will still be sent to the back-end MySQL, MySQL is still under great pressure.

What can I do? You are lost in thought.

Is there any good way to solve this problem?

Since Redis only stores data in memory, is it possible to write a copy of the data to disk?

In this way, when Redis restarts, we quickly restore the data from the disk to memory so that it can continue to serve normally.

Yes, this is a good solution, the process of writing data in memory to disk, called data persistence.

Data persistence: Be prepared

Now, here’s how you envision Redis data persistence:

But what about data persistence?

I guess you can easily imagine a scenario where Redis writes a copy to disk every time it writes to memory, like this:

Yes, it’s the simplest and most straightforward solution.

However, if you think about it carefully, this scheme has a problem: each write operation of the client needs to write both memory and disk, and the time of writing disk is much slower than that of writing memory. This is bound to affect Redis performance.

How do you get around this problem?

We can optimize this by saying that Redis write memory is done by the main thread, write memory is finished and return results to the client, and then Redis uses another thread to write disk, thus avoiding the performance impact of the main thread write disk.

This is indeed a good plan. Beyond that, let’s think about another way to persist data.

At this point you need to consider the Redis usage scenario.

Remember, when we use Redis, what scenarios do we usually use it for?

Yes, caching.

Using Redis as a cache means that although Redis does not store the full amount of data in the cache, our business application can still query the backend database for the data that is not in the cache, but it is slower to query the backend data, but it has no impact on the business results.

Because of this feature, our Redis data persistence can also be done in the way of “data snapshot”.

So what is a data snapshot?

To put it simply, you can think of it this way:

1. You think of Redis as a water glass, and writing data to Redis is like pouring water into the glass
2. At this point, you take a camera and take a picture of the cup. At the moment of taking a picture, the capacity of the water in the cup is recorded in the picture, which is the snapshot of the data of the cup

In other words, the data snapshot of Redis is to record the data in Redis at a certain time, and then only need to write the data snapshot to disk.

The advantage is that data is written to disk “once” only when persistence is needed, and the disk is not manipulated at all other times.

Based on this scheme, we can periodically take snapshots of Redis data and persist the data to disk.

Redis’ RDB ‘and’ AOF ‘are Redis’ RDB and AOF’

• RDB: Persist snapshots of data to disk at a certain point in time (create a subprocess to do this)
• AOF: Each write operation is persisted to disk (the main thread writes to memory, and the main thread or subthread can be configured to persist data according to the policy)

In addition to the differences mentioned above, they also have the following characteristics:

1. RDB uses binary and data compression to write data to disks, which reduces file size and speeds data recovery
2. AOF records each write command and provides the most complete data. However, the file size is large and the data recovery speed is slow

If you had to choose the persistence scheme, you could do this:

1. If your business is not sensitive to data loss, use the RDB solution to persist data
2. If your business has high data integrity requirements, use AOF to persist data

If your business has high requirements for Redis data integrity and you choose AOF, you will encounter these problems:

1. AOF records each write operation. As time goes by, the AOF file becomes larger and larger
2. With such a large AOF file, data recovery becomes very slow

What about this? Are data integrity requirements higher and data recovery more difficult? Is there any way to reduce the file size? How about faster recovery?

Let’s continue to analyze the characteristics of AOF.

Since every write operation is recorded in the AOF file, the same key may be modified several times, so we only keep the value that was modified last time, is that ok?

Yes, that’s what we often hear about “AOF rewrite”, or you can interpret it as “AOF slimming”.

We could rewrite the AOF file periodically to prevent it from growing in size so that the recovery time could be shortened.

Further thinking, is there any way to further shrink the AOF file?

To review the characteristics of RDB and AOF as we mentioned earlier:

1. RDB is stored in binary + data compression mode, and the file size is small
2. AOF records each write command and provides the most complete data

Can we take advantage of their respective strengths?

Sure, this is Redis’s “mixed persistence.”

Specifically, when AOF rewrite, Redis wrote a snapshot of the data in the AOF file in RDB format and appended every write command generated in the process to the AOF file. Because RDB is written in binary compression, AOF files become smaller.

When you use AOF files to recover data, the recovery time is even shorter!

Mixed persistence is supported only in Redis 4.0 and later.

With this optimization, your Redis will no longer need to worry about instance downtime, and when downtime occurs, you can quickly recover data in Redis using persistent files.

But is that all right?

If you think about it, even though we’ve optimized persistent files to a minimum, it still takes time to recover data, and your business applications are still affected in the meantime.

Let’s see if there’s a better solution.

If an instance goes down, it can only be resolved by recovering data. Can we deploy multiple Instances of Redis and keep their data synchronized in real time, so that when one instance goes down, we can choose one of the remaining instances to continue to provide service?

Yes, this solution is called “master-slave replication: Multiple copies”.

Primary/secondary replication: multiple copies

At this point, you can deploy multiple instances of Redis, and the architectural model looks like this:

The node that reads and writes data in real time is called master, and the node that synchronizes data in real time is called slave.

The advantages of the multi-copy scheme are:

1. Shorten the unavailability time: If the master is down, we can manually promote the slave to master to continue providing services
2. Improve read performance: The slave shares some read requests to improve the overall application performance

This is a good solution, not only saves data recovery time, but also improves performance, so what’s wrong with it?

You can think about it.

The problem is that when the master goes down, we need to “manually” promote the slave to master, which also takes time.

It’s much faster than recovering data, but it still requires human intervention. Once human intervention is required, the response time and operation time of the human must be counted, so your business application will still suffer during this time.

How to solve this problem? Can we automate this process of switching?

For this, we need a fail-over mechanism, which is what we often hear about sentinels.

Sentry: Automatic failover

Now we can introduce an “observer” to monitor the master’s health in real time. This observer is the “sentinel”.

How to do it?

1. At regular intervals, the sentry asked the master if he was okay
2. If the master replies normally, the status is normal. If the master replies timeout, the status is abnormal
3. The sentry detects an anomaly and initiates a master/slave switchover

With this scheme, there is no need for people to intervene in processing, everything becomes automated, isn’t it cool?

But there’s another problem, because if the master is fine, and the sentry asks the master, and there’s something wrong with the network between them, the sentry might misjudge.

How to solve this problem?

The answer is that we can deploy multiple sentinels on different machines to monitor the master’s state, and the process looks like this:

1. Multiple sentinels, at regular intervals, ask the master if he is healthy
2. If the master replies normally, the status is normal. If the master replies timeout, the status is abnormal
3. Once one sentry decides that the master is out of order (whether it’s a network problem or not), the other sentries are asked. If multiple sentries (setting a threshold) agree that the master is out of order, the master is truly out of order
4. After negotiation, multiple sentries determine that the master is faulty and initiate a master/slave switchover

So we use multiple sentinels to determine the master’s state in consultation with each other, which greatly reduces the probability of misjudgment.

After sentry negotiation determines that the master is abnormal, there is a question: which sentry should initiate the master/slave switch?

The answer is to select a sentry “leader” who switches over.

The question is again, how to choose this leader?

Imagine, in real life, how elections work?

Yes, voting.

In electing sentry leaders, we can make an election rule like this:

1. Each sentry asked the others to vote for him
2. Each sentry votes only once for the first sentry who asks for a vote
3. The sentry, who gets more than half of the votes first, is elected leader and initiates a master-slave switch

In fact, this election process is what we often hear about: “consensus algorithms” in distributed systems.

What is a consensus algorithm?

We deploy sentinels on multiple machines that need to work together on a single task, so they form a “distributed system.”

In the field of distributed system, the algorithm of how many nodes reach a consensus on a problem is called consensus algorithm.

In this scenario, multiple sentries negotiate to elect a leader they all agree on, using consensus algorithms.

The algorithm also stipulates that the number of nodes must be an odd number, so that even if a node fails in the system, the remaining “more than half” of the nodes are in normal state, they can still provide correct results. In other words, the algorithm is also compatible with the existence of faulty nodes.

There are many consensus algorithms in distributed systems, such as Paxos, Raft, the Sentinel election Leader scenario, which uses Raft consensus algorithm because it is simple enough and easy to implement.

Now we have multiple sentinels monitoring the state of Redis, so that we can avoid the problem of misjudgment. The architectural model looks like this:

Okay, so here’s a summary.

Your Redis from the simplest stand-alone version, through data persistence, master/slave multiple copies, sentinel cluster, this way optimized down, your Redis both performance and stability, are getting higher and higher, even if the node failure, do not worry about.

Your Redis should be deployed in this architectural pattern, and it should basically run stably for a long time.

.

As the volume of your business explodes over time, can your architectural model still handle that amount of traffic?

Let’s analyze it:

1. Stability: Redis is down, we have sentinel + copy, can automatically complete master/slave switch
2. Performance: As the number of read requests increases, multiple slaves can be deployed to separate read and write requests and share read pressure
3. Performance: The number of write requests increases, but we only have one master instance. What if that instance reaches the bottleneck?

See, as you get more and more write requests, a master instance might not be able to handle that much write traffic.

To solve this problem perfectly, you need to consider using “sharded clustering” at this point.

Sharded cluster: scale horizontally

What is a “sharded cluster”?

Simply speaking, one instance cannot bear the pressure of writing, so can we deploy multiple instances, organize these instances according to certain rules, treat them as a whole, and provide services to the outside world, so as to solve the bottleneck problem of centralized writing one instance?

So, the architecture model now looks like this:

Now the question is, how do you organize all these instances?

We make the rules as follows:

1. Each node stores a certain amount of data, and the total amount of data on all nodes is the total amount of data
2. Make a routing rule to route different keys to a fixed instance for reading and writing

Fragment clusters can be divided into two categories according to the location of routing rules:

1. Client Sharding
2. Server Sharding

Client sharding means that the routing rules for keys are placed on the client side, as follows:

The disadvantage of this solution is that the client needs to maintain the routing rules, which means that you need to write the routing rules into your business code.

How do you not couple routing rules into your business code?

You can optimize this by encapsulating the routing rule into a module that can be integrated when needed.

This is the solution adopted by Redis Cluster.

Redis Cluster has sentry logic built in, eliminating the need to deploy sentries.

When you use Redis Cluster, your business application needs to use the Redis SDK, which integrates the routing rules, so you don’t need to write them yourself.

Let’s look at server sharding.

In this scheme, routing rules are not left to the client, but an “intermediate Proxy layer” is added between the client and the server. This Proxy is what we often hear about Proxy.

The routing rules of data are maintained in this Proxy layer.

In this way, you don’t need to care how many Redis nodes there are on the server, you just need to interact with the Proxy.

Proxy will forward your request to the corresponding Redis node according to routing rules. In addition, when the cluster instances are not enough to support larger traffic requests, it can also be horizontally expanded and add new Redis instances to improve performance. All these are transparent and unaware to your clients.

The open source Redis sharding cluster scheme in the industry, such as Twemproxy and Codis, is adopted by this scheme.

There are many details involved in data expansion of sharded cluster, which is not the focus of this article, so we will not go into details for the time being.

At this point, when you use sharding cluster, for the future of greater traffic pressure, can be easy to face!

conclusion

So let’s summarize how we built a stable, high-performance Redis cluster step by step.

First of all, when using the simplest stand-alone Redis, we found that the data could not be recovered when Redis failed and went down. Therefore, we thought of “data persistence”, which persisted the data in memory to the disk, so that the data could be quickly recovered from the disk after Redis was restarted.

In data persistence, we are faced with the problem of how to persist data to disk more efficiently. Later, we found that Redis provided RDB and AOF schemes, corresponding to data snapshot and real-time command recording respectively. When data integrity is not high, we can choose RDB persistence. If you have high data integrity requirements, you can choose AOF persistence.

However, we found that the size of AOF files would get bigger and bigger over time. At this point, we came up with an optimization scheme to slim it down by using AOF rewrite. Later, we found that we could combine the advantages of RDB and AOF. The use of “hybrid persistence” in AOF rewrite further reduced the size of AOF files.

Later, we found that although we could restore the data through data recovery, it took time to recover the data, which meant that business applications were still affected. We further optimized the “multi-copy” scheme to keep multiple instances synchronized in real time, so that when one instance fails, others can be manually promoted to continue providing service.

But it also has a problem, manual uplift instance, need human intervention, human intervention operation also need time, we began to think of some way to make the process automation, so we introduced the “sentinel” cluster again, the sentry cluster by mutual agreement, found the fault node, and can automatically switch, thus greatly reduce the impact on business applications.

Finally, we focused on how to support more write traffic, so we introduced a “sharded cluster” to solve this problem, allowing multiple Redis instances to share the write load. In the future, we can add new instances and scale horizontally to further improve the performance of the cluster.

At this point, our Redis cluster can provide long-term stable, high-performance services for our business.

I’ve drawn a mind map to help you better understand how they relate to each other and how they evolve.

Afterword.

This should give you some insight into how to build a stable, high-performance Redis cluster.

In fact, the optimization ideas discussed in this article revolve around the core idea of “architectural design” :

• High performance: Read/write separation and fragmented cluster
• High availability: data persistence, multiple copies, and automatic failover
• Easy to expand: fragment cluster, horizontal expansion

When we talk about sentinels and sharded clusters, we also talk about distributed systems:

• Distributed consensus: Sentry Leadership elections
• Load balancing: Fragmented cluster data fragmentation and data routing

Of course, with the exception of Redis, you can use this thinking to build any data cluster and optimize it to see how it works.

For example, when you use MySQL, you can think about how MySQL is different from Redis. MySQL in order to achieve high performance, high availability, and how to do? The idea is similar.

We now see distributed systems and data clusters everywhere. I hope you can understand how these software evolved step by step, what problems they encountered in the process of evolution, and what solutions and trade-offs the designers of these software designed to solve these problems.

You only understand the principle of which, master the analysis of the problem, the ability to solve the problem, so that in the future development process, or learning other excellent software, can quickly find the “focus”, in the shortest time to master it, and can play their advantages in the practical application.

In fact, this thinking process is also the idea of “architectural design”. When designing software architecture, you are faced with the scenario of finding problems, analyzing problems, solving problems, evolving and upgrading your architecture step by step, and finally achieving a balance between performance and reliability. There’s a lot of software out there, but the ideas of architecture don’t change, and I want you to really absorb those ideas so that you can change.