00 preface
I am responsible for the report system of our company, and Xiao Pang is my younger brother. With the increase of business volume, a single instance could not support, so I set up several Instances of Redis to achieve the master-slave mode.
Studious fat ask me ah, far elder brother, how is the data between many instances to keep synchronous ah? Would you please teach me
I picked up in the hand of 82 years of boiled water pursed one mouthful, say with small fat: you see this article first, learned to operate, I again tell you about principle.
Juejin. Cn/post / 684490…
(PS: At the end of the article, there are questions I prepared for the big factory interview)
0.1 past wonderful
How is the MySQL query executed?
MySQL index
MySQL logs
MySQL transactions and MVCC
MySQL lock mechanism
Redis basis
Redis persistence
01 Primary/secondary Replication
The high reliability of Redis is mainly guaranteed by two points: one is to minimize data loss and the other is to minimize service interruption. Persistence ensures the first point; The second is guaranteed by the Redis cluster, which has multiple instances to keep data synchronized.
1.1 Read/Write Separation
Redis provides a master-slave library mode to ensure consistent data copies, with read-write separation between the master and slave libraries.
- Read operation: master library, slave library can receive;
- Write operations: first performed to the master, then the master synchronizes the write operations to the slave.
Why separate reading from writing?
Looking at the figure above, if all libraries were writable, a key would occur that would have different values in different instances. For example, the corresponding key K1 in the figure above has different values of V1, v2 and v3 in different instances, which is absolutely unacceptable.
Someone said, you can lock it
However, this involves locking, negotiating between instances to complete changes, and a lot of overhead, and Redis is known for being fast, which is unacceptable.
What should I do then? Read/write separation
Master/slave mode uses read/write separation, and all data changes are made only in the master library. The master database has the latest data, which is synchronized to the slave database so that the data from the master and slave databases is consistent.
The problem is synchronization. How do master and slave libraries synchronize? Let’s talk about it
1.2 Full Replication
When multiple Redis instances are started, replicaof (slaveof prior to Redis 5.0) commands are used to form the relationship between the master and slave databases, and then the first synchronization is performed in three stages.
For example, you have instance 1 (127.0.0.1) and instance 2 (127.0.0.2). After replicaof is executed in instance 2, instance 2 becomes the slave of instance 1.
replicaof 127.0. 02. 6379
Copy the codeAfter the relationship is established, Redis makes the first full copy. The process is as follows:
The first step, the process of establishing a connection and negotiating synchronization between master and slave libraries, is mainly to prepare for full replication.
The slave database sends the psync command to the master database to indicate data synchronization. The master database starts the replication according to the parameters of this command. The psync command contains the runID of the master library and the replication progress offset.
- RunID, a random ID automatically generated when each Redis instance is started, is used to uniquely identify the instance. Set runID to “? “because you do not know the runID of the master library when the slave and master library are first copied. .
- Offset: the value is -1, indicating the first replication. The master library receives the psync command and returns the FULLRESYNC response to the slave library with two parameters: the master runID and the master’s current replication progress offset.
The FULLRESYNC response represents the full replication taken by the first replication, that is, the master database copies all the current data to the slave database.
In the second step, the master library executes the bgsave command to generate an RDB file and send it to the slave library. After receiving the data from the library, clear your own data (to avoid inconsistency between master and slave) and load the data locally.
The master library can still receive requests while synchronizing data to the slave library. However, the write operations in these requests are not recorded in the RDB file just generated.
To ensure primary/secondary consistency, the primary library uses a Replication buffer in memory to record all write operations received after the RDB file is generated.
Third, the master library sends the changes in the Replication buffer to the slave library, which then re-performs them. In this way, master and slave are synchronized.
1.3 The master-slave cascade disperses the pressure
There is another problem: if there are too many slave libraries that have direct relationships with the master library, the master library will fork too many threads to generate RDB and send RDB files, which will cause the master library to block. To do that?
This is the master-slave cascading pattern: Select an instance with more resources as the cascading slave and a leaf node as the slave and execute the following command to make the selected slave the master, thus forming the cascading relationship.
Replicaof specifies the IP address of the selected slave library6379
Copy the code
As shown above, the master library has a master-slave relationship with two instances, and the slave library has a master-slave relationship with another slave library, similar to a tree structure. In this way, the main library can reduce stress.
In the figure above, the master library originally had a master-slave relationship with four slave libraries; With cascading, only two master-slave relationships need to be established, with one of the slave libraries doing the rest. It’s kind of like the father and the son.
This process, also known as long-connection-based command propagation, avoids the overhead of frequently establishing connections.
1.4 Incremental Replication
Come to think of it, the above master-slave replication has a problem: what if the network between the master and slave is disconnected?
In fact, before Redis 2.8, the master and slave were disconnected and full copy was made. But this method is too expensive and has been phased out. Redis after 2.8 uses incremental replication.
Incremental replication relies on a circular buffer, repl_backlog_buffer (as long as there are slave stocks), to which the master library writes write commands during the disconnection. Repl_backlog_buffer is a circular buffer, where the master library records where it wrote and the slave library records where it read.
As shown in the figure above, the master and slave point to the same position when they are not disconnected. Write from main library, read from library, go straight ahead. Suddenly, the connection is disconnected, the slave library cannot write, and the master library continues reading.
After the connection is restored, the library sends psync {runID}{offset} to tell the master library where it last read. The master library uses offset to find the location in repl_backlog_buffer where the slave library is disconnected, and the offset between the master and slave is what needs to be synchronized for incremental replication. For example, put D e and PUT D F in the figure above.
This incremental data is copied into repl_buffer, which the master library sends to the slave library to read in.
The entire process based on this incremental replication looks like this:
You might have noticed, right? Repl_backlog_buffer is a circular cache, and if the slave is disconnected for too long, it can cause unread operations from the slave to be overwritten by newly written operations from the master, causing data inconsistencies between the master and slave. What should I do?
After the connection is restored, the master library determines whether the connection is overwritten according to the offset position read from the library last time. If so, only one full synchronization can be performed obediently after the slave is connected to the master.
To avoid full synchronization, you can make the size of the REPL_backlog_buffer larger by repl_backlog_size.
The calculation formula is as follows: Buffer space = command writing speed of the primary library * operation size – Command transmission speed between the primary and secondary libraries * operation size
Repl_backlog_size = Buffer size x 2
This way, the probability of full synchronization is much lower.
02 summary
This article mainly explains why the master-slave mode of Redis should be separated from read and write. The process and principle of Redis full copy; How do I reduce the replication pressure on the master library when there are many slave libraries? How does Redis synchronize data if the master and slave are disconnected? I hope you got something out of it.
Well, that’s doggie’s summary of Redis master-slave replication. Thanks to all the tech community leaders for their efforts, especially geek time, which is awesome. If I have seen further, it is by standing on your shoulders. Hope you found this article helpful, and we’ll see you in the next article
2.1 Shoulders of giants
- Redis Design and Implementation
- time.geekbang.org/column/article/272852
03 Big factory interview questions & e-books
If you read here and like this article, please click on it.
I don’t know what to give you. Just a few hundred ebooks and the latest 2021 interview materials. Wechat search JavaFish reply ebook send you 1000+ programming ebook; 50 sets of interview questions from large factories; Reply 1024 send you a complete set of Java video tutorials.
The interview questions have answers, as follows: If you need them, come and take them, absolutely free, no tricks.
