From the stone fir code farm curriculum collation, has filled the map

1. Interview questions

How to ensure that message queues are highly available?

2. Interviewer psychological analysis

If someone asks you about MQ, high availability is a must, because of the disadvantages of MQ, which I have already mentioned, there are many, resulting in reduced system availability, etc. So as long as you use MQ, some of the next points to ask are bound to revolve around how the shortcomings of MQ can be addressed.

If you foolishly use MQ and never give it a second thought, you will be left with the impression that you are a technical slacker without any thought. If such students recruit to come in to do a 20K salary within the ordinary younger brother also ok. If you recruit to do more than 20 k salary of high, it would be miserable, let you design a system, there must be a pile of pits, accidents suffered by the company, the team on the back.

Last year, a very large Internet company, with a very core system, neglected MQ and did not consider how to ensure high availability of MQ. What if MQ failed, which resulted in the system being unavailable for several hours and the company losing tens of millions of yuan? The team took the blame for your trouble and your boss helped you take the blame

3, interview question analysis

It’s a good question to ask, because how can Kafka be highly available? How to ensure the high availability of ActiveMQ? This is a bad question for an interviewer who probably uses RabbitMQ and has never used Kafka before. I thought that was a sign of difficulty.

So skilled interviewers are asking how can HIGH availability of MQ be guaranteed? So which MQ you have used, you can say what you understand about the high availability of that MQ.

(1) High availability of RabbitMQ

RabbitMQ is an example of how to implement the first MQ high availability since it is master-slave based.

Rabbitmq has three modes: single-machine mode, common cluster mode, and mirrored cluster mode

1) Single-machine mode

Is demo level, generally is your local start to play, no one production with standalone mode

2) Common cluster mode

This means starting multiple rabbitMQ instances on multiple machines, one for each machine. But the queue you create will only be placed on one rabbtimq instance, but each instance will synchronize the metadata of the queue. When you’re done consuming, in fact, if you’re connected to another instance, that instance will pull data from the instance where the queue is.

This way is really troublesome, and it’s not very good, it’s not distributed, it’s just a normal cluster. Because this causes you to either have consumers connect randomly one instance at a time and pull data, or to have fixed connections to instances of that queue that consume data, which has pull overhead, which leads to single-instance performance bottlenecks.

And if the instance in which the queue is placed goes down, then other instances will not be able to pull from that instance. If you enable message persistence and let RabbitMQ store messages, they will not be lost until the instance is restored.

So it’s a bit awkward, because there’s no such thing as high availability. It’s more about throughput, which means that multiple nodes in the cluster serve reads and writes to a queue.

3) Mirror cluster mode

This mode, which is called the high availability mode of RabbitMQ, differs from the normal clustering mode in that the queue you create, both the metadata and the messages in the queue are stored in multiple instances, and then each time you write a message to the queue, it is automatically synchronized to the queues of multiple instances.

In this case, the advantage is that if one of your machines goes down, you can use the other machines. The disadvantages are, first, the performance overhead is too high, the message synchronizes all machines, resulting in the network bandwidth pressure and consumption is heavy! Second, it’s not scalable, because if you add machines to a queue that’s heavily loaded, you add machines that also contain all the data in that queue, there’s no way to linearly scale your queue

So how to enable the mirror cluster mode? I here is simple, avoid interview someone ask you you don’t know, actually very simple rabbitmq has good management console, is a new strategy in the background, this strategy is the mirror of cluster pattern, the specified time can require the data synchronization to all nodes, also can request is synchronized to a specified number of nodes, Then when you create a queue again, apply this policy and the data will be automatically synchronized to the other nodes.

(2) High availability of Kafka

This is a natural distributed message queue, meaning that the data for a topic is distributed across multiple machines, with each machine hosting a portion of the data.

In fact, rabbitMQ and the like are not distributed message queues, they are traditional message queues that provide some clustering, HA mechanism, because rabbitMQ queue data is stored on a node, in a mirror cluster, no matter how fun it is. It is also the complete data of this queue that each node puts.

Prior to Kafka 0.8, there was no HA mechanism. When any broker went down, partitions on that broker became invalid, unwritable and unreadable, and there was no high availability.

After Kafka 0.8, HA mechanism is provided, namely replica mechanism. The data of each partition will be synchronized to other machines to form multiple replica copies. Then all replicas elect a leader, so production and consumption deal with the leader, and the other replicas are followers. On writes, the leader synchronizes data to all followers, and on reads, the leader simply reads the data. Can only read and write to leader? Simply, if you can read and write to each follower at will, then you have to worry about data consistency, because the complexity of the system is so high that problems can easily occur. Kafka evenly distributes all replicas of a partition to different machines to improve fault tolerance.

In this way, there is what is called high availability, because if a broker breaks down, it is ok. The partition on that broker has copies on other machines. If there is a leader on that partition, a new leader will be elected. Continue reading and writing to the new leader. This is called high availability.

When the data is written, the producer writes to the leader, who then writes the data to the local disk, and the other followers actively pull the data from the Leader themselves. Once all the followers have synchronized their data, they send an ACK to the leader, who returns a write success message to the producer after receiving an ACK from all the followers. (Of course, this is just one pattern, and you can tweak this behavior)

When consuming, the user will only read a message from the leader. However, the user will only read a message if it has been ack successfully synchronized by all followers.

In fact, this mechanic, in depth, can go a lot further, but I’ll go back to the theme and orientation of this course, focusing on interviews, and at least you get a sense of how Kafka ensures high availability, right? Don’t know nothing, you can also draw pictures for the interviewer. If you meet an interviewer who is really a kafka expert, you can only say sorry, you have not studied too deeply.

But it’s important to understand that there’s a tradeoff, and you’re going to have to do a quick dive into kafka, not a deep dive into Kafka, which you don’t have time for. You just have to make sure that you probably didn’t know this before, but now that you do, you can probably say something when they ask you. And then there are a lot of other candidates, maybe worse than you, who haven’t seen this, who have been asked to the point where they can’t answer, whereas you can say something, which is sort of the point.