1. High availability of RabbitMQ

RabbitMQ is highly available in master/slave mode. RabbitMQ has three modes: single-machine mode, common cluster mode, and mirrored cluster mode. (1) Single-machine mode: The single-machine mode is demo level and will not be used in production. (2) Common Cluster mode The common cluster mode is to start multiple rabbitMQ instances on multiple machines, one on each machine. The queue created will only be placed on one rabbitMQ instance, but all other instances will synchronize the metadata of this queue. When you’re consuming, if you’re connected to another instance, it’s going to get the message from the instance that has the queue and it’s going to send it back to you.

And if that instance of the square queue goes down, then no other instance will be able to pull data; If message persistence is not enabled, the message will be lost. Even if message persistence is enabled, the message may not be lost, but the data can continue to be pulled until the instance is recovered. So this doesn’t provide high availability, it just improves throughput by having multiple nodes in the cluster service reads and writes to a queue. (3) This mode, which RabbitMQ provides, is the true high availability mode. Unlike normal clustering, you create a queue in which metadata and messages are stored in multiple instances, and each time a message is written to the queue, Will automatically synchronize messages to multiple queues.



The disadvantages are:



How to enable the mirroring cluster mode:

2. High availability of Kafka

(1) A basic architecture of Kafka: Multiple brokers, one of which is a node; You create a topic that can be divided into multiple partitions. Each partition can reside on a different broker, and each partition holds a portion of the data. This is a naturally distributed message queue.

Rabbitmq is not actually a distributed message queue, it is a traditional message queue that provides some clustering, HA mechanism, because rabbitMQ data in a queue is stored on a node regardless of configuration, and in a mirror cluster, each node holds the entire data in the queue.

Kafka had no HA mechanism prior to 0.8, which meant that when any broker went down, partitions on that broker were lost, unreadable and unwritable, and had no high availability.

Kafka after 0.8 mentioned the HA mechanism, also known as replica mechanism. The data of each partition is synchronized to other machines to form replica copies. Then all replicas elect a leader. The producers and consumers deal with the leader, and the other replicas are followers. On writes, the leader synchronizes data to all followers, and on reads, it simply reads from the leader. Why can only read and write leader: If you can read and write each follower at will, you need to worry about data consistency and the system complexity is too high, which may cause problems. Kafka makes all data replicas of a partition distributed across different machines more uniform. This improves fault tolerance. This is highly available, because if a broker goes down, that’s fine. The partition of that broker has copies on other machines. If the partition has a leader on it, then a modern leader will be elected. Continue reading and writing to the new leader.

Write the message:



Read data:

Previous article “Message queue usage, Pros and cons, technology selection” next article “How to Ensure that messages are not re-consumed”