preface

There is a limit to the performance of any single RabbitMQ service, and RabbitMQ is no exception. When a single RabbitMQ service is unable to handle messages, clustering can be used to achieve high availability and load balancing.

How much RabbitMQ clusters know

Normally, each service is called a node in a RabbitMQ cluster, and there are two types of nodes:

  • Memory node: Metadata is stored in memory. In order to synchronize data after the restart, the MEMORY node stores the address of the disk node in the disk. In addition, if the message is persisted, the memory node also stores the address in the disk. Because the memory node is fast to read and write, the client usually connects to the memory node.
  • Disk node: The metadata is stored on a disk (the default node type). Ensure that at least one disk node is used. Otherwise, data cannot be restored in the event of a system breakdown and high availability of the cluster cannot be achieved.

PS: Metadata refers to basic information including queue name attribute, switch type name attribute, binding information, Vhost and so on, excluding message data in the queue.

There are two main clustering modes in RabbitMQ: normal clustering mode and mirrored queue mode.

Common Cluster mode

In normal clustering mode, only metadata is synchronized between nodes in the cluster, that is, message data is not synchronized. So the question is, what if we connect to node A, but the messages are stored on node B?

Whether a producer or consumer is connected to a node that does not store queued data, it is internally forwarded to the node that stores queued data for storage. Although internal forwarding can be achieved, because messages are only stored in a node, if the node fails, will the message be lost? This problem does exist, so this normal clustering pattern is not serving the purpose of high availability.

Mirror queue mode

In mirrored queue mode, metadata and message content are synchronized between nodes, improving availability. While this solution improves availability, it also affects performance to a certain extent because of the network overhead associated with synchronizing data.

The RabbitMQ cluster is built

Let’s try to set up a RabbitMQ cluster:

  1. Delete the old data rm -rf /var/lib/rabbitmq/mnesia or delete var/lib/rabbitmq/mnesia in the install directory if you have started a PC version before. So perform the command rm – rf/usr/local/rabbitmq_server – 3.8.4 / var/lib/rabbitmq/mnesia /.

  2. You will need to run the following commands to start the RabbitMQ service with three different ports. You will need to specify the back-end management system port in addition to the RabbitMQ service port, and you will need to specify the node name prefix, because the cluster is used for communication, so the node name must be unique. The default node name is rabbit@hostname, and the following command indicates that the prefix is specified:

RABBITMQ_NODE_PORT=5672 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15672}]" RABBITMQ_NODENAME=rabbit1 rabbitmq-server  -detached
RABBITMQ_NODE_PORT=5673 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15673}]" RABBITMQ_NODENAME=rabbit2 rabbitmq-server  -detached
RABBITMQ_NODE_PORT=5674 RABBITMQ_SERVER_START_ARGS="-rabbitmq_management listener [{port,15674}]" RABBITMQ_NODENAME=rabbit3 rabbitmq-server  -detached
Copy the code

Start to enter/usr/local/rabbitmq_server 3.8.4 / var/lib/rabbitmq/mnesia directory view, found that created three nodes information:

Also through the ps – ef | grep rabbit can also be found three service process is started.

  1. The three services started now are not connected with each other. Now we need to take one node as the main node, and then the other two nodes need to join the main node to form a cluster service. It should be noted that before joining the cluster, node information needs to be reset, that is, nodes with data are not allowed to join the cluster.
// The rabbit2 node is added to the cluster after being resetrabbitmqctl -n rabbit2 stop_app rabbitmqctl -n rabbit2 reset rabbitmqctl -n rabbit2 join_cluster --ram rabbit1@`hostname  -s`//--ram indicates that this is a memory node
rabbitmqctl -n rabbit2 start_app

rabbitmqctl -n rabbit3 stop_app
rabbitmqctl -n rabbit3 reset
rabbitmqctl -n rabbit3 join_cluster --disc rabbit1@`hostname -s` //--disc indicates a disk node (also a disk node by default)
rabbitmqctl -n rabbit3 start_app
Copy the code
  1. When successful, execute the commandrabbitmqctl cluster_statusQuery noderabbit1As you can see below, two disk nodes and one memory node:

  1. Note that the cluster started here is only the default normal cluster. To configure it as a mirror cluster, run the following command:
rabbitmqctl -n rabbit1 set_policy ha-all "^" '{"ha-mode":"all"}'
Copy the code

At this point the RabbitMQ cluster is complete, but note that the.erlang. Cookie file is not considered to be consistent because it is a stand-alone version.

High availability cluster based on HAProxy + Keepalived

If there are multiple memory nodes in a RabbitMQ cluster, which one should we connect to? If the choice of strategies on the client to do, so there will be a great disadvantages, the most serious is to modify the client code every time extension cluster, so this way is not very desirable, so we need to when I was in the deployment of the cluster an intermediate broker component, the component must be able to achieve service monitoring and forwarding, For example, the Sentinel cluster pattern in Redis allows sentinels to listen to Redis nodes and failover.

In RabbitMQ cluster, Keepalived and HAProxy implement high availability and load balancing.

HAProxy

HAProxy is an open source, high-performance load balancing software, can also be used as load balancing software nGINx, LVS and so on. HAproxy supports layer 7 and layer 4 load balancing.

Load balancing

The so-called 7-layer load balancing and 4-layer load balancing are targeted at the OSI model, which is an OSI communication model as shown in the following figure:

As you can see in the figure above, layer 7 corresponds to the application layer and layer 4 corresponds to the transport layer. Common load balancing software such as NGINx works at Layer 7 and LVS (Linux Virtual Server) works at layer 4.

  • 4Layer load:

Layer 4 loads use Network Address Translation (NAT). After receiving a client request, you can modify the source IP address and port in the packet and forward the packet to the target server. Layer-4 load balancers can forward requests only based on the destination IP address and source IP address, but cannot determine or modify the type of the requested resource.

  • 7Layer load:

Based on the resource path requested by the client, it is forwarded to different target servers.

High availability HAProxy

HAProxy implements load balancing, but if only one HAProxy is deployed, it also has the risk of downtime. Once HAProxy goes down, the whole cluster will become unusable, so we need to implement clustering for HAProxy, so if HAProxy also implements clustering, which service should the client connect to? The problem seems to be back to square one, stuck in an infinite loop…

Keepalived

In order to achieve high availability of HAProxy, we need to introduce a Keepalived component, Keepalived component has the following features:

  • The load function monitors the status of nodes in the cluster. If a node in the cluster fails, failover can be implemented.
  • It can implement clustering itself, but only onemasterNode.
  • masterThe node will provide a virtual to the outside worldIPThe application only needs to connect to this oneIPWill do. You can think of it as a clusterHAProxyNodes vie for the virtual at the same timeIP, which node grabs, by which node to provide services.

VRRP protocol

VRRP is the Virtual Router Redundancy Protocol. The virtual IP mechanism provided by Keepalived is VRRP. It is a fault-tolerant protocol for routers to avoid single points of failure.

conclusion

This article introduces the RaabbitMQ cluster knowledge, and compares the difference between normal cluster and mirror cluster, finally through the practice to build a RabbitMQ cluster, and also introduces the common cluster has some disadvantages. You can combine HAProxy and Keepalived components to achieve a true highly available distributed cluster service.