Features of message queues

  • Durable: Queue persistence. If persistent, RabbitMQ will store the queue to local disk when it is shut down and will not be deleted when it is down or restarted. If set to non-persistent, the queue will be deleted when RabbitMQ is turned off.
  • Exclusive: exclusive queue. If exclusive is set, the current queue allows access only to pre-set Connections. If the setting is not exclusive, all connections are accessible.
  • AutoDelete: automatically deletes a queue. If automatic deletion is set, the queue will be deleted immediately when the consumer finishes consuming the messages in the queue. If the setting is not automatic deletion, the queue will continue to be retained when the consumer finishes consuming the messages in the queue.

Automatic message acknowledgement mechanism

If automatic message confirmation is enabled, the message is marked for deletion as soon as MQ sends it to the consumer. By default, when MQ sends messages to the corresponding Consumer, all messages belonging to that Consumer are sent to the corresponding channel at once. So if one of the consumers is consuming a long task and only part of the task is completed and then dies, we lose the messages being consumed and any messages sent to that consumer in the channel that have not yet been consumed.

As shown in the model above, if the distribution mode is circular, consumer-1 is assigned messages 1, 3 and 5 at one time, and consumer-2 is assigned messages 2, 4 and 6 at one time. If consumer-1 makes an error while consuming message 3 and goes down, both the consuming message 3 and the unconsumed message 5 in the channel are lost. If consumer-2 goes down while consuming message 2, the consuming message 2 and unconsumed messages 4 and 6 in the channel are all lost.

To ensure the integrity of business data, we need to change two points:

  1. After MQ sends the message to the consumer, the message is not marked for deletion.

  2. When a consumer is down, it can hand over consuming messages and unconsumed messages to other consumers to continue consuming.

The first point, if you want to achieve this, is to turn off automatic message confirmation. The purpose is to ensure that current messages are not lost when consumer consumption messages go down. With automatic message confirmation turned off, we need to manually acknowledge each message after it has been consumed normally so that it is marked as deleted and MQ sends the next message to the Consumer.

The second point, if implemented, is that instead of sending messages to the Consumer at once, as the default is, MQ needs to be set to send only one message at a time to the Consumer. The goal is to ensure that when a consumer consumes a message, there are no other messages in the corresponding channel, so that if the consumer goes down, messages that would have been assigned to that consumer can be allocated to other consumers for consumption by MQ.

A message model

RabbitMQ supports only five message publishing models prior to release 3.5: “Hello World!” Queues, Publish/Subscribe, Routing, Topics. Six distribution models are supported in version 3.7 (RPC) and seven distribution models are supported in version 3.9 (Publisher, Confirms).

RPC models and Publisher, Confirm models will not be described in this article. This article only describes five common message distribution models, which are sufficient to accommodate a variety of business scenarios.

Producers and one or more corresponding consumers must be connected to the same Virtual host, Channel, Exchange, and MQ (message queue) under the same user to communicate with each other.

1. “Hello World!” model

Point-to-point model, one producer, one message queue, and one consumer.

The producer sends messages directly to the message queue, and the consumer listens to the message queue, constantly retrieving messages from it and consuming them.

The point-to-point model can only handle some simple businesses. If there are too many messages and it takes time for consumers to process messages, the rate of message production will be much faster than the rate of message consumption. Over time, the information will pile up.

2. Work Queues model

Task model, one producer, one message queue, and multiple consumers.

Producers send messages directly to the message queue, and multiple consumers listen to the same message queue and consume messages in the message queue together.

Once a message in a queue is consumed by a consumer, it disappears, so the message is not re-consumed.

In the Work Queues model, MQ binds multiple consumers, and RabbitMQ distributes messages in a loop by default.

Loop: RabbitMQ will send each message to the next consumer in order, with the average consumer receiving the same number of messages. For example, A message numbered 1-10 has two consumers in MQ, so consumer A gets messages 1, 3, 5, 7, 9, and consumer B gets messages 2,4,6,8,10.

Since RabbitMQ distributes messages in a circular manner by default, messages can pile up and slow down the system if there are weak consumers. RabbitMQ also provides a more sophisticated way of distributing messages, with more messages for high spenders and fewer for low spenders, which requires additional Settings.

3. The Publish/Subscribe model

Broadcast model, one producer, one FANout switch, multiple message queues, and multiple consumers, with one switch bound to multiple queues and one message queue bound to one consumer.

The producer first sends the message to the switch, and the switch sends the message to all queues. After receiving the message, the queues send it to the corresponding bound consumers, so that a message is consumed by all consumers.

The first two models also need to be connected to a switch under the AMQP protocol and are not shown in the model diagram because they use the default switch in RabbitMQ.

4. The Routing model

Routing model, one producer, one direct switch, multiple message queues, and multiple consumers, with one switch bound to multiple queues and one message queue bound to one consumer.

Unlike the broadcast model, there is an arbitrary binding between the switch and the queue, that is, the switch is bound to all queues. In the routing model, the switch needs to bind a RoutingKey. A queue also needs to specify a RoutingKey. Instead of sending messages to all queues, the switch determines the RoutingKey of the queue and only sends messages to the queue for consumption if the RoutingKey of the queue matches the RoutingKey of the switch.

For example, in the above model, all MQ is bound to the “info” RoutingKey, and only the first MQ is also bound to the “error” RoutingKey. So when a producer sends a message with a RoutingKey of “INFO”, the exchange will send that message to all MQ; When a producer sends a message with a RoutingKey of “error”, the switch will only send the message to the first MQ.

Note: A RoutingKey generally consists of one or more words, separated by a “.” if multiple words are used.

5. Switchable viewer model

Dynamic routing model, one producer, one Topic switch, multiple message queues, and multiple consumers, with one switch bound to multiple queues and one message queue bound to one consumer.

In the routing model, if a queue needs to be bound to a single RoutingKey, then a separate RoutingKey needs to be set. If multiple Routingkeys need to be bound, then multiple different routingkeys need to be set separately, which can cause code redundancy and great inconvenience. Topics model provides dynamic routing matching rules that allow wildcards to be used in routingkeys based on the routing model so that a RoutingKey with a wildcard can match multiple specific routingkeys that meet the rule without having to be set multiple times, greatly simplifying the code.

RabbitMQ provides two wildcards:

  • * : Replace 1 word.
  • # : Replace zero or more words.

For example, in the above model, the first MQ binds to a RoutingKey for “*.rabbit.*” and the last MQ binds to a RoutingKey for “*.rabbit.#”. So when a producer sends a message with a RoutingKey of “user.rabbit.insert”, the exchange sends the message to the first MQ; When a producer sends a message with a RoutingKey of “user.rabbit.insert. All “, the exchange sends the message to the last MQ; When a producer sends a message with a RoutingKey of “user.rabbit”, the switch also sends the message to the last MQ.