Message queues

Message Queue Middleware (MQ) refers to the use of efficient and reliable messaging mechanism for platform-independent data communication, which can extend the data communication between processes in a distributed environment and integrate distributed systems based on data communication. It is mainly applicable to the following scenarios:

  • Project decoupling: Different projects or modules can use messaging middleware to transfer data, thus ensuring the relative independence of modules and achieving decoupling.
  • Traffic peaking: The ability to write burst traffic, such as split-kill data, to the messaging middleware and have it processed asynchronously by multiple consumers.
  • Elastic scaling: You can increase the processing power and throughput of your system by scaling horizontally across the messaging middleware.
  • Publish subscribe: Can be used in any publish subscribe mode.
  • Asynchronous processing: When we do not need immediate processing of data or do not care about the results of data processing, we can use middleware for asynchronous processing.
  • Redundant storage: Message-oriented middleware can persist data until you consume it and then delete it.

Ii. AMQP Agreement

The Advanced Message Queuing Protocol (AMQP) is an application-layer communication Protocol that provides unified messaging services and provides unified development specifications for messaging middleware. Different clients can post messages to or retrieve messages from the middleware; The clients that send and receive messages can be developed in different languages and implemented with different technologies, but must follow the same AMQP protocol. The AMQP protocol itself includes the following three layers:

  • Module Layer: located at the top of the protocol, it defines commands that are invoked by clients to implement their own business logic. For example, you can Declare a Queue using the queue. Declare command or Consume messages in a Queue using the basic. Consume subscription.
  • Session Layer: located in the middle Layer, it sends commands from the client to the server and replies from the server to the client. It provides a reliability synchronization mechanism and error handling for communication between the client and the server.
  • Transport Layer: Located at the bottom Layer, it mainly transmits binary data streams and provides frame processing, channel multiplexing, error detection and data representation.

3. Introduction to RabbitMQ

RabbitMQ fully implements the AMQP protocol and is based on the same model architecture. RabbitMQ extends the implementation of AMQP 0-9-1 and supports AMQP 1.0 through plug-ins. So RabbitMQ is, in a way, an implementation of AMQP in Erlang. RabbitMQ is currently the most widely used messaging middleware based on a number of excellent features, including the following:

  • Support for multiple messaging protocols, including all versions of STOMP and MQTT 3.1 through plug-ins in addition to AMQP;
  • There are a variety of switch types, which can meet most of the use requirements;
  • Supports multiple deployment modes and is easy to deploy.
  • Support cross-language development, such as Java,.NET, PHP, Python, JavaScript, Ruby, Go;
  • High availability and throughput can be achieved through clustering, and the Federation plugin can be used to connect different versions of service nodes across rooms and regions.
  • Pluggable authentication and authorization, TLS and LDAP support;
  • Supports continuous integration and can be flexibly extended with various plug-ins;
  • Ability to monitor and manage using a variety of methods, such as HTTP apis, command-line tools, and UI interfaces.

4. Model architecture

RabbitMQ follows the same model architecture as AMQP, which is illustrated below:

1. [k? : n]

The publisher (or producer) is responsible for producing the message and delivering it to the specified exchange.

2. Message

A message consists of a header and a body. The message header is used to store metadata associated with the message, such as the target exchange name, RountingKey, and other optional configuration information. The message body is the data that actually needs to be passed.

3. Exchange

The exchange is responsible for receiving messages from producers and routing them to one or more queues. If not, it is either returned to the producer or discarded, depending on the Mandatory attribute of the exchange:

  • When mandatory is true: If the exchange cannot find a queue that matches the criteria based on its type and routing key, the message is returned to the producer.
  • When mandatory is false: If the exchange cannot find a matching queue based on its type and routing key, the message is discarded.

4. BindingKey

The switch and queue are bound by a BindingKey.

5. Routingkey

When a producer sends a message to a switch, it usually specifies a RountingKey, which specifies the routing rule for the message. When RountingKey matches BindingKey based on the switch type, the message is routed to the corresponding queue.

6. Queue

Used to store routed messages. Multiple consumers can subscribe to the same message queue, in which case the queue distributes the received messages to all consumers in a round-robin manner. That is, each message is sent to only one consumer, and the same message will not be consumed by multiple consumers.

A Consumer is a Consumer.

Consumers subscribe to queues of interest and are responsible for consuming messages stored in queues. To ensure that messages can reliably reach consumers from queues, RabbitMQ provides message acknowledgement mechanisms, controlled by the autoAck parameter:

  • When autoAck is true: Messages are considered successful after being sent (written to the TCP socket), regardless of whether the consumer actually consumed the messages. When a TCP connection or channel is closed by accident, or when a consumer unexpectedly goes down during consumption, the corresponding message is lost. Therefore, this mode can improve throughput, but there is a risk of data loss.
  • AutoAck false: Manual confirmation is required after data processing is complete. RabbitMQ considers the message successfully processed only after manual confirmation. This ensures reliable delivery of data, but reduces system throughput.

8. Connection

The TCP connection used to deliver messages.

9. Channel

RabbitMQ uses a niO-like (non-blocking IO) design to duplicate TCP connections over channels, ensuring that each Channel is isolated as if it had a separate Connection. When the data traffic is not very large, the connection multiplexing technology can avoid creating too many TCP connections resulting in expensive performance overhead.

A Virtual Host is a Virtual Host.

RabbitMQ uses a virtual host for logical grouping and resource isolation. A virtual host is a small RabbitMQ server with its own queues, switches and bindings. Users can set up different virtual hosts according to different service scenarios. Virtual hosts are completely independent from each other. You cannot bind the switch on Vhost1 to the queue on Vhost2, which greatly ensures the isolation and data security between services. The default virtual host name is /.

11. Broker

A real deployed running RabbitMQ service.

Type of switch

RabbitMQ supports many types of switches. The following types are commonly used:

5.1 the fanout

This is the simplest exchange model in which messages are routed to all queues bound to the exchange. As shown below, any message sent to the X switch is routed to both Q1 and Q2 queues.

5.2 direct

Route messages to queues where the BindingKey and RountingKey are identical. As shown in the following figure, when the RountingKey of the message is orange, the message will be routed to Q1 queue. When the RountingKey of a message is black or green, the message is routed to Q2 queue.

It is important to note that when a switch is bound to multiple queues, their Bindingkeys can be the same, as shown in the following figure. When the RountingKey of the message is black, the message is routed to both Q1 and Q2 queues.

5.3 the topic

The message is routed to the queue that matches the BindingKey and RountingKey. The matching rules are as follows:

  • RountingKey and BindingKey use commas for multiple words.To connect;
  • BindingKey supports two special symbols:# 和 *. Among them*To match a word,#Matches zero or more words.

The following is an example from the official documentation. The binding of the switch to the queue is shown here, and the routing is as follows:

  • The key route forlazy.orange.elephantIs sent to all queues;
  • The key route forquick.orange.foxIs sent only to the Q1 queue;
  • The key route forlazy.brown.foxIs sent only to the Q2 queue;
  • The key route forlazy.pink.rabbitIs sent only to the Q2 queue;
  • The key route forquick.brown.foxThe message does not match any binding.
  • The key route fororange 或 quick.orange.male.rabbitThe message does not match any binding.

5.4 headers

A set of key-value pairs can be specified as bindingkeys when a switch is bound to a queue. You can specify a set of key-value pair attributes in the HEADERS that send a message, and when those attributes match the BindingKey, the message is routed to the queue. You can also specify the matching mode using the x-match argument:

  • X-match = all: The match is successful only when all key/value pairs are the same.
  • X-match = any: The match is successful if only one key/value pair is the same.

Headers switches have poor performance, so they are rarely used in real development.

6. Dead-letter queues

Another common concept in RabbitMQ is dead-letter queues. When a message becomes dead message in a queue, it can be re-sent to a dead-letter Exchange (DLX). Any queue bound to a dead-letter Exchange is called a dead-letter queue. It is important to note that dead letter exchanges and queues are created in exactly the same way as normal exchanges and queues, and their names express their functionality rather than their type. A normal message becomes a dead letter for three reasons:

  • The message is rejected (basic. Reject/ basic. Nack) and requeue is set to false.
  • Message expiration;
  • The queue length reaches the maximum. Procedure

We can add a dead letter exchange to a normal queue by setting the X-dead-letter-exchange parameter in the channel.queueDeclare method of queue creation. When a dead letter exists in the queue, it will be sent to the dead letter exchange and then routed to the dead letter queue. The following is an example:

// Create a dead letter switch
channel.exchangeDeclare("exchange.dlx"."direct");
// Declare a dead-letter queue
channel.queueDeclare(" queue.d1x ".true.false.false.null);
// Bind a dead letter exchange and a dead letter queue
channel.queueBind("queue.dlx "."exchange.dlx "."routingkey");

Map<String, Object> args = new HashMap<>();
args.put("x-dead-letter-exchange"."exchange.dlx");
// Specify a dead letter switch for a normal queue named myQueue
channel.queueDeclare("queue.normal".false.false.false, args);
Copy the code

In addition, you can re-specify the routing key for the dead letter. If no routing key is specified, the original routing key is used by default. You can reset the routing key as follows:

args.put("x-dead-letter-routing-key", "some-routing-key");
Copy the code

The resources

  1. Zhu Zhonghua. RabbitMQ Combat Guide. Publishing House of Electronics Industry. 2017-11-1
  2. RabbitMQ Tutorials, Documentation: Table of Contents

For more articles, please visit the full stack Engineer manual at GitHub.Github.com/heibaiying/…