What is a message queue
Message Queuing is an asynchronous way of communication between services. It is an important component of distributed system. It mainly solves the problems of application coupling, asynchronous message, traffic peak-cutting and so on, and realizes a technology of high performance, high availability, scalability and final consistency architecture.
Performance comparison of existing message queues
| features | AcitveMQ | RabbitMQ | Kafka | RocketMq | NSQ |
|---|---|---|---|---|---|
| Development of language | Java | erlang | scala | java | golang |
| Single throughput | All level | All level | The class of 100000 | The class of 100000 | All level |
| timeliness | Ms level | Us level | Ms level | Ms level | – |
| availability | High (master and slave) | High (master and slave) | Very high (distributed) | Very high (distributed) | Very high (distributed) |
| API completeness | high | high | high | high | low |
| Multilingual support | Support, Java preferred | Language has nothing to do | Support, Java preferred | Support, Java preferred | support |
| Provide a quick start | There are | There are | There are | There are | There is no |
| features | Mature products that are used in many companies; More documentation; Various protocols are well supported | The development is based on Erlang, so the concurrency ability is very strong, the performance is very good, the latency is very low; The management interface teaches richness | MQ has complete functions and good expansibility | Only major MQ functions are supported, such as some message query, message backtracking and other functions are not provided. After all, they are prepared for big data and widely used in the field of big data | Use Golang development, based on distributed, high performance, but the current documentation is not perfect, deployment is more troublesome |
A few questions about message queues:
1. How to ensure that the message queue is highly available?
- In most cases, we use clustering to ensure high availability, just as we do with databases;
- Taking RabbitMQ as an example, it usually uses two cluster modes, the default mode and the mirror mode, among which the mirror mode is the most commonly used. The mirror mode is shown in the figure below.
- The cluster of rcoketMQ has multi-master mode, multi-master and multi-slave asynchronous replication mode, and multi-master and multi-slaver synchronous double write mode, among which the deployment architecture of multi-master and multi-slaver mode is as follows
2. How to ensure that messages are not consumed repeatedly?
RabbitMQ, for example, does not guarantee that messages will not be repeated. If the business needs to guarantee that messages will not be repeated strictly, it can be achieved by the following methods:
- Ensure that each message has a unique number and that successful message processing occurs at the same time as the log of the de-duplication table;
- Take the message and do the database INSERT operation. Make the message a unique primary key, and avoid the problem of duplicate consumption due to the primary key conflict.
3. How to guarantee the reliable transmission of messages?
The reliability transmission of message queue should be analyzed from three perspectives: the producer loses the data, the message queue loses the data, and the consumer loses the data. The following is an example of RabbitMQ to illustrate:
- Producer loss: RabbitMQ provides Transaction and Confirm modes to ensure that producers do not lose messages. Transaction mechanism means that the transaction (Channel.txSelect ()) is opened before the message is sent, and then the message is sent, and if there is an exception during the sending process, The transaction is rolled back(Channel.TxRollback ()) and committed (Channel.TxCommit ()) if successful, but the downside is that throughput is reduced, so in general production environments use confirm mode more often. Once a channel is in confirm mode, all messages posted on that channel are assigned a unique ID. Once a message has been delivered to all matching queues, RabbitMQ sends an AVK to the producer. This lets the producer know that the message has arrived on its destination queue correctly. If RabbitMQ fails to process the message, a NACK message is sent, at which point the retry operation can be performed.
- Missing message queue data: Dealing with missing data message queue, is generally open persistent disk configuration, the persistence configuration can be used together and confirm mechanism, we can in the message after persistent disk, again to send an Ack signal producer, so if the message persistent disk before the RabbitMQ is dead, so producers don’t get Ack signal, Producers reissue, and persistence is mainly divided into two steps: 1. If the durable representation of a queue is set to true, it will be durable. 2. Set DeliveryMode =2 when sending messages. This will enable RabbitMQ to recover data even if it has been suspended.
- Consumer data loss: Consumer data loss is usually caused by the automatic acknowledgement message mode. In this mode, the consumer automatically acknowledges the receipt of the message, and then RabbitMQ will delete the message immediately. The solution is to manually confirm the message.
AMQP (Advanced Message Queuing Protocol; Advanced Message Queuing Protocol
AMQP Core Concepts
- Message Broker: An application that receives and distributes messages. RabbitMQ Server is Message Broker.
- Virtual host: Virtual address used for logical isolation, top-level message routing. A Virtual Host can have several exchanges and queues, and the same Virtual Host cannot have exchanges or queues of the same name.
- Connection: TCP Connection between Publisher/consumer and Broker. Disconnection is made only on the client side. Broker does not disconnect unless there is a network failure or a problem with the Broker service.
- Channel: A channel is a channel that can read and write messages. If a Connection is created for each visit to RabbitMQ, the overhead of establishing a TCP Connection will be huge and the efficiency will be low when there is a large number of messages. A channel is a logical Connection established within a Connection. If the application supports multiple threads, usually each thread creates a separate channel to communicate with. AMQP method includes a channel ID to help clients and Message Broker identify channels, so channels are completely isolated from each other. Channel, as a lightweight Connection, greatly reduces the overhead of establishing TCP Connection for the operating system;
- An Exchange: message arrives at the first stop of the broker and dispatches messages to a queue with a routing key matching the query table, according to the dispatching rules. The usual types are: Direct (Point-to-Point), Topic (Publish-Subscribe)
- Queue: where messages are finally sent to be picked up by consumers;
- Binding: A virtual link between an exchange and a queue. A Binding can contain a routing key. The Binding information is stored in a query table in an exchange and is used to dispatch messages.
- Routing key: A Routing rule that the virtual machine can use to determine how to route a particular message;
- Message: Message, the data sent between the server and the application, consists of Properties and Body, Properties can modify the Message, such as the priority of the Message, delay and other advanced characteristics; The Body is the Body of the message.