What is a message queue
Message queue is a kind of asynchronous communication between services, is an important component in distributed system, mainly to solve the application coupling, asynchronous message, traffic peak cutting and other problems, to achieve high performance, high availability, scalable and ultimately consistent architecture technology.
Performance comparison of existing message queues
| features | AcitveMQ | RabbitMQ | Kafka | RocketMq | NSQ |
|---|---|---|---|---|---|
| Development of language | Java | erlang | scala | java | golang |
| Single machine throughput | All level | All level | The class of 100000 | The class of 100000 | All level |
| timeliness | Ms level | Us level | Ms level | Ms level | – |
| availability | High (master slave) | High (master slave) | Very high (distributed) | Very high (distributed) | Very high (distributed) |
| API completeness | high | high | high | high | low |
| Multilingual support | Java is preferred | Language has nothing to do | Java is preferred | Java is preferred | support |
| Provide a quick start | There are | There are | There are | There are | There is no |
| features | Mature products, used in many companies; There are more documents; Good support for various protocols | Based on Erlang development, so the concurrency is very strong, the performance is very good, the delay is very low; Management interface teaching rich | MQ function is relatively complete, good scalability | Only the main MQ functions are supported, such as some message query, message backtracking, etc., are not provided, after all, they are intended for big data and are widely used in the field of big data | Using Golang development, based on distributed, high performance, but the current documentation is not perfect, deployment is more troublesome |
A few questions about message queues:
1. How to ensure that message queues are highly available?
- For the most part, we use clusters to ensure high availability, just like databases;
- Take RabbitMQ as an example. There are two common cluster modes: the default mode and the mirroring mode. The mirroring mode is the most common.
- RcoketMQ clusters have multi-master mode, multi-master, multi-slave asynchronous replication mode, and multi-Master, multi-Slaver synchronous dual-write mode. The deployment architecture of the multi-master, multi-Slaver mode is as follows
2. How to ensure that messages are not reused?
RabbitMQ uses RabbitMQ as an example. RabbitMQ does not ensure that messages do not repeat. To ensure that messages do not repeat, perform the following operations:
- Ensure that each message has a unique number and that the message processing success is accompanied by the de-table log;
- Get the message to do the database insert operation, give the message a unique primary key, because the primary key conflict can avoid the problem of double consumption.
3. How to ensure the reliable transmission of messages?
The reliability of message queue transmission should be analyzed from three perspectives: producer data loss, message queue data loss, and consumer data loss. RabbitMQ is used as an example.
- Producers lose data: RabbitMQ provides transaction and confirm modes to ensure that producers do not lose messages. A transaction (channel.txselect ()) is initiated before a message is sent, and the message is sent. The transaction will be rolled back(channel.txrollback ()) and commit (channel.txcommit ()) if the transaction is successfully sent. However, the downside is that throughput is reduced. Therefore, confirm mode is often used in production environments. Once a channel enters Confirm mode, all messages posted on the channel will be assigned a unique ID. Once the message has been sent to all matching queues, RabbitMQ will send an Avk to the producer, which lets the producer know that the message has reached the destination queue correctly. If RabbitMQ fails to process the message, a Nack message will be sent, at which point the retry operation can be performed.
- Message queue lost data: Dealing with missing data message queue, is generally open persistent disk configuration, the persistence configuration can be used together and confirm mechanism, we can in the message after persistent disk, again to send an Ack signal producer, so if the message persistent disk before the RabbitMQ is dead, so producers don’t get Ack signal, The producer resends, and there are two main steps to persistence: 1. If the durable queue is declared as true, it is a durable queue. 2. Set deliveryMode to 2 when sending messages so that RabbitMQ can be rebooted even if it hangs.
- Consumer data loss: Consumer data loss is usually caused by automatic confirmation messages. In this mode, consumers automatically acknowledge receipt of the message and RabbitMQ will delete the message immediately. Manually confirm the message.
AMQP (Advanced Message Queuing Protocol; Advanced Message Queue Protocol)
AMQP core concepts
- Broker: An application that receives and distributes messages. RabbitMQ Server is the Message Broker.
- Virtual host: Virtual address used for logical isolation, the top layer of message routing. A Virtual Host can have several exchanges and queues. A Virtual Host cannot have exchanges or queues with the same name.
- Connection: TCP Connection between Publisher/consumer and Broker. Disconnected operations only occur on the client side. The Broker does not disconnect unless there is a network failure or a problem with the Broker service.
- Channel: A channel is a channel for reading and writing messages. If a Connection is established for each RabbitMQ access, it is expensive and inefficient to establish a TCP Connection with a large number of messages. A channel is a logical Connection established within a Connection. If the application supports multithreading, it is common for each thread to create a separate channel for communication. The AMQP Method contains a channel ID to help the client and Message Broker identify the channel, so channels are completely isolated from each other. As a lightweight Connection, Channel greatly reduces the overhead of establishing TCP Connection in the operating system.
- Exchange: The first destination for a message to reach the broker. Routing keys in the query table are matched according to the distribution rules, and messages are sent to the queue. Direct (point-to-point), topic(publish-subscribe)
- Queue: The Queue where messages are finally sent to be picked up by consumers;
- Binding: Virtual connection between exchanges and queues. Binding can contain routing keys. Binding information is stored in the exchange query table and used for message distribution.
- Routing key: A Routing rule that a virtual machine can use to determine how to route a particular message;
- Message: A Message, which is the data transmitted between the server and the application. It consists of Properties and body. Properties can modify messages with advanced features such as Message priority and latency. The Body is the message Body content.