When using message middleware, there is a very important problem, how to ensure that each message sent by the sender, the receiver must be able to receive and consume successfully, will not consume less, will not consume again?
As shown in the figure, for the whole process to be secure and reliable, the message delivery process must ensure that MQ receives the message, MQ is persisted to ensure that the message is not lost, the message consumption process needs to ensure that the message is successfully consumed, before confirmation, and to prevent repeated consumption
Ensure that messages are sent to RabbitMQ correctly
Setting a channel to Confirm mode assigns a unique ID to all messages published on the channel. Once a message has been posted to the destination queue or written to disk (persistable messages), the channel sends an acknowledgement to the producer (containing the unique ID of the message). If RabbitMQ has an internal error that causes the message to be lost, an NACK (not acknowledged) message will be sent.
The sender confirmation pattern is asynchronous, and producer applications can continue sending messages while waiting for confirmation. When the acknowledgement message arrives at the producer application, the producer application’s callback method is triggered to process the acknowledgement message
Ensure that the message recipient consumes the message
Each message received by the consumer must be manually acknowledged after it has been consumed (receiving and confirming messages are two different operations) and then RabbitMQ removes the message from the queue
Speaking of timeouts, RabbitMQ only confirms the need to resend the message if the consumer connection is down. That is, RabbitMQ gives the consumer enough time to process the message as long as the connection is not broken. Ensure final consistency of data;
The special cases are listed below: If a consumer receives a message and disconnects or unsubscribes before confirmation, RabbitMQ will assume that the message has not been distributed and redistribute it to the next consumer that subscribes. If a consumer receives a message without confirmation and the connection is not broken, RabbitMQ considers the consumer to be busy and will not send more messages to the consumer.
Avoid duplicate message delivery
During message production, MQ internally generates an inner-msG-ID for each message sent by the producer as a basis for de-duplication (message delivery failure and retransmission) to avoid duplicate messages entering the queue.
Avoid repeated message consumption
Method 1: When consuming messages, it is required that there must be a unique Id (globally unique for the same business, such as payment Id, order Id, post Id, etc.) in the message body as the basis for deduplication to avoid repeated consumption of the same message
Method two: consume message interface to do idempotent processing
Ensure that messages are not lost in Mq
RabbitMQ ensures that persistent energy can be recovered from a server restart by writing it to a persistent log file on disk. When a persistent message is posted to the durable exchange, Rabbit does not send a response until the message is submitted to the log file.
Once a consumer consumes a persistent message from the persistent queue, RabbitMQ marks it in the persistence log as waiting for garbage collection. If RabbitMQ restarts persistent messages before they are consumed, Rabbit automatically reconstructs the exchange and queues (and bindings) and republishes the messages from the persistence log file to the appropriate queues
Reliability of message delivery
The reliability of message delivery depends mainly on the sender
The mainstream approach is
- Before sending a message, persist the message to the local (such as a database) to prevent the message from being sent again if the message fails to be sent
- Set the sender confirmation mode
- After the message is sent, the status of the message is stored in the database and set to be sent
- The sender sets the signature timeout and listens to RabbitMQ to see if the consumer has a signature
- If a sign-in is listened on, change the status of the message to successful delivery, otherwise wait until timeout
If the delivery is successful, the process is complete, but if the delivery times out, some compensation mechanism is required
- Start a scheduled task, every once in a while, from the database to find the status of the sending and time out of the message, redeliver
- If repeated attempts fail, manual action is required
