Make writing a habit together! This is the second day of my participation in the “Gold Digging Day New Plan · April More text challenge”. Click here for more details.

1 overview

Message Queue is an important component in distributed system. Its common usage scenarios can be simply described as follows:

When you don’t need immediate results, but the amount of concurrency needs to be controlled, that’s pretty much the time to use message queues.

Message queue mainly solves the problems of application coupling, asynchronous processing and traffic cutting.

The most popular message queues are RabbitMQ, RocketMQ, ActiveMQ, Kafka, ZeroMQ, MetaMq, etc. Some databases such as Redis and Mysql can also realize the function of message queues.

2 Application Scenarios

  • Asynchronous processing
  • The application of decoupling
  • Traffic peak clipping
  • Message communication
  • Log processing
  • Transaction message

2.1 Asynchronous Processing

Message queues are introduced to process non-core business logic asynchronously to improve response speed.

2.2 Application decoupling

A typical e-commerce system involves multiple core processes, from commodity display to ordering, payment, delivery, logistics, after-sales service, etc. In the microservice architecture, these business processes are usually decoupled into multiple application services. The communication between services can be either HTTP or RPC protocol interface invocation, as follows:

It can also be done by a message queue, as follows:

The order system realizes decoupling with the inventory system through message queue. After the order system generates an order, there is no need to reduce the inventory of the inventory system.

2.3 Flow peak clipping

For scenarios where instantaneous traffic surges may occur, such as seckill and scheduled traffic, message queues can be used to achieve smooth transition.

The request is directly thrown into the message queue to quickly respond to the request. Subsequent processing is handled asynchronously by the second kill business processing service.

2.4 Message Communication

Point-to-point communication:

To publish/subscribe to newsletters, such as social media, to people who follow:

2.5 Log Processing

Log processing refers to the use of message queues in log processing, such as Kafka, to solve the problem of large log transfer.

2.6 Transaction Messages

Message transaction is a two-phase commit based on message queue. In essence, it is a special use of message queue. It puts the local transaction and message sending in a distributed transaction to ensure that either the local operation succeeds and the message sending succeeds, or both fail.

Based on message queue two-phase commit is often used in high concurrency scenario, A distributed transaction into A message that the transaction (A system of local operating + send messages) + B local operation of the system, and the B system driven by news operation, as long as the message transaction is successful, then the operation must be successful, A message will be send to. In this case, B will receive the message to perform the local operation. If the local operation fails, the message will be resold until THE operation succeeds. In this way, the distributed transaction between A and B is realized in A disguised way. The principle is as follows:

2.6.1 RocketMQ transaction messages

RocketMQ already supports distributed transaction messages in version 4.3.0. RocketMQ uses the 2PC approach to commit transaction messages and adds a compensation logic to handle two-phase timeout or failure messages. The flow is shown below:

Its specific work flow is divided intoSend and submit normal transaction messagesandCompensation process for abnormal transaction messages.

The normal process
  1. Start one on the message queueTransaction Topic.
  2. The first service to execute in the transaction sends one"Half Msg"(The only difference between a semi-message and a normal message is that the message is not visible to the consumer until the transaction commits) to the message queue.
  3. After the half-message is successfully sent, the service that sends the half-message starts to execute the local transaction and decides whether to commit or roll back the transaction message based on the local transaction execution result.
  4. A successful local transaction makes this “half-message” becomeNormal messageFor subsequent steps in distributed transactions to execute their own local transactions. In the case of transaction messages, the Producer does not roll back if the Consumer fails to consume themHigh availabilityandFinal consistencyIf message consumption fails, RocketMQ takes care of itselfThe push messagesUntil consumption succeeds.)
The compensation process

RocketMQ provides transaction backchecks to resolve exceptions. If RocketMQ does not receive a commit or rollback request, the Broker periodically checks the status of the local transaction with the producer. Based on the status of the producer’s local transaction, RocketMQ determines whether the “half-message” is committed or rolled back. It is worth noting that we need to implement the backcheck logic interface based on our business logic, and then decide whether to commit or rollback based on the return value Broker.

3 Preventing message loss

In many scenarios, message loss is a serious problem, such as financial and e-commerce scenarios, and message loss prevention needs to be solved from three ends.

3.1 Preventing loss at the production end

  • Use transaction messages
  • Enable the comfirm mechanism

3.2 Anti-Loss Of the MQ server

  • Enable Persistence
  • Synchronous brush set

3.3 Anti-loss at the consumer end

  • Cancel automatic ACK
  • Retry mechanism

4 Message idempotent

Because of network reasons, the producer may send messages repeatedly, so the consumer must do idempotent processing of the message.

Idempotent: It means that the results of one or more requests for the same operation are consistent without side effects caused by multiple clicks.Copy the code

In MQ usage scenarios, it is possible to record whether a message has been processed by generating globally unique business unique identifiers.

5 Message Accumulation

Message accumulation mainly refers to the problem of insufficient client consumption capacity due to long consumption time or low consumption concurrency during local consumption.Copy the code

General solutions:

  1. Verify that the consumption time of messages is reasonable
  2. View the stack information of the client and check the service code
  3. For some special service scenarios, if the message accumulation has affected the service running and the message accumulation can be skipped, you can reset the consumption point to skip the message accumulation and start consuming from the latest point to quickly recover the service.
  4. Sudden increase in traffic, you can increase the consumption end service instances
  5. Adjust the concurrency of consumption appropriately

6 Orderly message

Ensure that sequential messages are placed in the same queue and that only one consumer consumes the queue.