In my last post, I looked at RocketMQ. In this article, I will talk about the production side of RocketMQ, how messages are sent.

The process of sending messages

Message sending from the RocketMQ client can be divided into three layers:

Business layer: business code that calls the MQ Client send API directly;

Message processing layer: After the RocketMQ Client obtains the message object sent by the service, it performs a series of operations such as checking parameters, preparing messages for sending, and packing parameters.

Communication layer: RocketMQ is based on an RPC communication service encapsulated by Netty. All RocketMQ components communicate with each other using this module.

** General process: **

  • When the Broker starts, information is registered with NameServer

  • When a client invokes Producer to send a message, it first obtains routing information for the topic from NameServer. The header code is GET_ROUTEINFO_BY_TOPIC

  • Routing information returned from NameServer, including the list of queues and brokers contained in the topic

  • The Producer side selects a queue based on the query policy to store messages

  • Each message generates a unique ID that is added to the attributes of the message. The key of the property is UNIQ_KEY

  • Special processing is performed on the message. For example, if the message exceeds 4M, the message will be compressed

  • Producer sends an RPC request to the Broker, saving the message to the Broker side. The code of the message header is SEND_MESSAGE or SEND_MESSAGE_V2 (the configuration file has special flags set)

The data structure of the message

Message

The physical carrier of information transmitted by a message system, the smallest unit that produces and consumes data, and each message must belong to a topic. Each Message in RocketMQ has a unique Message ID and can carry a Key with a business identity. The system supports Message query by Message ID and Key.

Core Field Configuration

Other Message configurations

Producer configuration

The field name The default value instructions
producerGroup DEFAULT_PRODUCER Producer Group name. If multiple producers belong to the same application and send the same message, they should be grouped into the same group
sendMsgTimeout 10000 Timeout for sending messages, in milliseconds
retryAnotherBrokerWhenNotStoreOK false If sending a message returns sendResult, but sendStatus! =SEND_OK, whether to retry sending
maxMessageSize 131072 The client limits the size of the message, and the server limits the size of the message
defaultTopicQueueNums 4 When sending messages, automatically create the number of queues created for topics that do not exist on the server

Message sending mode

Rocketmq provides three ways to send normal messages: synchronous, asynchronous, and one-way.

  • Synchronization: The sender sends a message and sends the next message after receiving the response from the server

  • Asynchrony: After sending a message, the server can continue sending the message or processing the subsequent task without waiting for the server to return. The sender receives the server response through the callback interface and processes the response result.

  • OneWay: the sender sends a message without waiting for a response from the server and no callback function is triggered. That is, only the request is sent without any response.

Comparison of sending modes: Send throughput, one-way > Asynchronous > Synchronous. However, the one-way transmission reliability is poor and messages may be lost, so the selection is determined according to actual requirements.

2. Message type

The message client provides a variety of SDKS: normal, sequential, transactional, and delayed messages

Producer Load balancing

When sending messages, the producer polls all queues by default, and the messages are sent to different queues. Queues can be distributed among different brokers.

[About batch delivery]

Sending messages in batches improves the performance of delivering small messages. At the same time, the following characteristics should be met:

  • Batch messages must have the same topic and message configuration

  • Delayed messages are not supported

  • It is recommended that a batch message should not exceed 1MB in size.

RocketMQ actually parses a batch message into a single message and then sends it, unifying the single-message, batch message logic at the bottom and making the overall framework design more robust.

Producer high availability

[Application Scenarios]

If you now have a cluster of three broker nodes with topic1, there are four queues created on each broker by default: Master – A (q0, Q1,q2, Q3), master- B (q0, Q1,q2, Q3), master- C (q0, Q1,q2, Q3), master- A (q0, Q1, Q2, Q3) If you continue sending topic1 messages, what if you avoid sending master-a again?

Rocketmq’s solution:

Send failure retry and Broker failure delay avoidance mechanisms. The retryTimesWhenSendFailed configuration item indicates the number of synchronization retries. The default value is two times and the normal sending time is one. There are three times in total. The selection of queues is controlled by sendLatencyFaultEnable. The default value is false. The broker failure delay mechanism is disabled and enabled when the value is true.

(1) Send failed and retry

RocketMQ supports both synchronous and asynchronous sending, and either method can be retried after a failure. If a single Broker fails, another Broker is selected to ensure that the message is sent properly.

Logic for retry failures:

(2) Broker avoidance mechanism

The RocketMQ Client maintains a broker-send delay relationship and selects a Broker with a lower level of send delay to maximize the Broker’s ability to eliminate brokers that are down, unavailable, or have a higher level of send delay. Ensure that messages are sent as normally as possible.

What if NameServer dies?

–

If Namesrv hangs, the new production consumption will not get the topic routing information, and MQExecption will be reported. If the producer consumer caches routing information for the producer’s cached Topic, if the NameServer all hangs, and the message can still be sent.