background

RocketMQ source version 4.0.0

Download the RocketMQ source code and start the Broker service and NameServer service normally in the local IDE.

The demo of messages sent by producers is as follows

An error was reported while sending the message

According to the error message, the routing information corresponding to **Topic (TopicTest) is not correctly obtained. ** It is no wonder that the Topic queue was not created after the Broker was started, so it is not possible to obtain routing information when sending messages.

The Broker has a mechanism for automatically creating topics by adding the following configuration to the Broker’s configuration file:

autoCreateTopicEnable = true
Copy the code

The problem is that even after this parameter is configured, messages are still sent with the above error after the Broker service is restarted, so WE want to take a closer look at how RocketMQ automatically creates topics.

Automatic Topic creation mechanism

Two problems need to be solved first

  • W1: Broker registers routing information
  • W2: indicates the policy for producers to obtain routing information

Routing Information Broker

  • The Broker node initializes the route information of the current Broker. The autoCreateTopicEnable parameter determines whether to add the route information of the default topic
  • Broker nodes periodically register routing information with NameServer

This section describes the routing information

  • List of topics that currently exist at the Broker node
  • Queue allocation per Topic (number of read and write queues)

1. When the Broker node is started, it starts a scheduled task to register routing information with NameServer periodically

2. Invoke the registration method. The topicConfigWrapper object contains information about the route to be registered

3. The topicConfigWrapper object is a wrapper around the topicConfigTable object

4. The topicConfigTable object type is ConcurrentMap, the Map Key is the Topic name, and the Map Value is the routing information (including the number of read and write queues) corresponding to the Topic.

Initialize topicConfigTable (topicConfigTable) {topicConfigTable (topicConfigTable);

  • Determine whether the Broker has turned on the automatic creation of topics (autoCreateTopicEnable = true)
  • If automatic Topic creation is enabled, a routing information named TBW102 will be added to topicConfigTable. This will be called TBW102.
  • Therefore, the route information registered with NameServer contains TBW102 default route information.

Routing information Capture (producer)

  • When a producer sends a message, it queries routing information based on the Topic specified in the current message. If the local cache does not find routing information, it attempts to query routing information from the NameServer service
  • If no route information about the specified Topic is found, the system uses the name of the default Topic to query the route again. If the route information about the default Topic is found, a message is normally sent

1. Obtain the routing information of the Topic to which the message is sent

2. Obtain routing information

TBW102 (same as the route name registered by the Broker). If isDefault = true, try to get the route name for the Topic TBW102.

Creating a Routing Message

After receiving a message, the Broker checks to see if the Topic for the message exists. If the Topic corresponding to the message does not exist, and the Broker allows the automatic creation of a Topic that does not exist, the Topic will be created automatically.

1. Check whether the routing information about the Topic exists. If no, the system automatically creates a Topic

2. For the operation of automatic Topic creation, check whether the route information contains the route of [default Topic] to determine whether the Broker has enabled automatic Topic creation

The whole process

I have summarized the process of creating a Topic automatically as “switching gears”, which is accomplished by the cooperation of the Broker, NameServer, and producer

  • When a production sends a message, if the specified Topic does not exist, NameServer returns a route to the default Topic, allowing the producer to send the message normally

  • When the Broker receives a message and finds that the Topic for the message does not exist and allows the automatic creation of a Topic, it creates a Topic for the message and synchronizes routing information to the NameServer periodically

  • The producer also periodically synchronizes the latest routing information from NameServer and caches it locally

  • When the producer sends a message, the routing information of the corresponding Topic can be queried from the local cache

Problem solving

Understanding how brokers automatically create topics quickly solved my problem.

Why is Topic routing information unavailable when sending messages, even though the Broker is configured to automatically create topics that do not exist? Here’s why

  • RocketMQ version 4.3.0 was introduced in the demo for sending messages
  • Locally started Broker service, RocketMQ version 4.0.0
  • The naming of the default theme is inconsistent between the two versions, so even though the Broker registers the route information for the default theme with NameServer, the producer does not get the route information

AUTO_CREATE_TOPIC_KEY for version 4.3.0

4.0.0 [default theme] is named TBW102

Since the Broker service was started from native source, I changed the default theme names in the source code to make them the same and restarted the Broker service

The producer successfully sends the message. The problem is resolved.