Since Kafka does not support delayed messaging and is currently used in the messaging middleware of the company’s technology stack, the business wants to use RocketMQ for the delayed messaging scenario, but if only the delayed messaging capability is needed to introduce an additional set of messaging middleware, it will increase the operation and maintenance costs. In this context, we want to extend the Kafka client to provide support for delayed messages.
This article introduces the principles of the four delayed message implementations and analyzes their advantages and disadvantages.
Scheme 1: time round algorithm
Each producer holds a queue of time-round delay messages, which are stored in memory.
- Slot = (current timestamp – time round start time) % total number of slots;
- Round = (current timestamp – Time round startup time)/total number of slots;
- Delayed message list: sorted by round.
Defect analysis:
- This scenario is applicable only to scenarios with a short delay (for example, within one minute) because memory resources are consumed heavily.
- Easy to lose messages.
Scheme 2: Single round time round algorithm + file storage (improved version of Scheme 1)
Using the single-round time round algorithm, the number of slots in a single round satisfies Max delay = slot count, and each slot points to a file.
Defect analysis:
- If the service is deployed in a container, the time round file will be lost after the service is rebuilt, and the message consistency cannot be guaranteed.
Solution 3: Multi-level partition + automatic degradation
Divide multiple partitions by level and demote messages to a specified partition based on remaining latency (expected send time – current time) until demoted to a real topic.
Defect analysis:
- Multiple sent-subscriptions are required. If you follow the hierarchy in the figure, a message with a delay of 2 hours will go through at least 8 subscriptions and 9 sends before it is finally sent to the target topic.
- Due to the different latency of each message within the same level, ensuring accurate message latency can result in a message needing to go through not only 8 subscriptions and 9 sends before it is finally sent to the target topic, but many times over.
For example, if the delay value of several messages in the queue is 50, 40, 36, and 56 respectively in the order of 30 to 60 minutes, each previous message must be consumed and then written back to the partition of the same level in order not to affect subsequent delayed messages. So, in the worst case, a long-delayed message can take hundreds of sent-subscriptions to complete.
Scenario 4: Multilevel delay, no support for delayed messages of any time precision (improved version of Scenario 3)
RocketMQ supports delayed message design. It does not support delayed messages of any time precision, but only supports specific levels of delay messages. The message delay levels are divided into 1s, 5s, 10s, 30s, 1m, 2m, 3m, 4M, 5m, 6m, 7m, 8m, 9m, 10m, 20m, 30m, 1h, 2h. There are 18 levels in total. Only one delay topic is created with 18 partitions. Each partition corresponds to a different delay level.
For example:
- When a 5-second delay message is sent, the message is sent to the second partition of order-topic.delay;
- When a 1-minute delay message is sent, the message is sent to the fifth partition of order-topic.delay;
- When sending an hour delay message, send the message to the 17th partition of order-topic.delay;
Advantages:
- To ensure that the messages in each partition are in chronological order, we only need to consume each partition sequentially and forward the messages that have reached the send time to the real topic.
- If the message does not reach the send time, then the offset does not need to be submitted, because any message after the offset on the same partition must also be unsent.
We finally adopted solution 4. In implementation, we started a KafkaConsumer for each process and subscribed to topics ending in ‘.delay’ using regular expressions to reduce thread consumption. When the message is sent to the delayed topic, the delay level is used as the message key, and the original message key is stored in the message header. When the message is sent to the actual topic, the real key and the real topic are obtained from the message header of the delayed message.