In every age, there is no ill-treatment of those who can learn
Hello, I’m Yes.
Today we are going to talk about the push and pull mode of message queue, which is also a popular interview topic. For example, if you put RocketMQ on your resume, they will basically ask you: Does RocketMQ use push or pull mode? Is it pull mode? Isn’t there PushConsumer?
Today we are going to talk about the push and pull mode and look at RocketMQ and Kafka.
Push-pull mode
Let’s first clarify which step of the message queue is being discussed by the push and pull pattern. Generally speaking, when we talk about the push and pull pattern, we refer to the interaction between Comsumer and Broker.
The default is that the Producer pushes the message to the Broker, rather than the Broker pulling the message.
Imagine that if a Broker is required to pull a message, the Producer must log the message locally and wait for the Broker to pull it. If there are many producers, the reliability of the message depends not only on the Broker itself, but also on hundreds of producers.
Brokers can also rely on mechanisms such as multiple replicas to ensure that messages are stored reliably. Reliability of hundreds or thousands of producers is difficult, so the default Producer pushes messages to the Broker.
So some situations are better distributed, and some are better managed centrally.
Push mode
Push mode refers to messages being pushed from Broker to Consumer, where the Consumer passively receives messages and the Broker takes the lead in sending messages.
So let’s think about what’s the advantage of pushing patterns?
Messages are real-time and can be sent to consumers immediately after being received by the Broker.
Easier for consumers to use, simple ah wait, anyway there is a message will be pushed over.
What are the downsides of push?
The push rate is difficult to adapt to the consumption rate. The goal of the push mode is to push messages as fast as possible. When the rate at which producers send messages to the Broker is greater than the rate at which consumers consume messages, over time the consumer side may be “flooded” because there is no consumption. When the push rate is too fast like a DDos attack the consumer is stupid.
In addition, different consumers have different consumption rates. As a Broker, it is difficult to balance the push rate of each consumer. In order to achieve an adaptive push rate, consumers need to tell the Broker when pushing, “I can’t do it, please slow down.” The Broker then needs to maintain the state of each consumer to change the push rate.
This actually increases the complexity of the Broker itself.
Therefore, the push mode is difficult to control the push rate according to the state of consumers, and is suitable for the situation where the amount of messages is small and the consumption power is strong and the real-time performance is high.
Pull mode
The pull mode is when the Consumer actively requests a pull message from the Broker, which passively sends a message to the Consumer.
Let’s think about what’s good about pull mode?
Pull mode initiative on the consumer, consumers can initiate according to their own situation to pull the message request. Assuming that the current consumer feels that he can no longer consume, he can either stop pulling according to a certain strategy or stop pulling at intervals.
Pull mode Broker is relatively easy, it just producers from the news, as for the consumption of natural initiated by the consumer, to a request to it the message bai, where to begin with, with many consumers tell it, it is a tool without feelings, if consumers don’t take none of its business, too.
Pull mode can be more suitable for batch sending of messages, based on the push mode can come to a message push, can also cache some messages after push, but when push, actually do not know whether consumers can deal with so many messages at one time. The pull pattern is more reasonable and can use the information requested by consumers to determine how many messages to cache and then send in batches.
What are the disadvantages of pull mode?
Message latency, after all, is the consumer pulling the message, but how will the consumer know when the message arrives? So it can only pull and pull, but not too often. Too often it becomes the consumer attacking the Broker. So you need to reduce the frequency of requests, say every two seconds, and you’re looking at messages that are two seconds late.
Message busy request, for example, the message is several hours, then within a few hours of the request is invalid, doing nothing.
Is that a push or a pull
It can be seen that push mode and pull mode have advantages and disadvantages, how to choose?
RocketMQ and Kafka both use pull, and there are push based message queues such as ActiveMQ.
Personally, I think the pull mode is more suitable, because today’s message queues have the need to persist messages, that is to say, it has a storage function, its mission is to receive messages, save the good news so that consumers can consume messages.
As a Broker, you should not be inclined to rely on consumers. I have saved the good news for you. Come and take it.
Although the Broker is generally not a bottleneck because the consumer side is slow to consume business, it is still a central point and can be as light as possible.
If RocketMQ and Kafka both choose pull mode, are they not afraid of pull mode’s disadvantages? Afraid, so they operate a wave, mitigating the disadvantages of the pull mode.
Long polling
RocketMQ and Kafka both use “long polling” to implement pull patterns, so let’s take a look at how they work.
In order to simplify, I will describe the number of messages that do not satisfy this pull, the total size, etc., as there is no message, which does not satisfy the condition anyway.
Long polling in RocketMQ
PushConsumer in RocketMQ is actually a method in pull mode that looks like push mode.
Because RocketMQ is behind the scenes asking for data from the Broker.
The RebalanceService thread does load balancing based on the number of queues in the topic and the number of consumers in the current consumer group. The pullRequests generated by each queue are placed in the blocking pullRequestQueue. The PullMessageService thread then continuously gets the pullRequest from the blocking pullRequestQueue and requests the broker over the network, achieving a quasi-real-time pull message.
I’m not going to cut this part of the code, but that’s the thing, and I’ll show it in a picture.
The processRequest method in the Broker’s PullMessageProcessor is used to process the pull message request. If there is a message, it will return it. What if there is no message? Let’s look at the code.
Let’s take a look at what the PendpullRequest method does.
The PullRequestHoldService thread will get the PullRequest request from the pullRequestTable every 5 seconds to see if the offset of the PullRequest request is smaller than the maximum offset of the current consumption queue. If this is true, a new message will be sent. NotifyMessageArriving is called, and finally the PullMessageProcessor’s executeRequestWhenWakeup() method is called to try again to process the request for the message. The entire long poll takes 30 seconds by default.
Simply put, the message is checked once every five seconds, and if it is, processRequest is called to process it again. That’s not real time, is it? 5 seconds?
Wait, there is also a ReputMessageService thread that continuously parses data from commitLog and dispatches requests, builds ConsumeQueue and IndexFile, and also wakes up requests. To make up for this slow delay every 5s
I’m not going to cut the code, but the message is written and it’s going to call pullRequestHoldService#notifyMessageArriving.
And finally, let me draw a picture of what’s going on.
Long polling in Kafka
Things like Kafka have parameters in pull requests that can cause consumer requests to block waiting in “long polling.”
The consumer requests a message, returns it immediately if there is one, and waits until time out if there is none, and then initiates another request.
The Broker must also cooperate. If the consumer requests a message, it must return immediately, and if there is no message, it will set up a delay operation and return when the condition is met.
Let’s take a quick look at the source code, but I’m going to cut some code to make it more important.
Let’s start with the consumer-side code.
The poll interface is a poll interface that is waiting for data to arrive or timed out.
Let’s see what the final client.poll call is.
The final call is Kafka’s wrapped selector, and the final call is Java NIO’s Select (timeout).
Now that the code on the consumer side is clear, let’s look at what the Broker does.
The Broker’s entry point for handling all requests is actually what I described in a previous article, under the handle method in the KafkaApis. Scala file, and this time the main character is the handleFetchRequest.
This method comes in, and I take the most important part.
FetchMessages method internal implementation of the following picture, the source code to the annotations have been very clear, we can zoom in the picture to see.
This purgatory has an interesting name, but it simply uses the time wheel I mentioned in previous articles to perform timed tasks, such as the delayedFetchPurgatory, which deals with delayed fetching operations.
Simple let’s think about it, the delay operation needs to implement what method, the delay of the first building operation mechanism, need to check to see if the message has arrived, and then have to have a message to the execution method, after also need to be completed after the approach of what, of course, also have a timeout after what method.
These methods actually correspond to DelayedFetch in the code. This class inherits DelayedOperation and contains:
- IsCompleted A method to check whether a condition is met
- Method to execute after the tryComplete condition is met
- Method called after onComplete completes execution
- Method to execute after onExpiration
The time wheel drives the judgment, but you can’t wait to see if the message is out of date, can you?
Kafka, like RocketMQ, will alert you to the arrival of a delayed request when it is being written. I won’t post the code, but you can see it further in ReplicaManager#appendRecords.
But even though the code is not posted, I still have to draw it.
The subtotal
As you can see, RocketMQ and Kafka both use a “long polling” mechanism, in which the consumer waits for a message. When there is a message, the Broker returns the message directly. If there is no message, it adopts a delayed processing strategy. When a new message arrives in the corresponding queue or partition, the message will be notified and returned in time.
In short, the consumer and Broker work together to hold when a pull request does not meet the condition, avoiding multiple frequent pull actions and reminding the Broker to return as soon as the message arrives.
The last
In general, push-pull mode has its pros and cons, but I personally feel that pull mode is more suitable for message queues in general.
After reading this article, the interviewer asks you to push or pull. I suggest you give him a crooked grin.
I’m yes, from a little bit to a billion bits. See you next time.