The interviewer asked me about a customer, and I almost passed out

Small knowledge, big challenge! This paper is participating in theEssentials for programmers”Creative activities

preface

Did you all read the producer’s last article? Read it before you read it

– How did I not know RocketMQ producer has so many uses? (Picture at the end, thanks)

In the last article we said producers, since there are producers must have consumers, and the same reason with raising chickens, chickens laid eggs you must have someone to eat it

You want to refute me to say I can not eat ah, I can hatch chicks ah!!

That I also have to refute you two words, first of all, brooding chicks also belong to consumption ah, but as eggs to eat is a different way of consumption, because, at last, the egg is gone!

Also, if the eggs of the whole world are hatching chicks, that is not a disaster! ?

Well, back to the subject. Keep studying. That’s what I love

Well, that’s right, this is the dividing line, serious face replace ing

RocketMQ version of message queue is a distributed message middleware with low latency, high concurrency, high availability and high reliability built by Ali Cloud based on Apache RocketMQ. Message queue RocketMQ not only provides asynchronous decoupling and peak-filling capabilities for distributed applications, but also features massive message stacking, high throughput, and reliable retry for Internet applications.

Some of the features are listed below

** Message query: ** Message queue RocketMQ provides three ways to query messages by Message ID, Message Key, and Topic
** Query message track: ** Through message track, it can clearly locate the complete link that the message is sent from the producer and delivered to the message consumer through the RocketMQ version server of message queue, which is convenient to locate and troubleshoot problems
** Cluster consumption and broadcast consumption: ** When using the cluster consumption mode, RocketMQ version of message queue assumes that any message needs to be processed by any consumer in the consumer cluster; When using broadcast consumption mode, message queue RocketMQ pushes each message to all registered consumers in the consumer cluster, ensuring that the message is consumed at least once on each machine
Reset consumption point: Resets consumption progress based on time or site, allowing users to backtrack or discard accumulated messages
** Dead-letter queue: ** Stores messages that cannot be consumed normally in a special dead-letter queue for subsequent processing
** Global message routing: ** Used for message synchronization between different regions of the world to ensure data consistency between regions

Below, you will follow the fish fish to surf 🏄🏄🏄

Consumers Consumer

Consumers Consumer, as the name suggests, is responsible for the consumption of the message, it must also and business is closely linked, because involves the message processing logic, we said a producer on the client, through the analysis of some problems to go deeper into the producers, this paper, the big fish also want by analyzing some common problems to give you, The key points and difficulties are understood, the rest is easy to understand, it is easy to connect the various points of knowledge

Message filtering and message lookup

A Consumer, as the name implies, is responsible for consuming messages, which must be relevant to the business because of the logic involved in processing messages

Message filtering is to filter the messages produced by producers in the way they want

Wrong, fish fish, explain to me, since all do not consume, at the beginning why hair this message ah, you producer do not produce do not send this message not line, also can reduce network transmission

Good, the guest officer, meet your needs, a consumer may be spending more than one Topic, but I don’t want to, all the Topic news, I just want to spend part of the filter will come in handy, of course, also can put these messages in the same Topic to transport a, but this is not waste resources?

That’s true, but I have money. I have servers and resources

There’s no more to talk about. Rich and capricious

If you know such a person, please introduce me. Here is my wechat, thank you ~

Message filtering is divided into Tag filtering and SQL filtering. Tag filtering should be familiar to everyone, which is the secondary Topic mentioned in the previous article. Topic must be set for each message

Conf file, add enablePropertyFilter=true, and then start the Broker to make the configuration file take effect

What about message lookup? Isn’t it true that the broker pushes the consumer directly? So why do we need to look for messages? There are two modes of getting messages: push and pull, and we’ll talk about these two modes, but at the bottom, they’re all done through pull, and push is actually a pull effect of consumer polling

When a consumer pulls a message from a broker, it is a search process, using the message’s consumption offset to look for messages in the Broker, where messages are stored in commitLog files. In addition, there is an IndexFile that supports brokers and messages queried by MsgID or MessageKey. With ID queries, it is easy to find commitLog files to read messages because the IDS are generated from the broker and offset. MessageKey is used to query messages, and MessageStore builds an index to speed up reading

Message queue load and redistribution mechanism

One of the things you can’t do with a sheep is pull up the wool

In the clustered consumption model, each message needs to be delivered to only one instance of the Consumer Group that subscribes to the topic. RocketMQ pulls and consumes messages in an active pull, specifying which message Queue to pull from.

Every time the number of instances changes, load balancing is triggered for all instances, and queues are allocated equally to each instance based on the number of queues and the number of instances.

Consumption pattern and pull pattern

Consumption patterns

For consumers, consumption patterns can be divided into two categories

Cluster consumption: When using the cluster consumption mode, message queue RocketMQ version assumes that any message needs to be processed by any consumer in the cluster.
Broadcast consumption: When using the broadcast consumption mode, message queue RocketMQ version pushes each message to all registered consumers in the cluster, ensuring that the message is consumed by each consumer at least once.

Cluster consumption pattern

Application scenario: This mode applies to the clustered deployment on the consumer end where each message needs to be processed only once.

In addition, because the consumption schedule is maintained on the server side, reliability is higher. Specific consumption examples are shown in the figure below.

Matters needing attention:

1. In the cluster consumption mode, each message is distributed to only one machine for processing. If you want to be processed by every machine in the cluster, use broadcast mode.

2. In the cluster consumption mode, it is not guaranteed that each failed message will be routed to the same machine.

Broadcast consumption pattern

Application scenario: This mode applies to the cluster deployment on the consumer end where each message is processed by each consumer in the cluster

Specific consumption examples are shown in the figure below.

Matters needing attention

1. Broadcast consumption mode does not support sequential messages or reset consumption sites.

2. In broadcast mode, the RocketMQ version of message queue guarantees that each message is consumed at least once by each client, but it does not re-cast the message of consumption failure. Therefore, the business side needs to pay attention to the situation of consumption failure.

3. In broadcast mode, each restart of the client consumes the latest news. Note Messages sent by the client to the server are automatically skipped when the client is stopped. Exercise caution when performing this operation.

4. In broadcast mode, each message is repeatedly processed by a large number of clients. Therefore, cluster mode is recommended as much as possible.

Pull mode

Message queue RocketMQ supports the following two ways to get messages, which is easy to understand: push and pull, like bungee jumping, one by oneself and one by others:

Push: Messages are pushed to Consumer by the RocketMQ version of the message queue. In Push mode, RocketMQ version of message queue also supports bulk consumption function, which can uniformly Push bulk messages to Consumer for consumption.
Pull: The message is actively pulled by the Consumer from the RocketMQ version of the message queue.

For any kind of messaging middleware, there are generally two ways for a consumer client to fetch and consume messages from the messaging middleware.

Technically, RocketMQ does not implement the push pattern, but rather wraps around the pull pattern, which, despite the name starting with push, is implemented in pull mode. Pull continuously polls the Broker for messages. When no new message exists, the Broker suspends the request until a new message is generated, cancels the suspension, and returns a new message. In this way, it is almost as real-time as the Broker actively pushes (with a corresponding real-time loss, of course). The principle is similar to long-polling.

Push model

Messages are actively pushed to consumers by messaging middleware (MQ message server agents); With Push, messages can be sent to consumers for consumption in as much real time as possible.

However, when the ability of the consumer to process messages is weak (for example, the process of processing a message by the business system on the consumer side is complex and there are many invocation links leading to a long consumption time). This can be summed up as a “slow consumption problem”), and MQ keeps pushing messages to consumers, where the buffer on the consumer side can overflow, causing exceptions

The pull model

The consumer client actively pulls messages from the messaging middleware (MQ message server agent); In Pull mode, how to set the frequency of Pull messages needs to be considered.

For example, it might be possible for 1,000 messages to come in a minute and then no new messages to come in 2 hours (summed up as “message delay and busy wait”).

If each Pull interval is long, the message delay will increase, that is, the time for messages to reach consumers will be longer, and the amount of messages in MQ will be larger. If each Pull interval is short and there are no messages to consume in MQ for a period of time, there will be a lot of RPC overhead for invalid Pull requests, which will affect overall MQ network performance

Other problems with consumption

Consumption starting point

Consumption starting point, why do we have this?

If the computer room network is not good or due to physical factors caused by the consumer suddenly down, after the restart will need to find a new consumption starting point to consume, so, there should be a consumption starting point mark

But each time let the consumer continue to consume from the location of the last outage is not good? Why have this, in a business scenario to consider, if the news is not very important, consumers have downtime for half a month to restart, there may be a lot of messages and half a month’s bandwidth consumption is very high, when in fact you don’t need to put all previous messages are spending again again, so you can adjust the starting point of consumption to the recent situation

Of course, this is only for situations where the message is not very important. In the financial industry, where every message is very important, you need to consider multiple situations and set up multiple safeguards to ensure that the message is not lost

Consumption trajectory: Customized consumption trajectory

Message trajectory refers to the complete link information gathered by data such as time and status of each relevant node in the whole process of sending a message from the producer to the RocketMQ server of message queue and then to consumer consumption processing. This trajectory can be used as a powerful data support for troubleshooting problems in production environments.

Consumption trajectory data

Message queue In RocketMQ version, a complete link of a message contains three roles: producer, server and consumer. Each role will add relevant information to the track link during message processing, and the current status of any message can be obtained by gathering these information.

Usage scenarios

If the Message sending and receiving in the production environment does not meet expectations, you can query the Message trace based on the Message ID, Message Key, or Topic time range to find the actual Message sending and receiving status and diagnose problems.

Order consumption

Sequential messages (FIFO messages) are strictly sequential published and consumed messages provided by RocketMQ version of the message queue. Sequential publishing and sequential consumption means that producers publish messages in a certain order for a given Topic; Consumers subscribe to messages in a predetermined order, meaning that the first published messages are received by the client first.

Sequential messages are divided into global ordering and partial ordering, which are controlled by the producer

Oh, right, my last article is about producers, and there is a problem of sequential consumption, so I won’t be verbose here

Although the order part is not a problem on the consumer’s side, the producer sending can cause a problem of sending messages repeatedly

Repeat purchases

In fact, sequential consumption, the main control logic or in the producer

Repeated consumption, the root cause of the problem is the producer sent repeated messages, so let’s first think about whether this problem can be avoided

If we just send messages, regardless of the consumption of the message, the result, etc., then messages to the broker will not be repeated, but in practice we do not allow this situation, so the message is completely unreliable, we should be able to guarantee at least once

For example, after a message is sent to the broker, it needs to wait for the broker to respond. It is possible that the broker has written, but no response has been received due to network reasons, and then the producer sends the message again, which is repeated

In this case, consumers may get duplicate messages, so many businesses cannot avoid this problem, so we need to change the Angle of solution

Message idempotency

Sequential messages (FIFO messages) are strictly sequential published and consumed messages provided by RocketMQ version of the message queue.

Idempotent is a mathematical concept, and it’s easy to understand, but it means that when you call the same API multiple times with the same parameters, the result is the same as when you call it just once

In fact, there are a lot of idempotent examples in life, such as we see some websites like the system, the “like” can not be cancelled, nor can the second “like”, this is an example of idempotent; There is also a common SQL statement programmers, a very common update field statement, execute the same SQL multiple times, but the result is the same, this is idempotent

Therefore, the business processing logic needs to be modified to ensure idempotency without affecting the final result

As for how to ensure idempotency, it depends on the business scenario, in fact, a lot of times the scenario idempotency can be guaranteed by the database unique key, right

conclusion

The consumer and producer client objects for the RocketMQ version of the message queue are thread-safe and can be shared across multiple threads. We can deploy multiple producer and consumer instances on a server (or multiple servers), and we can use multiple threads to send or receive messages within the same producer or consumer instance to improve message sending or receiving TPS.

The consumer is one of the RocketMQ clients, and it is one of the most common problems we develop

1. Message filtering and message lookup (Tag filtering and SQL filtering)

2. Message queue load and redistribution mechanism (load balancing)

3. Consumption patterns (clustering and broadcasting) and pull patterns (push and pull)

4. Consumption starting point, consumption trajectory, sequential consumption, repeated consumption, message idempotency

O praise

Well, that’s all. I’m the captain, your learning partner

I hope that one day I can support myself by writing, and I am still honing my skills. This time may be many years. Thank you for being my original reader and communicator. Please believe that as long as you give me a love, I will eventually return you a page of love.

Thank you again for reading here, I will continue to update the technical articles and some records of the soul of life articles, if you feel good, think [Captain] something, please like, follow, share three

Oh, right! The subsequent update of the article I will be put here in time, welcome to click to watch, are dry articles ah, suggest collection, at any time to check

Github.com/DayuMM2021/…

The interviewer asked me about a customer, and I almost passed out

Cluster consumption pattern

Broadcast consumption pattern

Related Posts

MyBatis integration SpringBoot

After reading this article, stop saying that you have learned MySQL

Set three brothers. List,Set,Map. Get the hang of everything!