Enter the message queue
Hello, everyone. I am the third brother, a programmer in an e-commerce company, responsible for the development of order system.
After losing a lot of hair, I finished the development of user order payment.
The order payment business goes like this. After the payment is completed, I need to update the order status, which is completed in this system. Next, I’m going to call the inventory system, reduce the inventory, well, the rest is the inventory system.
Development, joint tuning, testing, online, my life has become idle, every day is in the group to brag fart.
But two days later, the product sister came to her, she said, she wants to add a function, users after completing the order payment, to increase the user’s points.
It took me two days to do it. All I had to do was call the membership system.
One day, and sand carving group friends dou map, product sister came over, he said to access the message system, good, make!
Two days later, she said she was adding a marketing system.
Two days later, she said she was going to start a referral system, um… , come on!
Two more days…
So the system looks like this:
So I spent my days in the dark, maintaining interfacing with several systems and following them every time they released a new version.
I’m iterating and changing the docking code with several systems.
Monday, marketing systems;
Tuesday, inventory system;
…
This day, eye circles black I was tearing with downstream services, suddenly could not help but two legs war, she came, product sister (female) son (wang) came — she is a woman I can not refuse.
Fragile tears are streaming down my face, and my ape life is grey…
Unexpectedly, generation savior appeared, my good friend proud day came over, take nostril looking at me.
“You Loser, unexpectedly don’t know to use message queue, no wonder be bullied every day, hum!”
Why not use a message queue?
So I introduced message queues and refactored the system.
Well, I just update the order status, throw the rest to the message queue, and you downstream people go to the message queue to consume the message, don’t bother me.
…
It was another leisurely afternoon after the introduction of the message queue.
I didn’t do it in the group because I dropped out.
The other day, I was hurt like never before
I ridicule an old brother in the group, real technology, even the message queue can not!
The elder brother sent a picture of him and his girlfriend, “Single dog, good technology how, not even a girlfriend!”
I instant SAN value crazy!
“Single programmers, not single… Can you call yourself single when you’re dating someone new?” Some difficult remark, “there is no sister,” or “philosophy,” followed, and the crowd burst into laughter, and the air filled with gaiety.
So, this afternoon I stared at the empty demand sheet in a daze. Is there really no girl in the company? …
Well, that’s the end of the long prelude, and it’s time to move on to the body 😂.
Message queue analysis usage
In the introduction above, we have seen one of the most important uses of message queues:
- The decoupling
Message queues reduce coupling between systems and avoid excessive calls.
It’s like the executive sister of the company wants to announce something, she usually makes an announcement in the group, and then we deduct 1. One by one, she’d have to do it dozens of times.
- asynchronous
Again, the order payment we mentioned above, after payment, we have to deduct inventory, increase credits, send messages and so on, so that the link is long, the link is long, the response time is longer. By introducing message queues, everything else can be done asynchronously, except to update order status, so that it can respond to our upstream more quickly.
Why not do asynchrony in a multi-threaded way or something? –
Hey, with only multithreading for async, you’d have to write code to tune a bunch of downstream, so it comes back to decoupling.
- Peak clipping
Message queues can also be used for peak shaving.
To use an analogy, a river, suppose its downstream can hold a limited amount of water, in order to prevent the flood burst the dam, what should we do?
We can build a reservoir upstream, and when the flood crest comes, we can store the water first, and the sluice gate will release only as much water as can be held downstream.
In our system, the same thing. For example, if the traffic of a seckill system is usually very low, but you need to perform a seckill activity, the traffic will be crazy when the seckill activity is performed. Your server, Redis, and MySQL all have different capacity, so it must be a problem to directly accept all the traffic according to the order.
So again, we can queue requests and only send out as much traffic as our downstream service can handle, which can withstand heavy traffic for a short period of time.
In addition to these three purposes, the sequential nature of the queue itself can be used to meet the situation where messages must be delivered in order. Use queue + scheduled task to realize message delay consumption……
Message queue analysis nature
Old baby Ma teacher has three moves – pick up, change, hair.
There are three axes at the core of the message queue: send, save and consume.
The producer posts the message to a container called a “queue” and then pulls the message out of the container before forwarding it to the consumer [1].
The diagram above is the original model of the message queue, which contains one or two key messages and queue queues in the message queue:
-
Message: Data to be transferred, which can be as simple as a text string or in a complex custom format.
-
Queue: A first-in, first-out data structure that should be familiar to you. It is the container for storing messages, messages from the end of the team into the team, from the team head out of the team, the process of sending messages in the team, the process of receiving messages in the team.
So the essence of a message queue is to put the data to be transmitted in a queue.
Around this essence:
- The role that puts the data into the message queue is
producers - What pulls the data out of the queue is
consumers
Message queue Parsing model
[1] We have seen the nature of the message queue model above. As application scenarios change, the message queue model gradually evolves.
It’s like our text messaging, which started as a one-to-one message, then as a group message, and then as a group chat.
- Queuing models
The original message Queue is the original model described in the previous section, which is strictly a Queue. The messages are read in the order in which they are written. However, the queue does not have a “read” operation, which is to “delete” the message from the queue header.
If multiple producers send messages to the same queue, the messages that can be consumed in the queue are the sum of all the messages produced by these producers. The order of messages is the natural order in which these producers send messages.
If multiple consumers receive messages in the same queue, the relationship between these consumers is actually competitive, each consumer can only receive a part of the message in the queue, that is, any message can only be received by one of the consumers.
- Publish – subscribe model
If one piece of message data needs to be distributed to multiple consumers, and each consumer wants to receive the full message. Clearly, the queue model does not meet this requirement.
One possible solution is to create a separate queue for each consumer and let producers send multiple copies. This is stupid, and the same data can be duplicated multiple times, which is a waste of space.
To solve this problem, another messaging model has evolved: the publish-subscribe model.
In the publish-subscribe model, the sender of the message is called Publisher, the receiver of the message is called Subscriber, and the container where the message is stored on the server is called Topic. Publishers send messages to topics, and subscribers need to “subscribe to topics” before receiving messages. “Subscribe” here is both an action and a logical copy of the topic at consumption, with each subscription allowing the subscriber to receive all messages for the topic.
Compare this to the “queue pattern” : producers are publishers, queues are topics, and consumers are subscribers. The only difference is whether a single piece of message data can be consumed multiple times.
Message queue analysis cost
There is no such thing as a free lunch — the same goes for system design. The message queue is introduced, which solves some problems, but also increases the complexity and maintenance cost of the system. [5]
- High availability
Earlier, we introduced message queues, which reduced the coupling between systems. However, we also became more dependent on message queues. Think about it, if message queues were to fail, we would be completely cold downstream.
Therefore, our message queue deployment must be highly available, at least master/slave, and generally clustered/distributed.
- Message loss problem
We write data to the message queue, and the downstream service hangs before it can fetch the data from the message queue. If nothing is done, our data is lost.
What can be done?
The data in the message queue must be stored somewhere else to minimize data loss.
Again, where does it exist?
- Disk?
- Database?
- Redis?
- Distributed file system?
Synchronous or asynchronous storage?
- Problem of repeated consumption
What if our network is delayed or for some reason, resulting in downstream repeated consumption?
Are we solving this problem in message queues? Or is it serviced downstream?
There are other sequential messages and so on.
These problems need to be fully considered when selection.
Message Queue Analysis option
At present, there are four main message middleware in the market: Kafka, ActiveMQ, RabbitMQ, RocketMQ.
I compiled a picture of their comparison:
Message queue analysis ends
Ok, so here we have the basics of message queues.
Layman, have you entered this door? I was rolling around in it anyway.
“Do simple things repeatedly, do repetitive things carefully, and do serious things creatively!” –
I am three points evil, a full stack of literary and military development.
Like, pay attention to not get lost, let’s see you next time!
Reference:
[1]. The core foundations of the MQ Series are all here
[2]. “Jindachang Series” series – Message queue basis
[3]. Zhihu Q&A: How can message queue be explained in a simple way?
[4]. System learning message queue sharing (4) Message model: What is the difference between topic and queue?
[5]. What is a message queue?