MQ message queues (Application scenario, Product comparison)

What is Message Queue (MQ)

MessageQueue is a very important technology widely used in Internet projects. MessageQueue is usually used to solve problems such as traffic peak clipping, service decoupling, message communication, and final message consistency under high concurrent pressure.

When is MQ used?

MQ can be used in multiple domains, including asynchronous communication decoupling, enterprise solutions, financial payments, telecommunications, e-commerce, express logistics, advertising and marketing, social, instant messaging, mobile games, video, Internet of Things, Internet of Vehicles, and more.

In terms of application function. Such as:

• Log monitoring, as a communication channel for monitoring important logs, minimizes the impact on system performance.
• Message push, for social applications and Internet of things applications to provide point-to-point push, one-to-many broadcast push capabilities.
• Financial message, send financial message, realize financial quasi – real-time message transmission, reliable and safe.
• Telecom signaling, encapsulates telecom signaling into messages and transmits them to each control terminal to achieve quasi-real-time control and information transmission.

From the perspective of function:

• MQ is used in four scenarios: asynchronous processing, application decoupling, traffic clipping and message communication

2.1 Asynchronous Processing

Scenario Description: After registration, users need to send registration emails and SMS messages. There are two traditional approaches:

1. The normal serial mode

• Registration information is written to the database
• Send registration email
• Sending registration SMS

It takes 150ms in total

2. Parallel mode (enabling asynchronous threads)

• Registration information is written to the database
• Register email + SMS

The total time is 100ms

Assuming that each of the three business nodes uses 50 ms, leaving aside other overhead such as the network, the serial time is 150 ms and the parallel time might be 100 ms. Because the CPU can process a certain number of requests per unit of time, assume that CPU throughput is 100 times per second. In serial mode, the number of requests that the CPU can process in 1 second is 7 (1000/150). The number of requests processed in parallel is 10 (1000/100)

Summary: As described in the case above, performance (concurrency, throughput, response time) of the traditional approach can be bottlenecking. How to solve this problem?

A: The introduction of message queues will not be necessary for business logic, asynchronous processing, after the transformation of the architecture is as follows:

According to the convention above, the user’s response time is equivalent to the time it takes for the registration information to be written to the database, which is 50 milliseconds. After registering an email, sending a SHORT message and writing it to the message queue, the message queue is directly returned. Therefore, the writing speed of the message queue is very fast and can be ignored. Therefore, the response time of the user may be 50 milliseconds. As a result, the throughput of the system increased to 20 QPS per second after the architecture change. Three times better than serial and two times better than parallel. In addition, mainstream message queues support clustering, which can also solve the problem of slow processing.

Of course, not only registering the scene, but also uploading files and parsing Excel, you can also consider importing queues and setting up a transport center.

2.2 Application decoupling

Scenario Description: After a user places an order, the order system notifies the inventory system. Traditionally, the order system calls the inventory system interface. The diagram below:

Disadvantages of the traditional mode: if the inventory system cannot be accessed, the order reduction inventory will fail, resulting in order failure, order system and inventory system coupling

How to solve the above problems? The scheme after the introduction of application message queue is shown as follows:

Order system: after the user places an order, the order system completes the persistent processing, writes the message to the message queue, and returns the user to place the order successfully

Inventory system: subscribe to the order message, using the pull/push way to obtain the order information, inventory system according to the order information, inventory operation

What if: inventory system is not working properly when placing an order. It also does not affect the normal order, because after the order is written to the message queue, the order system does not care about other subsequent operations. Realize the order system and inventory system application decoupling

2.3 Flow peak clipping

Traffic clipping is also a common scenario in message queues, and is commonly used in split-kill or snatch activities.

Application scenario: The application system may not be able to carry the transient traffic due to heavy traffic. As a result, the application system may hang up directly, which is the legendary “down” phenomenon. To solve this problem, we would trap that huge flow of traffic in the upper layers of the system by moving it to MQ instead of flooding our interfaces directly, where MQ acts as a buffer.

2.4 Log Processing

Log processing refers to the use of message queues in log processing, such as Kafka, to solve the problem of large log transfer. The architecture is simplified as follows

• Log collection client, responsible for log data collection, periodically write kafka queue

• Kafka message queue, which receives, stores, and forwards log data

• Log processing application: Subscribe and consume Kafka data

• Common log collection architectures are:

• ELK/EFK

Iii. MQ product comparison

The mainstream MQ currently includes:

• RabbitMQ
• RocketMQ
• kafka
• Activemq

3.1 the MQ contrast

3.2 Advantages and disadvantages of mainstream MQ

For small and medium sized software companies, RabbitMQ is recommended, for one thing, the Erlang language is inherently highly concurrent and its management interface is very easy to use. Is the so-called, into also xiao, defeat also xiao! The downside is that, although RabbitMQ is open source, how many programmers in the country can customise Erlang? Fortunately, RabbitMQ has a vibrant community that can resolve bugs encountered during development, which is important for small and medium sized companies. Rocketmq and Kafka are not considered because, on the one hand, small and medium sized software companies are not as large as Internet companies, and the amount of data is not as large, so the choice of messaging middleware should be more comprehensive, so Kafka is excluded. Consider Alibaba’s RocketMQ.

Large software companies, depending on their use, choose between rocketMq and Kafka. On the one hand, large software companies have enough money to build distributed environments and enough data. For rocketMQ, large software companies can also devote staff to developing rocketMQ custom, after all, there are still a lot of people in the country capable of changing the JAVA source code. As for Kafka, depending on the business scenario, if there is log collection, kafka is definitely the first choice. The specific choice depends on the usage scenario.

When we choose the middleware, we usually choose the product through the following points:

1) Performance 2) Function support 3) development language (whether any member of the team is familiar with the development language of this middleware and whether it is easy to recruit developers of this language in the market) 4) How many companies have actually used it in the production environment, What is the effect of use? 5) Community support? 6) Is the learning degree of middleware simple and detailed? 7) Stability? 8) Is the cluster function complete

If I choose a message queue based on the above eight points, as a programmer familiar with Java, I would not hesitate to choose RocketMQ when faced with the re-selection of message queues. Besides the unfriendly monitoring and management functions, rocketMQ is a really excellent message queue middleware. And if your project is written in Java, you can customize it with RocketMQ for troubleshooting and optimizations.