1. Why message queues?
(1) Decoupling: Can decouple between multiple systems, will originally between through the network for message of asynchronous invocation way instead of using the MQ communication, as long as the operation is not need to be synchronized, can be changed to use MQ for the connection between the different systems, this project does not exist between the coupling, between systems will not have too big impact, even if a system, Messages are simply squeezed into MQ without being consumed, with no impact on other systems.
(2) Asynchronous: add an operation design to several steps, which do not need to be completed synchronously. For example, when the customer creates an order, they need to add a track to the customer track system, update the inventory in the destocking system, and modify the status of the customer in the customer system. So if the system is called directly, it will generate a lot of time, which is unacceptable to the customer; In addition, operations like adding customer track do not need to be synchronized. If MQ is used to create orders for customers, updates of track, inventory, status and other information will be put into MQ and then asynchronously operated, which can speed up the access speed of the system and provide better customer experience.
(3) Peak clipping: a system has a peak and a low peak of access flow. For example, there is a shopping spree at noon. Such as system flow rate is not high, only a second more than 100 concurrent requests and system processes without any pressure, all is calm, at a certain activity time, snapping up concurrent access to the system, such as the 5000 concurrent requests per second, and our system can only handle 2000 requests per second, so because the traffic is too big, Our systems, our databases, might crash. At this time if you use the MQ for flow peak clipping, the user of a large number of messages directly into MQ, then our system according to its own biggest consumption ability to these messages, you can guarantee the stability of the system, just may have to follow up the business logic, to give the user returns to a particular page or through other way to inform the result later.
2. What are the advantages and disadvantages of message queues?
Advantages: 1. Decouple the operations with complex structure and multiple design systems, reduce the operation complexity of the system and reduce the maintenance cost of the system. 2. Asynchronize some system operations that can be operated asynchronously to reduce the response time of the operation and provide better user experience. 3, can be high flow peak cutting, to ensure the smooth operation of the system. Disadvantages: 1. Reduced system availability. For example, introducing MQ into the system, what if MQ fails? In general, the more external dependencies are introduced, the more vulnerable the system is, and the failure of each dependency will lead to the collapse of the entire system. 2. Improved system complexity. There are various scenarios for MQ to consider, such as repeated consumption of messages, message loss, guaranteed consumption order, and so on…… 3. Data consistency problem. For example, system A has returned the operation success to the customer, but the operation of BC is successful, but the operation of D fails, resulting in data inconsistency.
3. What are the advantages and disadvantages of Kafka, ActivemQ, RabbitMQ and RocketMQ?
| features | ActiveMQ | RabbitMQ | RocketMQ | kafka |
|---|---|---|---|---|
| Single machine throughput | Ten thousand levels, throughput is an order of magnitude lower than RocketMQ and Kafka | Ten thousand levels, throughput is an order of magnitude lower than RocketMQ and Kafka | At level 100,000, RocketMQ is also an MQ that can support high throughput | At 100,000 level, Kafka’s biggest advantage is its high throughput. It usually works with big data systems to perform real-time data calculation and log collection scenarios. |
| The impact of number of topics on throughput | Topics can reach hundreds or thousands of levels, with a small drop in throughput. This is one of RocketMQ’s great strengths, being able to support a large number of topics on the same number of machines | Throughput drops dramatically as you go from dozens to hundreds of topics. So kafka tries not to have too many topics for the same number of machines. More machines are needed to support large scale topics | ||
| timeliness | Ms level | Microsecond, which is a feature of RabbitMQ, with the lowest latency | Ms level | The delay is within ms class |
| availability | High, based on master-slave architecture for availability | High, based on master-slave architecture for availability | Very high, distributed architecture | Very high. Kafka is distributed, with multiple copies of a single piece of data, and a few machines down, without data loss and unavailability |
| Message reliability | There is a low probability of data loss | After optimized parameter configuration, 0 can be lost | After parameter configuration, message loss can be zero | |
| Function support | The functionality of the MQ realm is extremely complete | Based on Erlang development, so the concurrent performance is very strong, excellent performance, low delay | MQ function is relatively complete, distributed scalability is good | The function is relatively simple, mainly support single MQ function |
| advantage | It is very mature and powerful, and has been used in many companies and projects in the industry | Erlang language development, excellent performance, low latency, throughput of ten thousand levels, MQ function is complete, management interface is very good, active community; Internet companies use it more | The interface is simple and easy to use, Ali production is guaranteed, the throughput is large, the distributed expansion is convenient, the community is more active, support large-scale topic, support complex business scenarios, can be customized based on the source code development | Ultra high throughput, MS class delay, high availability and reliability, easy distributed expansion |
| disadvantage | Occasionally, there is a low probability of message loss, and the community activity is not high | Throughput is low, Erlang voice development is not easy to customize, and cluster dynamic expansion is troublesome | Interfaces do not follow the standard JMS specification, and there are systems migrations that require significant code changes, and the technology risks being abandoned | It is possible to re-consume messages |
| application | It is mainly used for decoupling and asynchronism, and is rarely used in large-scale throughput scenarios | Have to use | It is used for large-scale throughput and complex business | It is widely used in real-time computing and log collection of big data and is the industry standard |
To sum up, the summary is as follows: To introduce MQ into general business system, ActiveMQ was used at the beginning, but now it is not used much. It has not been verified in a mass throughput scenario and the community is not active, so it is not recommended to use it again. Later rabbitMQ was used, but it was developed in The Erlang language, and if you don’t know Erlang, it’s almost out of your control for the company. It’s open source, has a very active community, and has fairly stable support. More and more companies are using RocketMQ, but be careful of the risk of abandonment. If the company has the strength to maintain the development, recommended use. Otherwise, use RabbitMQ. Kafka is the industry standard for real-time computing and log collection of big data.
Therefore, for small and medium sized companies with moderate technical capabilities, rabbitMQ is recommended, and for large companies with strong infrastructure development capabilities, RocketMQ is recommended.
Next article, How to Make Message Queues highly available