This is the 8th day of my participation in the August Text Challenge.More challenges in August

RocketMQ

Previously, message queues and protocols were introduced. From this chapter to formally enter the study of middleware. A few things you missed before you start learning — MQ disadvantages:

1. Reduced system availability: When the system introduces too many external dependencies, the stability of the system will deteriorate. If MQ goes down, it will affect online services, so make sure that MQ is highly available.

2. Increased system complexity: the service invocation between servers has changed from synchronous to asynchronous, and the data link is complex, which will bring many uncertain factors, which has been mentioned before;

3. Message consistency: after the addition of MQ, the systems are invisible to each other and can only “talk” through MQ. As for what to say, it is likely that because of this “obstruction”, the messages between systems will eventually be “wrong”;

background

RocketMQ timeline:

1. Ali’s message middleware originated from the Five-Colored Stone project in 2001, during which Notify message middleware was born (PS: a message engine developed by Taobao itself);

2. In 2010, B2B began to use ActiveMQ as message kernel on a large scale, but it did not meet the requirements of Ali business for middleware at this time: Message middleware that supports sequential messages and has massive message accumulation ability. MetaQ 1.0 was born in 2011;

3. With the continuous update and iteration of MetaQ, Ali abstracted RocketMQ, a general messaging engine, from it after version 3.0, and then opened it as an open source;

4. The baptism of war and fire in 2015 — “Double Eleven”; RocketMQ has a proven track record of reliability and stability; (PS: Questionable usability??) . With the trend of cloud computing, Alibaba launched Aliware MQ 1.0 based on RocketMQ to provide messaging services for large enterprises.

5. In 2016, Alibaba incubated RocketMQ into Apache;

Summary: Use this link to learn more about RocketMQ: The history of RocketMQ

RocketMQ characteristics

1. Flexible scalability. Naturally supported clusters, where each of the four core components can scale horizontally without a single point of failure;

2. Ability to accumulate massive information. Keep a low write delay while guaranteeing a large amount of message accumulation;

3. Supports sequential messages to ensure that consumers are consumed in the order in which they send messages. Sequential messages are divided into global ordered messages and local ordered messages. Generally, it is recommended to use local ordered messages, that is, the producer sends a certain type of messages to the same queue in sequence.

4. Message storage is another important feature of MQ. Storage is generally considered at the following two latitudes:

4.1. Message stacking capability: In order to pursue high performance of message storage, memory mapping mechanism is introduced to store all topic messages in the same file in sequence;

4.2 message storage: In order to avoid unlimited accumulation of messages in the message storage server, it is necessary to set the expiration time and space alarm mechanism;

5. Supports multiple message filtering methods. Message filtering is divided into server – side filtering and consumer – side filtering. The server side filtering can be carried out according to the requirements of message consumers. The advantages of filtering can reduce unnecessary transmission, while the disadvantages of increasing the burden of the server, relatively complex implementation. On the consumer side, filtering is completely customized by the specific application, which is more flexible. The disadvantage is that many useless messages are transmitted to message consumers;

6. Support transactional messages.

7. Support Huishuo consumption. A message that has been successfully consumed needs to be re-consumed due to business requirements. RocketMQ allows you to consume backwards and forwards in milliseconds.

RocketMQ has four core components

As mentioned earlier, each of RocketMQ’s four core components can scale horizontally to avoid a single point of failure. As shown in the figure:

1.Produce(producer) : Is responsible for producing and sending messages to be more precise, delivering messages generated by the application to the Broker. RocketMQ provides three ways to send messages:

1.1. Synchronous sending: After sending a message, the message will be blocked and wait for the message service to respond to confirm receiving the message before sending the next data. It is used for important message notification scenarios. Such as important notification mail, marketing messages, etc.

1.2. Asynchronous sending: Sending a message can proceed to the next packet without blocking and waiting for a response from the server. It is applicable to service scenarios where links may take a long time and are insensitive to response. Such as video transcoding notification;

1.3. Single send: Only responsible for sending messages without waiting for server response and no callback function triggered. It is applicable to scenarios that require very short time but do not have high reliability requirements, such as log collection.

A Producer Group is a collection of a class of producers that usually send one type of messages and send them logically. In terms of deployment structure, a producer identifies itself as a cluster by the name of a producer group.

3. Consumer: Responsible for Consumer messages. From the perspective of users, there are two types of consumers: pull consumers and push consumers.

3.1. Pull Consumer Actively pulls messages from the message server. As long as the messages are pulled in batches, the user application will start the consumption process.

3.2. Push Consumer encapsulates the pull, consumption schedule, and other internal maintenance of the message, leaving the callback interface performed when the message arrives for the user application to implement. So Push is passive consumption. However, the message is still pulled from the server, but the difference is that Push has more process of registering the consumption listener, when the listener is triggered before consuming the message;

4. Consumer Group: it is a collection of consumers who usually consume the same kind of messages and consume logically, so they are grouped together.

5. Consuming server: A message Broker is a message storage center. Its main function is to receive messages from producers, and consumers come to pull messages. The Broker also stores metadata related to the message, including user groups, consumption progress offsets, queue information, and so on.

As you can see from the deployment structure above, there are two types of brokers: Master and slave. Master can be written or read. Slave can be read. You can’t write.

The Broker can be clustered in a single master, multi-master, multi-master, and multi-slave mode

5.1. Single master: a single point of failure exists and is not recommended.

5.2. Multi-master: All servers are master servers without slave servers. The configuration is simple, but there is a risk of message loss.

5.3. Multiple Master and Slave (synchronous replication) : Configure a slave for each master. Multiple master-slave is used. This approach avoids a single point of problem and provides high availability of services and data. But the performance is low, synchronization means blocking, sending messages slightly higher latency;

5.4. Multiple Master and Slave (Asynchronous Replication) : Configure one slave for each master. The asynchronous replication mode is adopted, and the message delay between the master and slave is millisecond. The advantage of this method is that the lost message is very few, and the real-time of the message will not be affected, the disadvantage is almost the same as the multi-master, but this method after service downtime data loss is very little;

6. NameServer: Used to store Broker meta information and find Broker information for producers and consumers. The name server is designed to be almost stateless, and every Broker is registered with the name server when it is started. Producers and consumers get routing information from the name server based on the topic name.

7. Message: The entity to which data is transmitted. Each message must correspond to a topic so that it can be found and consumed. Topics and messages are similar to k-V relationships;

The theme

A Topic can be thought of as a categorization of messages, which is the first level of type of messages. For example, an e-commerce system can be divided into transaction messages, logistics messages and so on. The relationship between a topic and producers and consumers is very loose. A topic can have 0, 1 or more producers sending messages to it. A producer can also send messages to different topics at the same time.

The label

The Tag Tag, which can be thought of as a subtopic, is the second level type of message to provide additional flexibility for the user.

The queue

Topics are divided into one or more subtopics, called queues. Multiple queues can be set up under the same topic, and when a message is sent, the topic of the message is executed, and RocketMQ polls all queues under that topic before sending the message out. As shown in the figure:

Message consumption pattern

There are two patterns of news consumption: Clustering and Broadcasting. The default is cluster consumption. In this mode, a consumer cluster jointly consumes multiple queues of a topic, and a queue is consumed by only one consumer. If a consumer fails, other consumers in the group will continue to consume after the consumer fails. The broadcast consumer sends a message to each consumer in the consumer group for consumption.

Message order

1. Sequential consumption: indicates that consumers consume messages in the order sent by producers.

2. Parallel consumption: Message order is no longer guaranteed and the maximum amount of parallel consumption is limited by the thread pool specified by each consumer customer