The implementation logic for Kafka has been shared in previous articles. Kafka performs well in terms of high throughput, but sometimes we need to implement more complex business logic that is not too demanding in terms of throughput. RocketMQ is our choice.
Why RocketMQ when you have Kafka?
We know that Kafka has very good performance and very high throughput. Kafka’s single-machine write TPS claims to be in the millions per second; In pursuit of performance, Kafka standalone performance is higher. This also relies on sequential writes, Memory Mapped Files techniques, and zero Copy on the consumer side. Kafka implements Zero Copy based on SendFile, directly from the kernel space (DMA) to the kernel space (Socket), and then sends the nic. However, the complex application scenarios in daily work and strict requirements on data reliability are a little inadequate. We analyze it from the following aspects:
[Data reliability]
RocketMQ supports synchronous mode to improve data reliability. RocketMQ supports asynchronous/synchronous flush; Asynchronous/synchronous Replication; Kafka uses asynchronous flush mode and asynchronous Replication.
[Message ordering]
Kafka supports message ordering in some configurations, but messages are out of order when a Broker goes down.
RocketMQ supports strict message ordering. In a sequential message scenario, when a Broker goes down,
Sending messages will fail, but not out of order;
[About timing/delay messages]
Kafka does not support timed messages;
RocketMQ supports timed messages;
[About Distributed Transaction Messages]
Kafka does not support distributed transaction messages; RocketMQ supports distributed transaction messaging
[About message query mechanism]
Kafka does not support message query.
RocketMQ supports querying messages by Message Id as well as by Message content
[About the design of Broker]
Kafka does not perform as well as rocketMQ when the number of topic partitions in the broker is too high.
Both Kafka and rocketMq use file storage, but Kafka is partitioned one file at a time. When there are too many topics, the total number of partitions increases. There are too many files in Kafka, and when messages are flushed, files compete with disks, resulting in performance degradation. A partition a file, read and write sequentially. A partition can only be consumed by one consuming thread in a consuming group, so fewer consumers can consume at the same time.
RocketMq all queues are stored in one file, and the amount of messages stored in each queue is relatively small, so the increase of topics has a small impact on rocketMq performance. RocketMq can exist in a large number of topics, can adapt to more complex business.
RocketMQ architecture design
RocketMq technology architecture
RocketMQ is architecturally divided into four main parts, as shown in the figure above:
-
Producer: a role that publishes messages and supports distributed cluster deployment. The Producer uses MQ’s load balancing module to select the corresponding Broker cluster queue for message delivery, which supports fast failure and low latency.
-
Consumer: message consuming role, which supports distributed cluster deployment. Messages can be consumed in push and pull modes. At the same time, it also supports the consumption of cluster mode and broadcast mode. It provides real-time message subscription mechanism, which can meet the needs of most users.
-
NameServer: NameServer is a very simple Topic routing registry that acts like ZooKeeper in Dubbo and supports dynamic registration and discovery of brokers. There are two main functions: Broker management. NameServer accepts the registration information of Broker clusters and stores it as the basic data of routing information. It then provides a heartbeat detection mechanism to check whether the Broker is still alive. Routing information management, where each NameServer holds the entire routing information about the Broker cluster and the queue information for client queries. The Producer and Conumser can then use NameServer to know the routing information of the entire Broker cluster and deliver and consume messages. NameServer is also typically deployed in clusters, where instances do not communicate with each other. The Broker registers its routing information with each NameServer, so each NameServer instance has a complete routing information stored on it. When a NameServer goes offline for some reason, the Broker can still synchronize its routing information with other Nameservers. Producers and consumers can still dynamically perceive the routing information of the Broker. But NameServer does not offer the same voting functionality as ZK
-
BrokerServer: The Broker is responsible for storing, Posting, and querying messages and for ensuring high availability of services. To enable these functions, the Broker contains several important sub-modules.
-
Remoting Module: The entity of the entire Broker that handles requests from clients.
-
Client Manager: Manages clients (Producer/Consumer) and maintains Topic subscription information of consumers
-
Store Service: Provides convenient and simple APIS for storing messages to physical disks and querying messages.
-
HA Service: A high availability Service that provides data synchronization between Master and Slave brokers.
-
Index Service: Indexes messages delivered to the Broker based on a specific Message key to provide quick lookup of messages.
RocketMq architecture deployment
RocketMQ network deployment features
-
NameServer is a nearly stateless node that can be clustered and no information is synchronized between nodes.
-
Broker deployment is relatively complex, the Broker is divided into the Master and Slave, a Master can correspond to more than one Slave, but a Slave corresponds to only one Master, Master and Slave corresponding relation by specifying the same BrokerName, A different BrokerId is defined, with a BrokerId 0 for Master and a non-0 for Slave. Multiple masters can also be deployed. Each Broker establishes long connections to all nodes in the NameServer cluster and periodically registers Topic information to all NameserVers. Note: The current RocketMQ release supports one Master multiple slaves on deployment architecture, but only Slave servers with BrokerId=1 will participate in the read load of messages.
-
Producer establishes a long connection with one node (randomly selected) in the NameServer cluster, obtains Topic routing information from NameServer regularly, establishes a long connection with the Master providing Topic services, and sends heartbeat messages to the Master periodically. Producer is stateless and can be deployed in a cluster.
-
The Consumer establishes a long-term connection with one node (randomly selected) in the NameServer cluster, obtains Topic routing information from NameServer periodically, establishes a long-term connection with the Master and Slave that provide Topic services, and periodically sends heartbeat messages to the Master and Slave. A Consumer can subscribe to messages from either the Master or Slave. When a Consumer pulls a message from the Master, the Master server determines whether to read old messages and generate read I/O based on the distance between the offset and the maximum offset. And whether the Slave server is readable. You are advised to pull the data from the Master or Slave next time.
Together with the deployment architecture diagram, describe the cluster workflow:
-
Start NameServer. The NameServer listens to the port and waits for brokers, Producers, and consumers to connect to it, acting as a routing control center.
-
The Broker starts, maintains long connections to all Nameservers, and periodically sends heartbeat packets. The heartbeat package contains the current Broker information (IP+ port, etc.) and stores all Topic information. After registration, there is a mapping between topics and brokers in the NameServer cluster.
-
Before sending or receiving a message, a Topic is created. When creating a Topic, you need to specify which brokers the Topic will be stored on, or you can automatically create a Topic when sending a message.
-
The Producer sends a message and establishes a long connection with one of the NameServer clusters when starting the message. The Producer obtains from the NameServer which brokers the currently sent Topic exists. The polling selects a queue from the queue list. It then establishes a long connection with the Broker on which the queue resides to send messages to the Broker.
-
Similar to Producer, a Consumer establishes a long connection with one of the Nameservers, obtains information about which brokers are currently subscribed to, and then directly establishes a connection channel with the brokers to consume messages.
Data flow between modules
Production-consumption model
Production and consumption process
