RocketMQ- Architectural Design

1 Technical Architecture

RocketMQ is architecturally divided into four main parts, as shown in the figure above:

  • Producer: a role that publishes messages and supports distributed cluster deployment. The Producer uses MQ’s load balancing module to select the corresponding Broker cluster queue for message delivery, which supports fast failure and low latency.

  • Consumer: message consuming role, which supports distributed cluster deployment. Messages can be consumed in push and pull modes. At the same time, it also supports the consumption of cluster mode and broadcast mode. It provides real-time message subscription mechanism, which can meet the needs of most users.

  • NameServer: NameServer is a very simple Topic routing registry that acts like ZooKeeper in Dubbo and supports dynamic registration and discovery of brokers. There are two main functions: Broker management. NameServer accepts the registration information of Broker clusters and stores it as the basic data of routing information. It then provides a heartbeat detection mechanism to check whether the Broker is still alive. Routing information management, where each NameServer holds the entire routing information about the Broker cluster and the queue information for client queries. The Producer and Conumser can then use NameServer to know the routing information of the entire Broker cluster and deliver and consume messages. NameServer is also typically deployed in clusters, where instances do not communicate with each other. The Broker registers its routing information with each NameServer, so each NameServer instance has a complete routing information stored on it. When a NameServer goes offline for some reason, the Broker can still synchronize its routing information with other Nameservers. Producers and consumers can still dynamically perceive the routing information of the Broker.

  • BrokerServer: The Broker is responsible for storing, Posting, and querying messages and for ensuring high availability of services. To enable these functions, the Broker contains several important sub-modules.

    1. Remoting Module: The entity of the entire Broker that handles requests from clients.
    2. Client Manager: Manages clients (Producer/Consumer) and maintains Topic subscription information of consumers
    3. Store Service: Provides convenient and simple APIS for storing messages to physical disks and querying messages.
    4. HA Service: A high availability Service that provides data synchronization between Master and Slave brokers.
    5. Index Service: Indexes messages delivered to the Broker based on a specific Message key to provide quick lookup of messages.

2 Deployment Architecture

RocketMQ network deployment features

  • NameServer is a nearly stateless node that can be clustered and no information is synchronized between nodes.

  • Broker deployment is relatively complex, the Broker is divided into the Master and Slave, a Master can correspond to more than one Slave, but a Slave corresponds to only one Master, Master and Slave corresponding relation by specifying the same BrokerName, A different BrokerId is defined, with a BrokerId 0 for Master and a non-0 for Slave. Multiple masters can also be deployed. Each Broker establishes long connections to all nodes in the NameServer cluster and periodically registers Topic information to all NameserVers. Note: The current RocketMQ release supports one Master multiple slaves on deployment architecture, but only Slave servers with BrokerId=1 will participate in the read load of messages.

  • Producer establishes a long connection with one node (randomly selected) in the NameServer cluster, obtains Topic routing information from NameServer regularly, establishes a long connection with the Master providing Topic services, and sends heartbeat messages to the Master periodically. Producer is stateless and can be deployed in a cluster.

  • The Consumer establishes a long-term connection with one node (randomly selected) in the NameServer cluster, obtains Topic routing information from NameServer periodically, establishes a long-term connection with the Master and Slave that provide Topic services, and periodically sends heartbeat messages to the Master and Slave. A Consumer can subscribe to messages from either the Master or Slave. When a Consumer pulls a message from the Master, the Master server determines whether to read old messages and generate read I/O based on the distance between the offset and the maximum offset. And whether the Slave server is readable. You are advised to pull the data from the Master or Slave next time.

Together with the deployment architecture diagram, describe the cluster workflow:

  • Start NameServer. The NameServer listens to the port and waits for brokers, Producers, and consumers to connect to it, acting as a routing control center.
  • The Broker starts, maintains long connections to all Nameservers, and periodically sends heartbeat packets. The heartbeat package contains the current Broker information (IP+ port, etc.) and stores all Topic information. After registration, there is a mapping between topics and brokers in the NameServer cluster.
  • Before sending or receiving a message, a Topic is created. When creating a Topic, you need to specify which brokers the Topic will be stored on, or you can automatically create a Topic when sending a message.
  • The Producer sends a message and establishes a long connection with one of the NameServer clusters when starting the message. The Producer obtains from the NameServer which brokers the currently sent Topic exists. The polling selects a queue from the queue list. It then establishes a long connection with the Broker on which the queue resides to send messages to the Broker.
  • Similar to Producer, a Consumer establishes a long connection with one of the Nameservers, obtains information about which brokers are currently subscribed to, and then directly establishes a connection channel with the brokers to consume messages.

Please refer to github.com/apache/rock…