RocketMQ Primer

RocketMQ is a distributed and queue model open source messaging middleware, formerly known as MetaQ. It is a queue model messaging middleware developed by Ali. After being opened to Apache Foundation, RocketMQ has become apache’s top open source project, featuring high performance, high reliability, high real-time and distributed.

At the same time, it is widely used in many fields, including asynchronous communication decoupling, enterprise solutions, financial payment, telecommunications, e-commerce, express logistics, advertising and marketing, social, instant messaging, mobile applications, mobile games, video, Internet of Things, Internet of vehicles, etc.

It has the following characteristics:

  • Can ensure strict message order
  • Provides rich message pull patterns
  • Effective subscriber level expansion capabilities
  • Real-time message subscription mechanism
  • Hundred million message accumulation capability

RocketMQ architecture analysis

RocketMQ architecture

NameServer:

  • Provides lightweight service discovery and routing. NameServer accepts registrations from Broker clusters and provides a signal detection mechanism to check if the Broker still exists
  • Each NameServer records complete routing information (Broker related topics and other meta information, and provides the Producer Consumer to find Broker information) and provides corresponding read and write services.

Broker: A message storage center that receives messages from producers and stores them, and from which consumers receive messages

  • Single Broker nodes maintain long connections and heartbeat with all NameServer nodes and periodically register Topic information with NameServer (the underlying communication is Netty based).
  • The Broker is responsible for message storage and supports lightweight queues on a topic-scale. A single Broker can support tens of thousands of queues and supports a push and pull message model.
  • It has the ability to accumulate hundreds of millions of messages, and can strictly guarantee the order of messages

Producer:

  • Responsible for generating messages, producers send messages to the message server that are generated by the business application system

  • Producers support distributed deployment. Distributed producers send messages to Broker clusters through multiple load-balancing patterns. The sending process supports fast failure and low latency

  • There are three ways to send messages: synchronous, asynchronous, and unidirectional

Consumer:

  • Responsible for consuming messages, the consumer pulls information from the message server and enters it into the user application
  • Distributed deployment in the push and pull model is also supported.
  • It also supports cluster usage and message broadcasting. It provides a real-time message subscription mechanism that meets the needs of most consumers.

Broker Server

Broker Server is responsible for message storage and delivery, message query, HA high availability, etc.

Remoting Module: Broker portal that handles requests from clients

Client Manager: Manages clients (producers/consumers) and maintains topic subscriptions for consumers

Store Service: Provides a simple API for storing or querying messages in a database

HA Service (high availability Service) : Provides data synchronization between master and slave brokers

Index Service: Index messages to provide quick lookup capabilities

RocketMQ overall process

  1. Start NameServer. After NameServer is started, it listens to the port and waits for brokers, Producers, and consumers to connect to it, acting as a routing control center
  2. BrokerStart, keep long connections with all NamesRVs, and periodically send heartbeat packets
    • The heartbeat package contains the current Broker information (IP+ port, etc.) and stores all Topic information
    • After registration, the Namesrv cluster has a mapping between topics and brokers
  3. Create a Topic before sending or receiving a message. When you create a Topic, you need to specify which brokers the Topic will be stored on. A Topic can also be created automatically when a message is sent
  4. ProducerSend a message
    • At startup, a long connection is established with one of the Namesrv clusters, and the brokers of the currently sent Topic are obtained from Namesrv
    • Then establish a long connection with the corresponding Broker and send messages directly to the Broker
  5. ConsumerNews consumption
    • Establish a long connection with one of the NamesRVs to find out which brokers the current subscription Topic exists on
    • Then establish a connection channel directly with the Broker to start consuming message *RocketMQ’s message domain model

RocketMQ Message

Topic: Represents the first level of type of message and is the most fine-grained subscriber unit (producer passing message and consumer extracting message identity)

  • A message must have a Topic
  • A Group can subscribe to messages from multiple topics
  • A Topic is typically a domain, such as a trade message

Tag: indicates the second-level type of messages, which can use the same Topic and different tags to represent different tasks of the same business module. For example, transaction messages can be divided into transaction creation messages and transaction completion messages

  • Helps keep code clean and consistent
  • Simplify the query system provided by RocketMQ

Message: A Message is a Message to be delivered. Message must contain a Topic, with optional Tag and key-vaule key-value pairs

Message Queue: All Message queues are persistent

  • A Topic can have multiple queues
  • The introduction of Queue enables the storage of messages to be distributed and clustered, with horizontal scaling capability

Groups: They are divided into Producer groups and Consumer groups with the same roles

  • When the original producer crashes after the transaction, the broker can contact different instances of producers in the same producer group to commit or rollback the transaction.

  • The consumer instance of the consumer group must have the exact same topic subscription

RocketMQ features

Message Model:

  • Clustering: When using the clustered consumption pattern, MQ assumes that any message needs to be processed by only any one consumer in the cluster
  • Broadcasting (broadcast) : When using broadcast consumption, MQ pushes each message to all registered clients in the cluster, ensuring that the message is consumed at least once by each machine

Message Order

  • When using DefaultMQPushConsumer, you can decide to use the messages sequentially or simultaneously

    • Orderly use of messages means that messages are consumed in the same order as the producer sends messages for each message queue. (If you want to deal with situations where global ordering must be enforced, make sure that the topic you use has only one message queue)

    If sequential use is specified, the maximum concurrency used by the message is the number of message queues subscribed to by the consumer group

    • Concurrently using messages, the maximum concurrency of message use is limited only by the thread pool specified for each user client

    Message order is no longer guaranteed in this mode

Message Types

  • Transaction message
  • The order message
  • Delay message

RocketMQ standalone installation

  1. Download compiled source code

      # download $> > wget wget http://mirror.bit.edu.cn/apache/rocketmq/4.6.0/rocketmq-all-4.6.0-source-# extract $> unzip rocketmq - all - 4.7.0 - source - the zip >cdRocketmq - all - 4.7.0 /Compile # $
  > mvn -Prelease-all -DskipTests clean install -U
  > cdDistribution/target/rocketmq - 4.7.0 / rocketmq - 4.7.0Copy the code

  2. Start the Name Server

     Start the Name Server service
 > nohup sh bin/mqnamesrv &
 After the startup is complete, check log $
 > tail -f ~/logs/rocketmqlogs/namesrv.log
  The Name Server boot success...
Copy the code

  3. Start the Broker

    In the conf directory, RocketMQ provides configuration files for various brokers:

    • broker.conf: Single master, asynchronous disk flushing.
    • 2m/: Two active disks are flushed asynchronously.
    • 2m-2s-async/: Two master disks, two slave disks, asynchronous replication, asynchronous disk flushing.
    • 2m-2s-sync/: Two master and two slave disks, synchronous replication, and asynchronous disk flushing.
    • dledger/: Dledger cluster, at least three nodes
     Start the Broker service
 > nohup sh bin/mqbroker -n localhost:9876 &
 After the startup is complete, check log $
 > tail -f~ / logs/rocketmqlogs/broker. Log The broker [172.30.30.233% s: 10911] The boot success...Copy the code

    Where, parameters:

    • With the -c parameter, configure the primary Broker configuration that is read

    • Use the -n parameter to set the RocketMQ Namesrv address

  4. Send and Receive Messages

    Before sending/receiving a message, we need to tell the client (producer/consumer) the address of the Name Servers. RocketMQ provides several ways to do this:

    • In the code: producer.setNamesrvaddr (” IP :port”)
    • Java property configuration: Rocketmq.namesrv.addr
    • Environment variable configuration: NAMESRV_ADDR
    • HTTP Endpoint

    For simplicity, we use the environment variable: NAMESRV_ADDR, as follows:

     Set the address of Name Servers to $
 > export NAMESRV_ADDR=localhost:9876
 # production message $
 > sh bin/tools.sh org.apache.rocketmq.example.quickstart.Producer
 SendResult [sendStatus=SEND_OK, msgId= ...
 # Consume message $
 > sh bin/tools.sh org.apache.rocketmq.example.quickstart.Consumer
 ConsumeMessageThread_%d Receive New Messages: [MessageExt...
Copy the code

Is everyone still ok? If you like, move your hands to show 💗, point a concern!! Thanks for your support!

Welcome to pay attention to the public number [Ccww technology blog], original technical articles launched at the first time