Message queue overall design idea

The main idea is to design an overall data flow in which messages are consumed.

There are Producer messages, Broker messages, and Consumer messages.

1.Producer: Sends messages to the Broker.

2.Broker: The concept of Broker mainly comes from Apache’s ActiveMQ and refers specifically to the server of message queues.

The main function is to transfer the message from the sender to the receiver, which involves the storage of the message and the message communication mechanism.

3. Message Consumer: Receives a message from the message queue, and the Consumer replies with a Consumer confirmation.

1) The dump of messages: the delivery of messages at a more suitable time or through a series of means to help the message finally reach the consumer machine.

2) Specification of a paradigm and a common pattern to meet the needs of decoupling, final consistency, staggered peaks, etc.

3) In fact, it is simply a message forwarder, which makes one RPC into two RPCS. The sender delivers the message to the broker, and the broker forwards the message to the receiver.

This sums up to two RPCS plus one dump, or three RPCS if confirmation of consumption is to be done.

To implement the basic functions of the above message queue:

  • Transmission of messages
  • storage
  • consumption

It needs to involve the following three aspects of design:

  • Communication protocol
  • Storage option
  • Consumer relationship maintenance

Communication protocol

Message: Both the Message carrier, the Message sender needs to know how to construct the Message, the Message receiver needs to know how to parse the Message, they need to describe the Message in a unified format, this unified format is called Message protocol.

Traditional communication protocol standards include XMPP and AMQP, but now more message queues use their own communication protocols from the point of view of performance.

1.JMS

JMS (Java MessageService) actually refers to the JMS API. JMS is a message standard proposed by Sun company early, which aims to provide unified message operations for Java applications, including creating messages, sending messages, receiving messages, etc.

JMS typically contains the following roles:

JMS provides two message models:

  • Point to point
  • And a publish-subscribe model.

When using the point-to-point model, messages are sent to a queue whose messages can only be consumed by one consumer.

With a publish-subscribe model, messages can be consumed by multiple consumers.

In the publish-subscribe model, producers and consumers are completely independent and do not need to perceive each other’s presence.

2.AMQP

AMQP is the Advanced Message Queuing Protocol.

AMQP is not a concrete message queue implementation, but a standardized messaging middleware protocol.

The goal is to allow applications of different languages and systems to communicate with each other, and to provide a simple unified model and programming interface. Current mainstream ActiveMQ and RabbitMQ support AMQP protocol.

AMQP is a protocol, or more accurately, a binary Wire -level protocol. This is an essential difference from JMS. AMQP does not qualify from the API layer, but directly defines the data format for network exchange.

JMS and AMQP comparison

JMS: Allows communication between only Java-based messaging platforms

AMQP: AMQP allows multiple technologies to communicate protocols simultaneously

3.Kafka communication protocol

Kafka’s producers, brokers, and consumers use a self-designed PROTOCOL based on the TCP layer. Kafka’s protocols are tailored to Kafka’s own business needs.

Store selection

For distributed systems, there are several storage options

  • memory
  • Local file system
  • Distributed file system
  • nosql
  • DB

Memory is obviously the fastest in terms of speed, and can be a good choice for scenarios where message loss is allowed and message stacking capability is not high, such as logging.

DB is the simplest solution to implement reliable storage. Hbase is suitable for scenarios that require high reliability and consistency, such as transaction messages. Hbase is also a good choice for scenarios that do not require 100% data integrity and require performance and message accumulation.

Theoretically, in terms of speed, file system > Distributed KV > Distributed file system > database, while reliability is the opposite.

Still, make the most reasonable choice based on the business scenario you support. If your message queue is used to support payment/transaction, etc., with high reliability requirements, but not so high performance and volume requirements, and you don’t have the time and energy to devote to file storage system research, DB is the best choice.

Hbase is also a good choice for scenarios that do not require 100% data integrity and require performance and message accumulation. Typical scenarios such as Kafka can use Hadoop for message landing.

Consumption relationship processing

Our message queue is now tentatively capable of dumping messages.

The next important thing is to resolve the send/receive relationship and deliver the message correctly.

The message Queue on market defines a heap of confusing term, such as the Topic of the JMS specification/Queue, Kafka Topic/Partition/ConsumerGroup inside, the inside of the RabbitMQ Exchange and so on.

Put aside the phenomenon to see the essence, nothing more than unicast and broadcast difference.

The so-called unicast is point-to-point; Broadcasting, on the other hand, is point-to-multipoint.

In order to realize the broadcast function, we must maintain the consumption relationship. Usually, the message queue itself does not maintain the consumption subscription relationship. Mature systems such as ZooKeeper can be used to maintain the consumption relationship and send notifications when the consumption relationship changes.

Message queues need to support advanced features

In addition to the above basic function of message queue, message queue also needs to support transaction, message retry and other functions in some special scenarios.

  • Order of messages
  • Delivery reliability assurance
  • Message persistence
  • Support for different message models
  • Multi-instance clustering capability
  • Transaction characteristics, etc.

That’s how to design a message queue MQ, but I’ll cover the advanced features that message queues need to support later.