Message queue design

preface

Recently I was asked how you would design a message queue, and from what angles you would design it. I looked back at my understanding of how RabbitMQ and Kafka, two of the most popular messaging queues, work, and gathered some of the answers to these questions.

Message queues refer to my blog stream source’s Java notes – Message queues

The body of the

The message queue

The main functions that message queues are intended to implement

  • Decoupling: Message-based models, which are concerned with “notification” rather than “processing” and are relatively more concerned with results than processes, can reduce system-to-system coupling through message queues, in other words, the inefficiency of one system does not drag down other systems.
  • Final consistency: Because of the high cost of strong consistency, we actually choose the final consistency when designing message queues: mainly using “logging” and “compensation.”
  • Broadcast: With publish-subscribe, producers simply publish messages and do not need to maintain or care about who consumes them.
  • Peak shifting and flow control: When the producer’s release rate is out of balance with the consumer’s consumption rate, load balancing and current limiting are used to perform peak flow clipping.

The nature of queues

  • One RPC changes to two RPCS: From direct onceRPCThe interface is called twiceRPC: Publishing and consuming messages.
  • Content store: with the help ofbrokerStore content to avoid message loss.
  • Select the right timing for delivery: The right timing can be understood as staggered delivery of messages smoothly, avoiding message accumulation.

Key points of queue Design

RPC communication protocol

You can choose RPC frameworks such as Thrift and Dubbo, or rewrite RPC frameworks using Memchached or Redis protocols. However, the former is recommended. RabbitMQ uses the AMQP protocol. Kafka uses a TCP layer protocol designed for RabbitMQ.

Store selection

In terms of speed, file system > Distributed KV > Distributed file system > database, while reliability is the opposite. If 5-digit QPS performance is required for a single broker, file-based storage is the preferred solution. On the whole, it can be processed by data file + index file.

The Rabbitmq storage

All messages in the queue are written to a file as an append. When the file size exceeds the specified limit, the file is closed and a new file is created for writing messages. The file name (*.rdq) starts at 0 and then accumulates. When a message is deleted, the relevant information is not deleted from the file immediately. Instead, some records are made, and when the garbage data reaches a certain percentage, garbage collection is started to merge the data from logically adjacent files into one file.

Kafka’s storage

Each partion is equivalent to a huge file divided evenly among multiple equally sized segment data files.

It’s made up of two parts. Index file and data file respectively. These two files correspond to each other in pairs. The suffix “.index “and”.log “denote the segment index file and data file respectively.

Consumption relationship processing

Generally speaking, there are two kinds of consumption relationships:

  • Unicast: Point-to-point, as in queue messages
  • Broadcast: one to many, as in a subject message

The maintenance of broadcast relationships is generally maintained on common storage, such as Config Server and ZooKeeper, because message queues themselves are clusters.

RabbitMQ and Kafka both support queue and topic messages.

Implementing transaction

ACID properties of transactions include atomicity, consistency, isolation and persistence. The most important one is consistency, which can be achieved by:

  1. Two-stage submission (Two-stage submission protocols are too costly and for arbitrationdownMachine or single point of failure has a significant impact on business.
  2. Local transaction, local landing, compensation send. (Transaction-based message: send a message to the message-oriented middleware, and then execute a local transaction. When the local transaction succeeds, send a commit confirmation to the message-oriented middleware, and then the message can be perceived by other business consumers.)

The affairs of the RabbitMQ

RabbitMQ supports two ways to implement transactions:

  • The transaction model:txSelect().txCommit()As well astxRollback().txSelectUsed to link the currentchannelSet totransactionMode,txCommitUsed to commit transactions,txRollbackUsed to roll back transactions
  • Confirm patternRelative:transactionThe pattern has higher message throughput and ensures atomicity of transactions through producer validation and consumer validation.

Kafka’s affairs

Kafka implements transactions in the second way, using a TransactionalID (a unique identifier for a Transaction) to associate ongoing transactions and a Transaction Coordinator to control the flow of transactions.

Anti-loss/anti-weight

The high availability of message queues, provided that the broker’s interface to receive and acknowledge messages is idempotent, and that several consumer machines process messages are idempotent, turns the availability of message queues over to the RPC framework. So how do you guarantee idempotence? The easiest way to do this is to share storage. Broker multiple machines sharing a DB or a distributed file/KV system processing messages is naturally idempotent.

Lost-proof and lost-proof are two tradeoff problems. Repeated delivery can solve the lost-proof problem well, so the idempotent design of messages must be considered.

RabbitMQ anti-loss mechanism

  • Production phase: failure callback mechanism, publisher confirmation, message persistence
  • Storage phase: standby exchange, dead letter exchange, transaction, high availability queue, transaction based high availability queue, message persistence
  • Consumption phase: consumer confirmation, message persistence

Kafka anti-loss mechanism

  • Producer confirmation mechanism
  • Producer failure callback mechanism
  • Failure retry mechanism
  • Consumer confirmation mechanism
  • A copy of the mechanism
  • qualifiedBrokerselectLeadermechanism

Message idempotent design scheme

  • Database based on the primary key index/unique index to achieve
  • Based on theRedisTo implement, usesetOperations are inherently idempotent
  • Determine whether an operation is to be added or updated by checking the data first
  • Determine whether the data is new or old by prefacing a Bloom filter to the database and using a primary key index
  • KafkaCan be achieved byProducerIDandSequenceNumberEnsure that messages are idempotent
  • Using state machines to ensure idempotency, order states can be initialized, order in progress, order failed, order successful, thus limiting repeat orders
  • For the front-end order adoptiontokenIdempotent check, prevent repeated click or network reasons caused by repeated submission

Asynchronous/batch and performance

Asynchronous: liberated thread and I/O, I/O can use I/O multiplexing technology, reduce the performance loss of the establishment of I/O channel, generally can use multithreading, NIO to achieve asynchronous.

RabbitMQ uses Netty to implement asynchrony, Kafka uses AIO’s Future to implement asynchrony.

Batch: Messages are processed in batches to reduce the number of network transfers.

RabbitMQ supports batch sending and batch consuming. Kafka can set the batch commit size by setting batch.size. The default value is 16K, and all messages sent to the same partition will be sent when they reach this value.

Kafka also supports MMAP (memory-mapped files), DMA (direct memory access), sequential reads and writes to disks, and more to improve performance.

Push or pull mode

There are two modes of message queuing:

  • The pull model: Consumers take the initiative to pull messages from message middleware, which has poor real-time performance, but the server does not need to careconsumerThe state can be protectedcunsumerStability.
  • Push mode: Message middleware actively pushes messages to consumers. This mode has good real-time performance, but it is easy to accumulate messages.
  • Prevailing message queuing principles:producerPush the message tobroker.consumerfrombrokerPull the message.

RabbitMQ supports both pull and push modes (the default). Kafka supports only pull.

Slow consumption

The push pattern causes message accumulation in the following ways:

  • If the speed of the consumer is much slower than the speed of the sender, the message will appear inbrokerThe accumulation;
  • brokertoconsumerPush a bunch ofconsumerUnprocessed messages,consumernotrejectiserror“And kicked the ball back and forth.

The Pull pattern avoids message heap problems:

  • consumerYou can consume what you want without being bombarded with messages you can’t handle
  • brokerInstead of logging the state of every message to be sent, heap messages maintain queues and offsets for all messages.

Message delay and busy

The biggest drawback of the pull model is that consumers cannot accurately decide when to pull the latest information.

There are two solutions:

  • pullThe interval delay usually uses several steps of growth wait. Wait for 5ms, then 10ms, then 20ms, then 40ms… Until a message arrives, then back to 5ms.
  • If trying to pull fails, not directlyreturnInstead, hang the connection therewaitIf a new message arrives, the server connectsnotifyup

Order consumption

RabbitMQ implements message ordering by splitting queues: split queues, each corresponding to a consumer, which is queued internally by a memory queue and distributed to different workers at the bottom:

  • We’re breaking it upqueue, through the unique identifierhashComputation ensures that the same business data will enter the samequeue.
  • aqueueFor a consumer, consumers are internally queued with memory queues and then distributed to different underlying onesworkerTo deal with.



KafkaMessage order is achieved by using message key order preserving strategy:

  • KafkaMessage ordering is not guaranteed globally, but only for a partition of the topic
  • By specifyingkey(such as order number), have the samekeyMessages are distributed to the samepartition, the same business related message will enter the samepartition.partitionThey are sorted internally to keep the message locally ordered.
  • apartition(Partition) corresponds to oneconsumer, an internal single consumer consumes, but that does not mean a single consumer is a single thread.
  • Create multiple memory queues on the consumer side that are identicalkeyAre routed to the same memory queue; Each thread then consumes one memory queue to ensure orderliness.