Why message queues

There are many application scenarios, typically producers and consumers. Message queues can be used when asynchronous processing is required, when flow control of queues is required, and when service decoupling is required.

Message queue has many benefits. For example, in some scenarios, message queue components can be temporarily expanded horizontally to resist heavy traffic and achieve the effect of peak filling. Produce the queue of consumers to realize the problem of inconsistent rates on both sides; It can also be other: adding stream computing processing; Implement an observer pattern between microservice-level systems as a publish/subscribe system; Used to broadcast messages to a large number of recipients. In short, the role of many, many application scenarios.

But the message queue is not complete, according to the actual situation to determine, according to the characteristics of various message queues to choose to use.

How do I select a message queue

The following indicators can serve as a reference

  1. Reliable delivery of messages: ensure that no messages are lost
  2. Cluster: Supports Cluster to ensure that services are not unavailable due to a node breakdown and messages cannot be lost.
  3. Performance: The performance is good enough to meet the performance requirements of most scenarios.

The above three points are often concerned in the work, the actual work is often encountered, but there are some other factors, just a reference

So what are the common message queues

  1. RabbitMQ
  2. RocketMQ
  3. Kafka

Redis is not a message queue, but it is used so much that we are familiar with it, so we put it here. They appear at different times and solve problems at different times.

First, RabbitMQ, which is an old message queue, came into existence in order to ensure reliable communication between telecommunication systems.

  • His message queue capacity is probably tens of thousands to hundreds of thousands of data per second, characterized by lightweight and fast, which is also his slogan.
  • And it also uses a very large number of messaging middleware.
  • One of the features is that it provides an Exchange module between the producer and the queue, which allows you to decide which queue the message should be placed in, and the rules are very flexible.
  • Finally, this middleware supports many languages, arguably one of the richest

His also has shortcoming, and is more headache

  • RabbitMQ does not support message stacking very well. It is designed to be a pipeline and a large backlog of messages is an anomaly and should be avoided. Sometimes this buildup is not of your own making, such as when someone else’s code steals your consumer’s computing resources, and you don’t expect it
  • RabbitMQ has the worst performance of the message queues we’ve covered, although it’s good enough for everyday use
  • It’s a terrible programming language, elang is really a big hole, not recommended.

The second is RocketMQ

This is a middleware boasting financial machine stability, all aspects are good, performance, stability and reliability are trustworthy. As a domestic product, the advantage is that the Chinese document is more complete, the disadvantage is that the use of people is not much, ecology compared with other or inferior.

Rocketmq supports transactions and hundreds of millions of levels of message stacking, just slightly lower than Kafka. There is also the ability to scale out subscribers efficiently

And finally Kafka, the big name

Kafka is developed in Scala and the Java language, and is designed with a lot of batching and asynchronous thinking, which makes Kafka extremely high performance. Kafka’s performance, especially for asynchronous send and receive, is the best of the three, but it’s not on the order of magnitude different from RocketMQ, handling around hundreds of thousands of messages per second.

However, its logic of batch sending is not good in real-time, and it is not suitable for some high real-time scenarios. But other than that, it’s used a lot in large interconnects.

Use of message queues

Common pain points used in message queues

  1. Lost a message

    Losing messages is a serious problem. The vast majority of lost messages are caused by developers not understanding their features. In order to avoid the loss of the message is not known, you can add a continuously increasing sequence number in the message, to determine whether the message is lost, which is lost.

    But there are some problems with distribution. For example, Kafka does not guarantee that the entire cluster is contiguous, only that individual instances are contiguous. So the general is to ensure partition data consistency.

  2. Ensure that messages are delivered reliably

    1. Production node

      Make sure the send is successful and catch any exceptions

    2. Storage stage

      Proper configuration ensures that data is saved to disk. If clustered, make sure the message is sent to multiple brokers

    3. After consuming the message, try to make sure the message has been consumed correctly before telling the queue that the message has been consumed

  3. Duplicate messages

    1. Message queues have several kinds of precision, At most once, At least once, Exactly once. The accuracy provided by general middleware is also known as AL least once. Kafka, too

    2. So what you do is you do idempotent operations on the consumer side. But some operations are not inherently idempotent, so there are common method operations to set idempotent

      1. Using the unique constraint of the database to achieve idempotence
      2. Set preconditions for updated data
      3. Record and review the operation

    3. Processing message Backlog

      1. The producers are too fast, so they slow down consumption on the producers’ side;
      2. Consumers are too slow to improve the consumption rate of consumers. Generally speaking, the reasonable design is that the efficiency of consumers is higher than the efficiency of producers.
      3. Whether the consumption end is abnormal;