What is a message?
A “message” is a unit of data transferred between two computers.
Messages can be very simple, such as containing only text strings; It can also be more complex and may contain embedded objects.
What is a queue?
A Queue is a first-in, first-out (FIFO) data structure.
What is a message queue?
Message queue is a container for storing messages. Message queue is an important component in distributed system. The main purpose of using message queue is to improve system performance, reduce peak load and reduce system coupling through asynchronous processing.
What are the two basic message models?
1. Point-to-point (P2P) model
Using queues as message communication carriers; To satisfy the producer-consumer pattern, a message can only be consumed by one consumer, and unconsumed messages remain in the queue until consumed or timed out.
For example, if our producer sends 100 messages, two consumers consume them. Normally, both consumers consume half of the messages in the order in which they are sent.
2. Publish/subscribe (Pub/Sub) model
The Publish/subscribe model (Pub/Sub) uses topics as message communication vehicles, similar to the broadcast pattern.
After a publisher publishes a message, the message is delivered to all subscribers via a topic. Users who subscribe after a message is broadcast will not receive the message.
What is JMS?
The JAVA Message Service (JMS) is a JAVA Message Service. JMS clients can transfer messages asynchronously.
The JMS API is a standard or specification for messaging services that allows application components to create, send, receive, and read messages based on the JavaEE platform. It makes distributed communication less coupled, message service more reliable and asynchronous.
ActiveMQ is implemented based on the JMS specification.
JMS provides two message models:
(1) Peer – 2 – Peer
(2) the Pub/sub
What is AMQP?
Advanced Message Queuing Protocol (AMQP) is an application-layer standard that provides unified messaging services. AMQP is an open application-layer Protocol designed for message-oriented middleware and compatible with JMS. The client and messaging middleware based on this protocol can pass messages, regardless of the client/middleware and product, different development languages and other conditions.
AMQP RabbitMQ is based on the AMQP protocol.
Five message models are provided:
(1) direct exchange;
(2) the fanout exchange;
(3) topic change;
(4) headers exchange;
(5) the system exchange.
In essence, the last four are not much different from JMS’s PUB/SUB model, except for a more detailed breakdown of routing mechanisms.
AMQP defines the wire-level Protocol for messages, while JMS defines the API specification. In the Java architecture, multiple clients can interact with each other over JMS without the need to modify the code, but the cross-platform support is poor. AMQP is naturally cross-platform and cross-language.
JMS supports complex message types such as TextMessage and MapMessage. AMQP supports only byte[] message types (complex types can be serialized and sent).
Because of the routing algorithm provided by Exchange, AMQP can provide a variety of routing methods to deliver messages to message queues, whereas JMS only supports queue and topic/subscribe.
What is middleware?
Middleware is an independent system software or service program that distributed applications use to share resources between different technologies.
Middleware is located on the operating system of the client/server and manages computer resources and network communication. It is not low-level operating system software or business application software. It is not directly used by end users and cannot directly bring value to customers.
Middleware is refers to under the network environment in the operating system, database, system software and application software such as a connecting role between the distributed software, mainly to solve the heterogeneous network environment of distributed application software interconnection and interoperability problems, provides a standard interface, protocol, shielding the implementation details, improve application system easy portability.
What is messaging middleware?
Message queue middleware (message middleware for short) uses efficient and reliable message transmission mechanism to carry out asynchronous data transmission and integrate distributed system based on data communication.
By providing message queue model and message delivery mechanism, message middleware can extend the communication between processes in distributed environment. It can provide application decoupling, elastic scaling, redundant storage, traffic peak cutting, asynchronous communication, data synchronization and other functions in distributed environment.
As an important component in distributed system architecture, message-oriented middleware plays an important role.
What are the components of message-oriented middleware?
1 Broker
Message server, which acts as a server to provide message core services.
2 Producer
The message producer, the originator of the business, is responsible for producing messages to transmit to the broker.
3 Consumer
The message consumer, the handler of the business, is responsible for retrieving messages from the broker and performing business logic processing.
4 Topic
Topic, a unified gathering place for messages in the publish-subscribe mode. Different producers send messages to topic, which is distributed by MQ server to different subscribers, and messages are broadcast.
5 Queue
Queue: In PTP mode, a particular producer sends a message to a particular queue, and a consumer subscribes to a particular queue to receive a particular message.
6 Message
Message body: packets encoded according to fixed formats defined by different communication protocols to encapsulate service data and realize message transmission.
What are the usage scenarios of messaging middleware?
1 Asynchronous Communication
Some businesses do not want or need to process messages immediately. Message queues provide asynchronous processing, allowing users to queue a message without processing it immediately. Put as many messages on the queue as you want, and then process them as needed.
2 decoupling
Reduce the degree of strong dependence between projects and adapt to heterogeneous systems. It is extremely difficult at the beginning of a project to predict what requirements the project will encounter. Through the messaging system, an implicit data-based interface layer is inserted in the middle of the process, which is implemented by both processes and can be independently extended or modified as applications change, as long as they adhere to the same interface constraints.
3 the redundant
In some cases, the process of processing data will fail. Unless data is persisted, it will be lost. Message queues persist data until it has been fully processed, thus avoiding the risk of data loss. In the insert-retrieve-delete paradigm used by many message queues, your processing system needs to make it clear that a message has been processed before it can be removed from the queue, ensuring that your data is stored safely until you are done using it.
4 extensibility
Because message queues decouple your processing, it is easy to increase the frequency with which messages are queued and processed, simply by adding additional processing. No code changes, no parameters adjustments. Facilitate distributed capacity expansion.
5 Overload Protection
The application still needs to continue to play a role in the case of the surge of traffic, but such sudden traffic cannot be extracted and predicted; It’s a huge waste of resources to be on call in order to be able to handle these instantaneous spikes. Using message queues enables key components to withstand sudden access pressures without completely collapsing under sudden overload of requests.
6. Recoverability
The failure of a component does not affect the entire system. Message queuing reduces coupling between processes, so that even if a process that processes messages dies, messages that are queued can still be processed after the system recovers.
7 Order Guarantee
In most usage scenarios, the order in which data is processed is important. Most message queues are inherently sorted and ensure that the data will be processed in a particular order.
8 the buffer
In any significant system, there will be elements that require different processing times. Message queues help execute tasks most efficiently through a buffer layer that helps control and optimize the speed at which data flows through the system. To adjust system response time.
9 Data flow processing
Massive data flows generated by distributed systems, such as service logs, monitoring data, user behavior, etc., are collected in real time or in batches and then analyzed by big data. It is a necessary technology of the Current Internet to collect such data through message queues.
What are the advantages of introducing message-oriented middleware?
1. Improve system performance through asynchronous processing (peak clipping and response time reduction)
2. Reduce system coupling
What are the disadvantages of introducing message-oriented middleware?
1. Reduced system availability: The availability of the system is reduced to some extent, such as message loss, message middleware outage, etc.
2. Increased system complexity: After joining MQ, you need to ensure that messages are not re-consumed, deal with message loss, ensure that messages are delivered sequentially, and so on.
3. Consistency problem: Message queue can realize asynchrony. Asynchrony brought by message queue can indeed improve the response speed of the system, but consumers without correct consumption may introduce consistency problems.
What are the commonly used message-oriented middleware?
There are many messaging middleware products out there, and you can do secondary development based on open source middleware products, or implement a simple messaging middleware feature yourself using file systems, Redis, etc.
Let’s take a look at several commonly used message-oriented middleware and compare them.
What are the considerations for selecting a message-oriented middleware technology?
1. Functional dimensions
1.1 Priority queue
A priority queue is different from a first-in, first-out queue. Messages with a higher priority have the privilege of being consumed first, which guarantees different message levels for the downstream.
However, there is a prerequisite for this priority: If the consumer is consuming faster than the producer, and there is no message backlog in the messaging middleware server (commonly referred to simply as the Broker), it makes little sense to prioritize the messages that are sent because the consumer consumes the message as soon as the producer sends it. This is equivalent to having at most one message in the Broker, and priority is meaningless for individual messages.
1.2 Delay Queue
The delay queue stores the corresponding delayed message. The so-called “delayed message” means that after the message is sent, consumers do not want to get the message immediately, but wait for a specific time before they can get the message for consumption.
Delay queues are generally divided into two types: message-based delay and queue-based delay.
Message-based latency means that each message is set to a different delay time, so that every time a new message enters the queue, it is reordered by the delay time, which of course has a significant impact on performance.
In practical applications, queue-based delay is mostly used. Queues of different delay levels are set, such as 5s, 10S, 30s, 1min, 5mins, 10mins, etc. Messages in each queue have the same delay time, which avoids the performance pain of delayed sorting. Timeout messages can be delivered by certain scanning policies, such as timing.
1.3 Dead letter Queue
For some reason, the message cannot be delivered correctly. To ensure that the message will not be discarded, it is usually placed in a queue with a special role. This queue is usually called a dead letter queue.
Corresponding to this and the concept of a “queue”, just think, if the consumer is when the consumer the exception that would not have to confirm this time consumption (Ack), and then after the operation of the rollback message message will always be on the top of the queue, and then be processed and rollback, causes the queue in a infinite loop.
To solve this problem, you can set up a fallback queue for each queue, which, along with the dead-letter queue, provides a mechanism for handling exceptions. In practice, the role of rollback queues can be played by dead-letter queues and retry queues.
1.4 Retry Queue
A retry queue is a fallback queue. When a message fails to be consumed by a consumer, the message is rolled back to the Broker to prevent it from being lost.
Different from the rollback queue, the retry queue is generally divided into multiple retry levels. Each retry level also sets the redelivery delay. The more retries, the greater the delivery delay.
For example, a message fails to be consumed for the first time and is put into retry queue Q1. The redelivery delay of Q1 is 5s, and the message is redelivered after 5s. If the message fails to be consumed again, it is put into Q2, which has a redelivery delay of 10 seconds. The message is redelivered after 10 seconds. In this way, the more retries it takes, the longer it will take to repost, so you need to set an upper limit. If the number of reposts exceeds the number of reposts, the queue will be dead letter.
Retry queue and delay queue have something in common in that the delay level needs to be set. The difference between them is that the delay queue action is triggered internally, while the retry queue action is triggered externally by the consumer. The delay queue works once, while the scope of the retry queue is passed backwards.
1.5 Consumption Pattern
Consumption mode is divided into push mode and pull mode.
Push mode refers to the mode in which the Broker actively pushes messages to the consumer, which has good real-time performance, but requires a certain flow mechanism to ensure that the messages pushed by the server will not overwhelm the consumer.
In the pull mode, the consumer takes the initiative to request the Broker to pull (usually timing or quantitative) messages. The real-time performance is worse than that of the push mode, but the amount of messages pulled can be controlled according to its own processing capacity.
1.6 Broadcast consumption
Messages generally have two delivery modes: P2P (point-to-point) and publish/subscribe (Pub/Sub).
In the point-to-point model, once a message is consumed, it is not stored in the queue, so it is impossible for a message consumer to consume a message that has already been consumed. Although queues can support multiple consumers, a message will only be consumed by one consumer.
The publish subscribe pattern defines how to publish and subscribe to messages to a content node, called a topic, which can be thought of as a mediation for message delivery, with a publisher publishing messages to a topic from which message subscribers subscribe. Topic enables subscribers and publishers of messages to remain independent of each other, and does not need to contact each other to ensure the transmission of messages. Publish/subscribe mode is adopted in one-to-many broadcast of messages.
1.7 Message Tracing
Typically, a message is processed after it is consumed and cannot be consumed again.
Message backtracking, on the other hand, means that a message can be consumed after it has been consumed.
For messages, often face the problem is “lost” news, as for the loss as a result of the message middleware defect is real or lost due to the misuse of use party generally difficult to trace, if message middleware itself have news back function, can pass back consumption emersion “lost” message and then find out the source of the problem lies. Message backtracking can be used for more than this, including index recovery, local cache reconstruction, and some business compensation schemes.
1.8 Message accumulation and Persistence
Traffic peak clipping is a very important function of message-oriented middleware, which actually benefits from its message stacking capability.
In a sense, if a message-oriented middleware does not have the ability to stack messages, it cannot be considered a qualified message-oriented middleware.
Message heap can be stored and disk heap.
RabbitMQ is typically a heap in memory, but this is not always the case, either paging to disk (which affects throughput) or persisting messages directly to disk using lazy queues.
Kafka is a typical disk heap where all messages are stored on a disk. In general, the capacity of disk is much larger than the capacity of memory, and the capacity of disk heap is the size of the entire disk.
From another point of view, message heap also provides redundant storage for message middleware.
1.9 Message Tracking
It is no stranger to link tracing in distributed architecture systems.
For message middleware, message link tracing (hereinafter referred to as message tracing) is also important.
The most popular understanding of message tracking is to know where a message is coming from, where it exists, and where it is going. Based on this function, we can carry out link tracing service for messages sent or consumed, and then quickly locate and troubleshoot problems.
1.10 Message Filtering
Message filtering refers to providing downstream users with specified categories of messages according to predefined filtering rules.
In the case of Kafka, it is entirely possible to send messages of different categories to different topics, thereby enabling some sort of message filtering, or Kafka can sort messages within the same topic by partition. However, in a more strict sense, message filtering should be carried out according to certain filtering rules and in certain ways. Using Kafka as an example, messages can be filtered using the ConsumerInterceptor interface provided by the client or the Filter function of Kafka Stream.
1.11 the multi-tenant
Also known as multiple lease technology, it is a software architecture technology that enables multiple users to share the same system or program components while still ensuring data isolation between users.
RabbitMQ can support multi-tenant technology, with each tenant represented as a vhost, which is essentially a small independent RabbitMQ server with its own queues, switches, bindings, etc., and its own privileges. A Vhost is like a virtual machine in a physical machine. It provides logical separation between instances, allows data to be stored securely for different programs, separates many clients from each other in RabbitMQ, and avoids naming conflicts such as queues and switches.
1.12 Support for multiple protocols
Messages are carriers of information. In order for both producers and consumers to understand the information they carry (producers need to know how to construct messages and consumers need to know how to parse messages), they need to describe messages in a unified format, which is called message protocol. A valid message must have some form, and a message without a form is meaningless. General message-level protocols include AMQP, MQTT, STOMP, XMPP, etc. (JMS in the messaging domain is more of a specification than a protocol). The more supported protocols, the wider their application scope and versatility will be.
RabbitMQ’s ability to support THE MQTT protocol, for example, gives it a place in iot applications. There are also message-oriented middleware that run on their own proprietary protocols, such as Kafka.
1.13 Cross-language support
For many companies, there are multiple programming languages in the technology stack, such as C/C++, Java, Go, PHP, etc. Message middleware itself has the feature of application decoupling, and if it can further support multiple client languages, then the efficiency of this feature can be expanded. The level of support across languages can also be a proxy for the popularity of a message-oriented middleware.
1.14 Flow control
Flow control aims at the problem of speed mismatch between sender and receiver, and provides a speed matching service to suppress the sending rate so that the reading rate of the receiver’s application program can adapt to it.
Common flow control methods include stop-and-wait, sliding Windows, and token buckets.
1.15 Message ordering
As the name implies, message sequentiality refers to ensuring that messages are ordered.
A very common application scenario of this function is CDC (Change Data Chapture). For example, if the order of the binlog transmitted by MySQL is wrong, for example, the original sequence of adding 1 to a piece of Data and then multiplying by 2, it is changed to multiplying by 2 and then multiplying by 1 after sending the wrong sequence, resulting in Data inconsistency.
1.16 Security Mechanism
Two security mechanisms, authentication and permission control, have been added since Kafka 0.9. Identity authentication refers to the authentication between clients and servers, including the authentication between clients and brokers, between brokers, and between brokers and ZooKeeper. SSL and SASL authentication mechanisms are supported. Permission control refers to the permission control on read and write operations of clients, including the permission control on messages or Kafka cluster operations. Permission control is pluggable and supports integration with external authorization services. RabbitMQ also provides security mechanisms for authentication (TLS/SSL, SASL) and permission control (read and write operations).
1.17 Message idempotency
There are generally three delivery guarantees to ensure that messages are transmitted between producers and consumers:
A message may be lost, but it is never transmitted again.
Messages are never lost, but may be repeated At least once.
Every message will be transmitted Exactly once and only once.
For most message-oriented middleware, only At most once and At least once transport guarantees are provided. For the third guarantee, it is difficult to guarantee message idempotency.
Since version 0.11 Kafka has introduced idempotence and transactions. The idempotence of Kafka refers to the idempotence of a single producer for a single partition and a single session. Transactions can be written atomically to multiple partitions, that is, messages written to multiple partitions are either all successful or all rolled back. Together, these two features make Kafka EOS (Exactly Once Semantic) capable.
However, if we want to consider the global idempotent, we also need to consider the upstream and downstream aspects comprehensively, that is, the related business level. Idempotent processing itself is also an important issue that needs to be considered at the business level. Taking the downstream consumer level as an example, it is possible for a consumer to consume a message and then fail to acknowledge the message and then have to consume the original message again after recovery. This type of message idempotent cannot be guaranteed by the messaging middleware level.
If global idempotent is to be guaranteed, more external resources need to be introduced, such as unique identifiers with serial numbers and a de-duplication table downstream.
1.18 Transactional messages
Transaction itself is a familiar term. A Transaction consists of all operations performed between a Begin Transaction and an End Transaction. There are a number of messaging middleware middleware that support transactions, both Kafka and RabbitMQ, but a transaction is a transaction in which a producer sends a message, either successfully or unsuccessfully. Message-oriented middleware can be used as a means to implement distributed transactions, but it does not provide the capability of globally distributed transactions.
2. The performance
Functional dimension is an important reference dimension in the selection of message-oriented middleware, but it is not the only one. Sometimes performance is more important than functionality, and performance and functionality are often incompatible. Kafka can degrade performance when it comes to idemidemics or transactions. RabbitMQ can also degrade performance when it comes to rabbitmq_tracing.
The performance of messaging-oriented middleware generally refers to its throughput. Although RabbitMQ has an advantage over Kafka in terms of functionality, Kafka’s throughput is 1-2 orders of magnitude higher than RabbitMQ’s. Kafka’s stand-alone QPS can be maintained at 100,000 levels or even up to a million.
Throughput of message-oriented middleware is always limited at the hardware level.
Take the network adapter bandwidth as an example. If the bandwidth of a single network adapter is 1Gbps, the message body size should not exceed (1Gb/8)/100W, which is equal to about 134B. In other words, if the message body size exceeds 134B, it is impossible to reach the million level of throughput. This calculation can also be applied to memory and disk.
As an important indicator of performance dimension, delay is often ignored in the field of message-oriented middleware, because generally the scenarios using message-oriented middleware do not have high requirements on timeliness. If timeliness is required, RPC can be completely used to achieve it.
Message-oriented middleware has the ability of message stacking. The larger the message stacking is, the longer the end-to-end delay is. Meanwhile, delay queue is also a feature of some message-oriented middleware.
So why worry about latency in message-oriented middleware? Message middleware can decoupling system, for a time delay lower message middleware, it can make the upstream producers after sending the message can be quickly return, also can let the consumer more quick access to news, in the absence of accumulation can let whole application level between the upstream and downstream linkage as a more efficient, Although it is not recommended to use message-oriented middleware in time-sensitive scenarios, if the use of message-oriented middleware is good in terms of latency, the performance of the overall system will be greatly improved.
3. Reliability and availability
Message loss is a common problem when using message middleware, and message reliability behind it is also a key factor to measure the quality of message middleware.
Especially in the field of financial payment, information reliability is particularly important.
However, when it comes to reliability, it must be about availability. Note the difference between the two. Reliability of message-oriented middleware refers to the guarantee degree of message loss. The availability of message-oriented middleware is the percentage of time spent without failure, usually measured in nines.
In a narrow sense, distributed system architecture is the application implementation of the consistency protocol theory, which can also be traced back to the consistency protocol behind messaging middleware for message reliability and availability.
Kafka uses a pacifica-like consistency protocol. It uses in-sync-Replica (ISR) to ensure synchronization between multiple replicas and supports strong consistency semantics (implemented through acks).
RabbitMQ implements multiple copies and strong consistency semantics through mirrored ring queues. Multiple replicas ensure that the slave can be promoted as the new master to continue to provide services to ensure availability if the master node is down and abnormal.
Kafka was originally designed for log processing, leaving a bad impression that data reliability was not required, but as versions have been upgraded and optimized, reliability has greatly improved.
RabbitMQ is currently the most popular financial payment service, while Kafka is the most popular log processing service, big data service, etc. With RabbitMQ’s performance improving and Kafka’s reliability increasing, we believe that both of us will be able to get a share of the pie.
Synchronous flush is an effective way to enhance the reliability of a component, and messaging middleware is no exception. Both Kafka and RabbitMQ support synchronous flush.
Another aspect to mention here is extensibility, which I narrow down to the dimension of availability. Extensibility of messaging middleware increases its availability and scope. For example, RabbitMQ supports multiple messaging protocols, and this is an extensibility implementation based on its plug-in. In terms of cluster deployment, thanks to Kafka’s horizontal scaling capability, it can basically achieve a linear capacity increase level. In the introduction to LinkedIn practice, Kafka clusters with more than a thousand devices are mentioned.
4. Operation and maintenance management
In the process of using message-oriented middleware, there will inevitably be a variety of abnormal situations, both client side and server side, so how to timely and effective monitoring and repair? Business line traffic has peaks and valleys, especially in e-commerce, so how to conduct effective capacity assessment before, especially during the rush period? Kicks the power supply, the network cable is dug up and other events emerge in endlessly, how to effectively do the remote live? All of these are inseparable from the derivative of message-oriented middleware – operation and maintenance management.
O&m management can be further divided into application, audit, monitoring, alarm, management, disaster recovery, and deployment.
Application and audit are well understood. Resource management and control at the source can not only effectively correct the usage specifications of the application party, but also perform traffic statistics and traffic assessment through allocation and monitoring. Generally, application and audit are highly integrated with the internal system of the company, so it is not suitable for the use of open source products.
Monitoring and alarm are also easy to understand. Comprehensive monitoring of the use of message-oriented middleware can provide benchmark data for the system and coordinate with alarms when abnormal conditions are detected, so as to facilitate the rapid intervention of operation and maintenance and developers. In addition to general monitoring items (such as hardware, GC, etc.), message-oriented middleware also needs to pay attention to end-to-end delay, message auditing, message stacking, etc.
Management tools are indispensable to capacity expansion, downgrade, version upgrade, cluster node deployment, and troubleshooting. A complete set of management tools can achieve twice the result with half the effort when encountering changes.
Faults can be large or small, such as some application exceptions, machine power failure, network exceptions, disk damage and other single-machine faults. Multiple copies in a single-machine room can cope with these faults. If the equipment room fault involves remote Dr, the key point is how to effectively replicate data.
5. Community strength and ecological development
For the current popular programming languages, such as Java and Python, if you encounter some exceptions in the process of use, you can basically solve them with the help of search engines, because the more people use a product, the more pits it has stepped on, and the more corresponding solutions it has.
The same applies to message-oriented middleware. If you choose a “less popular” type of message-oriented middleware, you may be able to use it well in some aspects, but you may be stuck with slow version updates, difficult issues and difficult community support. On the other hand, if you choose a “popular” message-oriented middleware, it can be updated quickly to make up for the shortcomings of the past, but also to adapt to the rapid development of technology to change some new features, which can put you “on the shoulders of giants.”
In terms of operations management we mentioned that Both Kafka and RabbitMQ have a number of open source monitoring and management products, which are due to the rapid growth of their communities and ecosystems.
Example: RocketMQ and Kafka selection
RocketMQ features: Support for delayed delivery, delayed consumption, broadcast consumption, message traces, retry queues, dead letter queues, etc. Kafka does not support these features.
Significant features of Kafka: extensive use of big data components, extensive use of streaming components, high throughput, and millions of TPS common to a cluster that RocketMQ cannot achieve.
RocketMQ scenarios include payment, order, and service notification.
Kafka scenarios include log burying and big data processing.