concept

Message Queuing Protocol

Compare three of the most common and popular TCP/ IP-based messaging protocols and provide a quick summary of each of their advantages: AMQP, MQTT, and STOMP, all of which are supported in RabbitMQ 3.0 – we’ll use these as examples and come back to them later.

AMQP

AMQP- Advanced Message Queuing Protocol, intended as an open alternative to existing proprietary messaging middleware. The two most important reasons to use AMQP are reliability and interoperability. As the name implies, it provides a variety of message-related features, including reliable queuing, subject-based publish and subscribe messaging, flexible routing, transactions, and security. AMQP directly swaps routing messages by topic and based on title in the form of fan out.

Such a rich feature set allows for a lot of fine-grained control. You can restrict access to queues, manage their depth, and so on. Features such as message properties, comments, and captions make it ideal for a variety of enterprise applications. The protocol is designed to improve the reliability of many large companies that rely on messaging to integrate applications and move data across their organizations. In the case of RabbitMQ, there are many different language implementations and a large sample available, making it a good choice for building a large, reliable, resilient or clustered messaging infrastructure.

AMQP is a binary wired protocol designed to enable interoperability between different vendors. The adoption rate of AMQP is high when other protocols fail. Companies like JPMorgan use it to process a billion messages a day. NASA uses it for nebula cloud computing. Google uses it for complex event handling. Here are some other AMQP examples and links:

  • It is used in one of the world’s largest biometric databases – India’s Aadhar Project – with 1.2 billion identities.
  • It is used on the Ocean Observatories Initiative, a structure that collects eight terabytes of data per day.
  • Amqp.org has more examples and links.

MQTT

MQTT (message queue telemetry transmission) was originally developed by the IBM pervasive computing team, in collaboration with industrial partners. Over the past few years, the protocol has been transferred to the open source community, and with the beginning of mobile applications, its popularity has increased significantly, and it is making its way into the hands of standards groups

The aggregation principles and objectives of MQTT are simpler and more centralized than AMQP – it provides publish and subscribe messages (without queues), and is specifically targeted at resource-constrained devices and low bandwidth, high latency networks, such as dial-up lines and satellite links. Basically, it can be used effectively in embedded systems.

One of MQTT’s advantages over more powerful “enterprise messaging” agents is that its deliberately low footprint makes it an ideal choice for mobile and developing “Internet of Things” style applications today. In fact, companies like Facebook are using it as part of their mobile applications because it has such low power consumption and small network bandwidth.

Some MQTT-based agents support thousands of concurrent device connections. It offers three qualities of service: 1) fire and forget/unreliable, 2) “at least once” to ensure it is sent at least once (but may be sent more than once), and 3) “exactly once”.

The advantages of MQTT are simplicity (only five API methods), a compact binary packet payload (no message attributes, compressed headers, and more concise than text-based HTTP), and it is well suited for simple push messaging schemes such as temperature updates, stock price codes, oil pressure feeds, or movement notifications. It is useful for connecting machines together, such as connecting Arduino devices to Web services using MQTT.

STOMP

STOMP (Simple/Streaming Textual Oriented Messaging Protocol) is the only text-based protocol of the three, making it more similar to HTTP in terms of coverage. Like AMQP, STOMP provides a message (or frame) header or frame body with attributes. The design principle here is to create something simple and widely interoperable. For example, you can use something as simple as the Telnet client to connect to the STOMP agent.

However, STOMP does not handle queues and topics. It uses the SEND semantics with a “target” string. The broker must map to content understood, such as topics, queues, or exchanges. The consumer then subscribes to those destinations. Since these destinations are not mandated by the specification, different agents may support different destination styles. Therefore, porting code between agents is not always straightforward.

However, STOMP is simple and lightweight (despite the online benefits of verbosity), with a language binding for guanfan. It also provides some transaction semantics. One of the most interesting examples is the RabbitMQ Web Stomp, which allows you to expose messages in the browser via websockets. This opens up some interesting possibilities – such as using all types of information to update browsers, mobile applications or machines in real time.

2. Message queues

What is a message queue?

Message Queuing is an asynchronous means of communication between services, using both no-service and micro-service architectures. Messages are stored on the queue until they are processed and deleted. Each message can only be processed once at all for the user. Message queues can be used to separate heavyweight processing, caching, or batch work, and to alleviate peak workloads.

In modern cloud architectures, applications are broken down into separate building blocks that are smaller and easier to develop, deploy, and maintain. Message Queuing provides communication and coordination for these distributed applications. Message queues can significantly simplify the encoding of decoupled applications while improving performance, reliability, and scalability.

With message queues, different parts of the system can communicate with each other and perform processing operations asynchronously. Message queues provide a lightweight cache for temporarily storing messages and allow software components to connect to the queue to send an end node that receives the message. These messages are usually small and can be requests, recovery, error messages, or plaintext messages, etc. To send a message, a component called the “creator” adds the message to the queue. The message is stored in a queue until another component named “processor” retrieves the message and does something about it.

Many creators and processors can use queues, but a message can only be processed once by a box of processors. Therefore, this messaging mode is often referred to as one-to-one or point-to-point communication. If the message needs to be processed by multiple processors, the fan-out design pattern can be used to combine message queuing with publish/subscribe message transceiver.

3. Messaging middleware

  • RabbitMQ
  • Kafka
  • RocketMQ
  • Qpid
  • Artemis
  • NSQ
  • ZeroMQ

ActiveMQ

ActiveMQ is a JMS Provider implementation that fully supports the JMS1.1 and JMS1.4 specifications. It’s fast, just multilanguage clients and protocols, can be easily embedded in enterprise applications, and has many advanced features.

The main features

  • Compliance to the JMS specification: The JMS specification provides good standards and guarantees, including: synchronous or asynchronous message distribution, one-time or only one-time message distribution, message receiving and subscription, and so on. The benefit of conforming to the JMS specification is that these basic features are available no matter what JMS implementation provider is used;
  • Connection flexibility: ActiveMQ provides a wide range of connection protocols, such as HTTP/S, IP multicast, SSL, TCP, UDP, and so on. Support for many protocols gives ActiveMQ great flexibility
  • Supports many kinds of protocols: OpenWrite, STOMP, REST, XMMP, AMQP;
  • Persistence plug-ins and security plug-ins: ActiveMQ provides a variety of persistence options. Moreover, the security of ActiveMQ can also be customized and authorized according to user requirements.
  • Supported client language variety: in addition to Java, there are C/C++,.NET, Perl, PHP, Python, Ruby;
  • Agent Cluster: Multiple ActiveMQ agents can form a cluster to provide services;
  • Unusual simplicity of administration: ActiveMQ was designed with the developer in mind. Therefore, it does not require a dedicated administrator, because it provides simple and useful administrative features. There are many ways to monitor data at different levels of ActiveMQ, including using JMXJMX in JConsole or in ActiveMQ’s Web Console. By handling JMX alarm messages, by using command-line scripts, and even by monitoring various types of logs.

The deployment environment

ActiveMQ can run on platforms supported by the Java language. Using ActiveMQ requires:

  • Java JDK
  • ActiveMQ installation package

advantages

  • Cross-platform (Java writing is platform-independent, ActiveMQ can run on almost any JVM);
  • You can use JDBC: You can persist data to a database. Although using JDBC can degrade ActiveMQ performance, the database has always been the storage medium developers are most familiar with;
  • Support for JMS specification: Support for unified interfaces provided by the JMS specification;
  • Support for automatic reconnect and error retry mechanisms;
  • Security mechanism: Support various security configuration mechanisms based on Shiro, JAAS, etc., Queue/Topic can be authenticated and authorized;
  • Monitoring is complete: Has complete monitoring, including Web Console, JMX, Shell command line, Jolokia’s RESTful API;
  • Friendly interface: The Web Console is available for most situations, and there are many third-party components available, such as Hawtio;

disadvantages

  • Community activity is not as high as RabbitMQ;
  • Based on other users’ feedback, there will be inexplicable problems and lost messages;
  • At present, the focus is placed on Apollo, ActiveMQ 6.0 product, and the maintenance of 5.x is less.
  • It is not suitable for thousands of queues.

RabbitMQ

RabbitMQ, released in 2007, is a reusable enterprise messaging system based on AMQP (Advanced Message Queuing Protocol). It is one of the most mainstream messaging middleware.

The main features

  • Reliability: provides a variety of techniques that allow you to trade off performance against reliability. These technologies include persistence mechanisms, delivery verification, publisher verification, and high availability mechanisms;
  • Flexible routing: Messages are routed via switches before they reach the queue. RabbitMQ provides several built-in switch types for typical routing logic. If you have more complex routing needs, you can combine these switches. You can even implement your own switch type and use it as a RabbitMQ plugin.
  • Message Cluster: Multiple RabbitMQ servers on the same LAN can be aggregated together and used as a single logical proxy.
  • High availability of queues: Queues can be mirrored on machines in the cluster to ensure message security in case of hardware problems;
  • Support for multiple protocols: Support for multiple message queue protocols;
  • Multi-language support: written in Erlang, support for almost any programming language you can think of;
  • Administrative interface: RabbitMQ has an easy-to-use user interface that enables users to monitor and manage many aspects of the message Broker;
  • Tracing mechanism: If a message is abnormal, RabbitMQ provides a message tracking mechanism so that the consumer can find out what happened.
  • Plug-in mechanism: Many plug-ins are provided to extend in many ways, or you can write your own plug-ins.

The deployment environment

RabbitMQ runs on platforms supported by Erlang, including Solaris, BSD, Linux, MacOSX, Tru64, Windows, and more. RabbitMQ requires:

  • ErLang language pack
  • The RabbitMQ installation package

advantages

  • Due to the nature of the Erlang language, message queuing performs well and supports high concurrency.
  • Robust, stable, easy to use, cross-platform, multi-language support, complete documentation;
  • Message confirmation mechanism and persistence mechanism, high reliability;
  • Highly customizable routing;
  • The management interface is rich, and it is also widely used in Internet companies, with high community activity.

disadvantages

  • Although combined with the concurrency advantage of Erlang language itself, the performance is better, but not conducive to secondary development and maintenance;
  • The broker architecture is implemented, meaning that messages can be queued on a central node before being sent to the client. This feature makes RabbitMQ easy to use and deploy, but makes it slower to run due to the added latency of the central node and larger message encapsulation;
  • Need to learn more complex interface protocol, learning and maintenance costs are high.

RocketMQ

RocketMQ is an open source product from Ali and is implemented in Java language. It is designed with reference to Kafka and has made some improvements of its own. The message reliability is better than that of Kafka. RocketMQ is widely used within Alibaba in order, transaction, recharge, stream calculation, message push, log streaming, binglog distribution and other scenarios.

The main features

  • Queue based model: with high performance, high reliability, high real-time, distributed and other characteristics;
  • Producer, Consumer and Queue all support distributed production;
  • Producers send messages in turn to several queues, a collection of queues called topics. If a Consumer makes a broadcast consumption, a Consumer instance will consume all queues associated with the Topic. If you do cluster consumption, multiple Consumer instances will average the collection of queues associated with the Topic.
  • Can guarantee strict message order;
  • Provides a rich message pull pattern;
  • Efficient subscription to horizontal expansion capability;
  • Real-time message subscription mechanism;
  • Billion level message accumulation capacity;
  • Less external dependence;
The deployment environment

RocketMQ runs on platforms supported by the Java language. Using RocketMQ requires:

  • Java JDK
  • Install Git and Maven
  • RocketMQ installation package

advantages

  • Support more than 10,000 persistent queues on a single machine;
  • RocketMQ all messages are persistent, first written to the system PageCache, then brush disk, can ensure that both memory and disk have a copy of the data, and access, directly read from memory.
  • The model is simple and the interface is easy to use (the interface of JMS is not practical in many cases).
  • The performance is very good, allowing a large number of messages to pile up in the Broker;
  • Support a variety of consumption patterns, including cluster consumption, broadcast consumption, etc.;
  • The distributed extension design of each link supports master-slave and high availability;
  • The development degree is more active, the version is updated faster;

disadvantages

  • There are not many supported client languages, currently Java and C++, among which C++ is not mature;
  • RocketMQ also lacks the attention and maturity of the first two;
  • There is no Web administration interface, and a CLI administration tool agent queries, manages, and diagnoses various problems;
  • Interfaces such as JMS are not implemented in the MQ core

Kafka

Apache Kafka is a distributed message publish and subscribe system. It was originally implemented by LinkedIn as a distributed commit log system based on a unique design, and later became part of the Apache project. Kafka is efficient, scalable, and persistent. Its partitioning, replicability, and fault tolerance are all good features.

The main features

  • Fast Persistence: Message persistence can be performed at O(1) overhead;
  • High throughput: 10W/s throughput can be achieved on an ordinary server;
  • Completely distributed system: Broker, Producer and Consumer all automatically support distributed distribution and automatically realize load balancing;
  • Support synchronous and asynchronous replication two high availability mechanisms;
  • Support data batch sending and fetching;
  • Zero-copy technology: reduce IO operation steps, improve system throughput;
  • Data migration and expansion are transparent to users;
  • The machine can be expanded without stopping;
  • Other features: Rich message pull model, efficient subscriber level scaling, real-time message subscription, billion level message stack capability, periodic deletion mechanism;

The deployment environment

Using Kafka requires:

  • Java JDK
  • Kafka installation package

advantages

  • The client language is rich: support Java,.NET, PHP, Ruby, Python, Go and other languages;
  • High performance: stand-alone write TPS is about 1 million pieces/s, message size is 10 bytes;
  • It provides a completely distributed architecture and Replica mechanism, which has high availability and reliability and theoretically supports infinite accumulation of messages.
  • Support batch operation;
  • The consumer uses Pull to get the message. Message order, through control to ensure that all messages are consumed and consumed only once;
  • Excellent third party Kafka Web management interface Kafka-Manager;
  • It is mature in the logging field and is used by multiple companies and multiple open source projects.

disadvantages

  • When Kafka has more than 64 queues/partitions on a single machine, there will be a significant spike in Load. The more queues, the higher the load and the longer the response time of sending messages;
  • In short polling mode, the real-time depends on the polling interval time.
  • [Fixed] Failure to consume does not support retry;
  • Message ordering is supported, but messages can be out of order when an agent goes down.
  • Community renewal is slow.

4. Publish/subscribe messaging

Publish/subscribe messaging is an asynchronous mode of communication between services that is suitable for both no-service and micro-service architectures. In the publish/subscribe mode, any message posted to a topic is immediately accepted by all subscribers to the topic. Publish/subscribe messaging can be used to enable event-driven architectures or to separate applications to improve performance, reliability, and scalability.

The basics of publish/subscribe messaging

In modern cloud architectures, applications are broken down into separate building blocks that are smaller and easy to develop, deploy, and maintain. The publish/subscribe messaging provides timely event notification for these distributed applications.

The publish/subscribe pattern enables messages to be broadcast asynchronously to different parts of the system. Message topics, similar to message queues, can provide a lightweight mechanism for broadcasting asynchronous event notifications, and can also provide terminal nodes that allow software components to connect to topics to send and receive messages. When a message is broadcast, a component called the “publisher” pushes the message to the topic. Unlike message queues, where messages are processed in batches before they are retrieved, message subjects transmit messages without or with minimal message queues and push messages to all subscribers immediately. All components that subscribe to this topic receive every message that is broadcast, unless the subscriber sets a message filtering policy.

Subscribers to a message topic usually perform different functions and can perform different operations on the message simultaneously. Publishers do not need to know who is using the broadcast message and subscribe to it without knowing where the message is coming from. This transceiver pattern is slightly different from message queues, where the component sending a message usually knows where it is being sent.

Technology selection

Not surprisingly, given the impact of basic messaging on data-driven applications, there are a number of technologies that support some form of message queuing, publish and subscribe messaging, or both.

Technologies such as Apache ActiveMQ, Amazon SQS, IBM WebSphere MQ, RabbitMQ, and RocketMQ were originally designed primarily as use cases for message queuing. Other technologies, such as Apache Kafka and Google Cloud Pub/Sub, are primarily used to support publish-subscribe use cases. Other often newer solutions, such as Apache Pulsar, support message queues and pub-sub messaging.

RabbitMQ, Kafka, and ActiveMQ are messaging technologies used to provide asynchronous communication and decoupling processes (separating senders and receivers of messages). They are referred to as message queues, message brokers, or messaging tools. RabbitMQ, Kafka, and ActiveMQ all share the same basic purpose. But work can be done in different ways. Kafka is always a high throughput distributed messaging system. RabbitMQ is a reliable messaging broker based on AMQP. Both ActiveMQ and Kafka are Apache products and are written in Java; RabbitMQ is written in Erlang.

Kafka is based on distributed architecture, RabbitMQ is implemented based on AMQP protocol, RocketMQ is based on Kafka, changed to a master-slave structure, which is optimized in terms of transactional and reliability. Generally speaking, e-commerce, finance, etc., which have high requirements for transaction consistency, can consider RabbitMQ and RocketMQ, and Kafka can be considered for those with high requirements for performance.

https://stackshare.io/stackup…