We are pleased to announce that StreamNative has opened source “MoP” (MQTT on Pulsar). MoP introduces the MQTT protocol processing plug-in into the Pulsar Broker. As a result, Apache Pulsar can support the native MQTT protocol.
Like KoP, MoP is a pluggable protocol handling plug-in. By adding the MoP protocol processing plug-in to an existing Pulsar cluster, users can migrate existing MQTT applications and services to Pulsar without modifying the code.
This allows MQTT applications to take advantage of Features of Pulsar, such as The Apache Pulsar architecture that separates computing and storage, and features such as Apache BookKeeper that keeps event streams and Pulsar layered storage.
What is Apache Pulsar
Apache Pulsar is a cloud-native distributed messaging and streaming data platform that manages hundreds of billions of events every day. Pulsar was originally developed by Yahoo, opened source in late 2016, and became an Apache Software Foundation top project in 2018. Pulsar connects important Yahoo applications, such as Yahoo Finance, Yahoo Mail, and Flickr, to an integrated messaging platform for data.
Pulsar is a multi-tenant, high-performance solution for passing messages between servers. Pulsar supports the following key features:
- A single instance of Pulsar natively supports multiple clusters, enabling seamless replication of messages across clusters across geographies
- Extremely low release latency and end-to-end latency
- Support extends to millions of topics
- Message delivery is guaranteed through the persistent message storage mechanism provided by Apache BookKeeper
Apache Pulsar has been widely used in many fields. Tencent, Verizon Media, Splunk, China Mobile, BIGO, and others are all using Pulsar to achieve business goals.
For more user stories, see streamnative. IO /blog
What is the MQTT
Message Queuing Telemetry Transport (MQTT) is a lightweight communication protocol based on the Pubish /subscribe mode.
Built on top of TCP/IP, the protocol was released by IBM in 1999. The greatest advantage of MQTT is that it provides real-time and reliable messaging services for connecting remote devices with very little code and limited bandwidth. As a low-overhead, low-bandwidth im protocol, MQTT is widely used in the Internet of Things, small devices, mobile applications and other aspects.
Why do we need MoP
Apache Pulsar provides a unified messaging model for queue and flow workloads. Apache Pulsar supports a protobuf-based binary protocol to ensure high performance and low latency. Protobuf is good for implementing the Pulsar client. Pulsar.apache.org/docs/en/cli…
In addition, the project also supports Java, Go, Python, and C ++ languages, as well as third-party clients provided by the community. Pulsar.apache.org/docs/en/cli…
Pulsar supports multi-tenancy and builds persistence mechanisms based on Apache BookKeeper. As a result, more and more companies are exploring using Pulsar to build their underlying services and transform their business architecture. However, to adopt Pulsar’s unified messaging protocol, users must rewrite existing applications written using other messaging protocols.
To address this, StreamNative has been working on new projects. This year, StreamNative opened source ➡️KoP (Kafka- on-pulsar) and ➡️AoP (AMQP-on-PulSAR) protocol processing plug-ins to facilitate migration of applications and services that use Kafka and AMQP protocols to Pulsar.
- The KoP is a pluggable protocol processing plug-in. By adding the KoP protocol processing plug-in to an existing Pulsar cluster, users can migrate existing Kafka applications and services to Pulsar without changing the code
- AoP is a pluggable plug-in for protocol processing. By adding the AoP protocol processing plug-in to an existing Pulsar cluster, users can migrate existing RabbitMQ applications and services to Pulsar without changing the code
StreamNative received a large number of requests from users to help them migrate from MQTT to Pulsar. At the same time, StreamNative recognized the need to natively support the MQTT message transfer protocol on Pulsar. As a result, StreamNative began working on bringing a generic protocol processing plug-in framework into Pulsar. The framework allows developers using the MQTT transport protocol to use Pulsar.
The MoP architecture
MoP is a pluggable protocol handling plug-in. The native MQTT protocol is supported on Pulsar by using the Topic, Cursor and other features of Pulsar.
The following figure shows the combination of the MoP protocol processing plug-in and the Pulsar cluster. The MQTT Proxy service and the MQTT protocol processing plug-in both run with the Pulsar Broker.
The MoP concept
Quality of service of the message
To meet the requirements of different scenarios, MQTT supports the following three QoS levels:
- QoS0: Send messages at most once, or not at all when network transmission is blocked. Messages sent are not saved.
- QoS1: Sends a message at least once. If the sender does not receive the acknowledgement packet, a DUP message is sent again until the sender receives the acknowledgement packet.
- QoS2: The message is successfully sent only once. Messages must be stored in the local environment of the sender and receiver until they are properly processed. This QoS level Indicates the highest message service level.
Currently, the MoP protocol processing plug-in supports only QoS0 and QoS1 levels of message quality of service. QoS2 support is planned for future releases.
MoP Proxy
In MoP, MoP Proxy is an optional component that is primarily used to Proxy MoP services. The MoP Proxy supports scaling the MoP to multiple nodes to achieve scale-out services. The MoP Proxy is used to correctly forward the message data passed between the MQTT Client and the Pulsar Broker, so the MQTT Client only needs to connect to the MoP Proxy to send and receive data. There is no need to pay attention to the Topic’s Pulsar Broker.
The MoP Proxy can sense changes in the Pulsar Broker to which a Topic belongs. If the Pulsar Broker changes, the MoP Proxy can send the MQTT Client’s network packets to the new Pulsar Broker.
The following figure illustrates the service process for the MoP Proxy.
- The MQTT client establishes a connection to the MoP Proxy.
- The MoP Proxy sends a lookup request to the Pulsar cluster to determine the URL of the owner broker for the current Topic.
- The Pulsar cluster returns the URL of the Owner Broker to the MoP Proxy.
- The MoP Proxy establishes a connection to the Topic’s owner broker and starts transferring data between the MQTT client and the Topic’s owner broker.
Currently, the MoP Proxy runs as a plug-in with the Pulsar Broker. You can modify the configuration to enable Proxy.
For more information, please refer to github.com/streamnativ…
Start using a MoP
MoP is an open source project using the Apache License V2.
Download the latest release of the MoP Protocol handling plug-in and start using the MoP Protocol handling plug-in. Github.com/streamnativ…
For information on how to handle plug-ins using the MoP protocol, refer to the documentation: github.com/streamnativ…
If you encounter any problems in use, you can submit an issue in the MoP repository and we will respond as soon as possible. You are also welcome to contribute features to MoP. Github.com/streamnativ…