This article is published on my official account “Programmer Cxuan”. Welcome to pay attention to it
So said and done!
A reader left me a message saying that he wanted to know what the MQTT protocol is, and complimented me by the way.
So the requirements of readers must be met ah, so now @ the little sister, to listen to the class!
What is the MQTT protocol
MQTT is a Message Queuing Telemetry Transport protocol based on the PUBLish-subscribe mode under the ISO standard and based on the TCP/IP protocol cluster. It is designed to improve the performance of network equipment hardware and network performance. MQTT is widely used in IoT (Internet of Things) applications such as automotive, manufacturing, oil, and gas.
After understanding the concept of MQTT and application scenarios, we will come down to walk into the STUDY of MQTT, first take a look at what MQTT concepts.
MQTT basis
The MQTT protocol is a lightweight binary protocol. MQTT has an obvious advantage over HTTP: it has a lower packet overhead, which means it is easier to travel over the network. Another advantage is that MQTT is easy to implement on the client side and is easy to use, making it ideal for today’s resource-limited devices.
You may be a little secretive about these concepts, why XXX? This starts with the design of MQTT.
The MQTT protocol was invented in 1999 by Andy Stanford-Clark (IBM) and Arlen Nipper (Arcom, now Cirrus Link). They need an agreement to connect oil pipelines via satellite to minimize battery loss and bandwidth. So they set out several requirements for this agreement:
- The protocol must be easy to implement;
- The data in this protocol must be easy to transmit and the consumption cost is small;
- This agreement must provide quality of service management;
- This protocol must support continuous session control
- Assume that the data is unknowable, and do not insist on the type and format of the data to be transmitted, to maintain flexibility.
These designs are at the heart of MQTT, which has evolved to become a necessary message detection protocol for the Internet of Things (IoT). The official recommended version is MQTT 5.
Publish and subscribe
The publish-subscribe pattern, which I’m sure those of you who are exposed to message-oriented middleware architectures have heard of, is an alternative to the traditional client-server architecture where the client communicates directly with the server.
Publishing-subscription mode pub/sub is different. Publishing-subscription mode separates publisher (which sends messages) from subscribers (which receive messages). Publisher and subscribers do not communicate directly. They don’t even know if each other exists, and the communication between them is mediated by a third party component, the Broker.
The most important aspect of pub/sub is the uncoupling of Publisher and subscriber. This coupling degree has the following three dimensions:
- Spatial decoupling: Publisher and Subscriber do not know the existence of each other, for example, there is no IP address and port interaction, let alone message interaction.
- Time solution: Publisher and Subscriber do not have to run at the same time.
- synchronous
SynchronizationUncoupling: Both component operations such as publish and subscribe do not interrupt the publish or subscribe process.
In short, the publish/subscribe model eliminates the traditional direct client-server communication, leaving it to the broker to broker and understand the spatial, temporal, and synchronous dimensions.
Can expand sex
Pub/SUB is a better extension of the traditional client-server pattern because brokers are highly parallel and are based on an event-driven pattern. Scalability also includes message caching and intelligent routing of messages, as well as the ability to connect millions of connections through clustering proxies, using load balancers to distribute load across more single servers, and this is where MQTT goes deep.
In case you don’t understand what is event-driven, LET me explain the concept of event-driven here.
Event-driven is a programming paradigm, a programming paradigm is a concept in software engineering, it refers to a method of programming or a way of programming, for example object-oriented programming and procedural programming are a programming paradigm, The flow of a program in an event-driven context is determined by events such as user actions (mouse clicks, keyboard), sensor outputs, or message transmissions from other programs. Event-driven programming is the dominant paradigm used in graphical user interfaces and other applications, such as the Web, that center on performing certain actions in response to user input, and it also applies to driver programming.
The message filter
The broker plays a crucial role in the pub/ SUB architectural pattern, one of which is the ability to filter messages so that each subscriber only receives messages of interest to them.
The broker has several options to filter
- Subject-based filtering
MQTT is subject based message filtering. Each message has a topic. Receiving clients subscribe to the topics of interest to Borker, and after subscribing, the broker ensures that the client receives the messages published to the topics.
- Content-based filtering
In content-based filtering, the broker filters messages based on specific content, and the receiving client filters what they are interested in. One significant disadvantage of this approach is that the content of the message must be known in advance and cannot be encrypted or easily modified.
- Type-based filtering
Type filtering based on message (event) is a common filtering method when using object-oriented languages.
To meet the challenges of the publish/subscribe system, MQTT has three quality of service levels. You can specify whether messages are sent from the client to the broker or from the broker to the client. A broker must know how to handle this situation.
The difference between MQTT and message queue
As we now know, MQTT is a message queue transmission detection protocol that appears to be based on message queues, but differs from message queues.
In the traditional pattern of message queue, a message will be stored in the message queue waiting to be consumption, each incoming message is stored in the message queue, until it is client (often called consumer) received, if there is no client consumption message, the message will exist in the message queue waiting to be spending. However, in the message queue, there is no case where the message is not consumed by the client, but there is a case where there is no topic subscriber subscription in MQTT.
In the traditional message queue mode, a message can only be consumed by one client, and the load is distributed among each consumer in the queue. In MQTT, each subscriber receives a message, and each subscriber has the same load.
In the traditional message queue model, a separate command must be used to explicitly create a queue, and only after the queue is created can messages be produced or consumed. In MQTT, topics are flexible and can be created on the fly.
HiveMQ is now open source, and the HiveMQ Community edition implements the MQTT Broker specification and is compatible with MQTT 3.1, 3.1.1, and MQTT 5. The HiveMQ MQTT Client is a Java-based MQTT Client implementation that is compatible with MQTT 3.1.1 and MQTT 5. Both projects can be found on HiveMQ at github github.com/hivemq.
As we know, the broker separates Publisher and Subscriber, so the client connection is brokered by the broker, so before we can dive into MQTT, we need to know what client and broker mean.
Important MQTT concepts
MQTT client
When we talk about the concept of a Client, we generally refer to the MQTT Client, where Publisher and Subscriber are both MQTT clients. The concept of publisher and subscriber is actually a relative concept, which refers to whether the current Client is publishing or receiving messages, and the publishing and subscription functions can also be implemented by the same MQTT Client.
An MQTT client is any device that runs an MQTT library and connects to an MQTT Broker over a network, ranging from a microcontroller to a full-fledged server. Basically, any MQTT device that uses THE TCP/IP protocol is called an MQTT Client. The client implementation of the MQTT protocol is very simple and straightforward. Ease of implementation is one of the reasons MQTT is well suited for small devices. The MQTT client library is available for multiple programming languages. For example, Android, Arduino, C, C++, C#, Go, iOS, Java, JavaScript, and.NET.
MQTT broker
The counterpart of an MQTT client is an MQTT Broker, which is at the heart of any publish/subscribe institution and can handle up to millions of connected MQTT clients, depending on the implementation.
The broker receives all messages, filters them, determines which client subscribed to each message, and sends the message to the corresponding client. The broker also holds session data, including subscribed and missed messages. The broker is also responsible for client authentication and authorization.
MQTT Connection
MQTT is based on the TCP/IP protocol. Therefore, both the CLIENT and broker of MQTT need to support TCP/IP.
An MQTT connection is always between the client and the broker, not between the client and the client. To initiate a connection, the client sends a CONNECT message to the broker, which responds with a CONNACK message and status code. Once the connection between the client and broker is established, the broker keeps the client’s connection open until the client issues a disconnection command or the connection is interrupted.
Message message
MQTT messages are divided into CONNECT and CONNACK messages.
CONNECT
We mentioned above that the client needs to send a CONNECT message to the broker in order to initialize the connection. If the CONNECT message is in the wrong format or takes too long to open the Socket (because the Socket needs to be initialized based on the TCP/IP stack), Or if it takes too long to send a connection message, the broker closes the connection.
An MQTT client sends a CONNECT connection that may contain the following information:
Let me just explain what this information is. Okay
ClientId: Obviously, this is the ID of each client that connects to the MQTT Broker. This ID should be unique to each client and broker. If you do not need the broker to hold state, you can send an empty ClientId, which has no state. In this case,ClientSessionIt needs to be set to true, otherwise the connection will be rejected.
What is a clientSession and we’ll talk about that next.
CleanSession: CleanSession The session flag tells the broker client whether to establish a persistent session. In a persistent session (CleanSession = false), the broker stores all of the client’s subscriptions as wellQuality of Service (Qos)Is all the lost messages of the client to which 1 or 2 is subscribed. If the session is not persistent (CleanSession = true), then the broker stores nothing for the client and clears all information from the previously persistent session.Username/Password: MQTT sends the username and password for client authentication and authorization. If the message is not encrypted or hashed, the password will be sent as plain text. Therefore, it is strongly recommended that username and password be encrypted and securely transmitted. Brokers such as HiveMQ can authenticate with SSL certificates and therefore do not require user names and passwords.LastWillxxxLastWillxxx: LastWillxxx is a will that a client will set when it connects to the broker. This will be stored in the broker when the client uses theAbnormal causeWhen disconnected from the broker, the broker sends the will to the clients that subscribed to the topic.keepAlive: keepAlive is the interval, usually in seconds, between the client and the broker when the connection is established. This is the maximum time that the client and broker can endure without sending messages.
After talking about sending a CONNECT message between the client and broker to establish a connection, let’s talk about the CONNACK message that the broker needs to acknowledge the CONNECT.
CONNACK
When the broker receives a CONNECT message, it is obliged to respond with a CONNACK message. The CONNACK message has two parts
-
SessionPresent: Current session identifier that tells the client whether the current broker has a persistent session to interact with the client. The SessionPresent flag is related to the CleanSession flag. When a client connects with CleanSession set to true, the SessionPresent flag is always false because no persistent session is available. If CleanSession is set to false, then SessionPresent is true if ClientId session information is available and the broker already stores session information. Otherwise, if there is no session information for ClientId, then SessionPresent is false.
-
ReturnCode: The second flag in the CONNACK message is the connection acknowledgement flag. This flag contains a return code that tells the client whether the connection attempt was successful. The connection confirmation flag has the following options.
For detailed instructions on each connection, please refer to docs.oasis-open.org/mqtt/mqtt/v…
Message type
release
When the MQTT client is ready to send messages after connecting to the broker, MQTT uses topic-based filtering. Each message should contain a topic that the broker can use to send messages to interested clients. In addition, each message contains a Payload, which contains the data to be sent in bytes.
MQTT is data independent, meaning that it is up to the publish-Publisher to send XML, JSON, binary, or text data.
The structure of the PUBLISH message in MQTT is as follows.
Packet IdentifierThis PacketId identifies the unique message identifier between the client and broker. PacketId is only related to Qos levels greater than zero.TopicName: The subject name is a simple string,/It stands for hierarchical structure.Qos: This number represents the level of quality of service (QOS), which comes in three levels: 0, 1, and 2. The service level determines the guaranteed type of message that reaches the client or broker and determines whether the message is lost.RetainFlag: This flag indicates that the last retained flag bit received by the broker istrueMessages are stored on the server (either in memory or in a file).
The MQTT server will only save the last received message with the RETAIN flag bit true for each Topic. That is, if the MQTT server Retained a message for a Topic, the same message will be overwritten when the client publishes a new message.
PayloadThis is the actual content of each message. MQTT is data independent. You can send any text, image, encrypted data, and binary data.Dupflag: This flag indicates that the message was repeated and resended because the expected client or broker did not acknowledge it. This flag is only relevant to Qos greater than 0.
When a client sends a message to the broker, the broker reads the message, recognizes the message according to the Qos level, and then processes the message. Processing the message is essentially determining which subscriber subscribed to the topic and sending the message to them.
The client that initially publishes the message only cares about sending the PUBLISH message to the broker. Once the broker receives the PUBLISH message, the broker is responsible for sending it to all subscribers. The client that publishes the message does not know if anyone is interested in the published message, nor does it know how many clients have received messages from the broker.
To subscribe to
The client sends a SUBSCRIBE message to the broker to receive information about the topic it is interested in. The SUBSCRIBE message is simple. It contains a unique packet identifier and a subscription list.
Packet IdentifierThis PacketId is the same as the PacketId above, which represents the unique identifier of the message.ListOfSubscriptions: SUBSCRIBE messages can contain multiple subscriptions of a client. Each subscription is composed of a topic and a Qos. A topic in a subscription message can contain wildcards.
A confirmation message
After a client sends a SUBSCRIBE message to the broker, the broker sends a SUBACK acknowledgement message to the client to acknowledge each subscription. The SUBACK contains the packetId of the original SUBSCRIBE message and the return code list.
Among them
Packet Identifier: This packet identifier is the same as the one in SUBSCRIBE.ReturnCodeThe broker sends a return code for each topic/Qos pair of SUBSCRIBE messages received. For example, if the SUBSCRIBE message has five SUBSCRIBE messages, the SUBACK message contains five return codes as a response.
So far we’ve looked at three message types, publish – subscribe – confirm, and here’s a schematic of the three.
unsubscribe
A SUBSCRIBE message corresponds to an UNSUBSCRIBE message, and after this message is sent, the broker deletes the subscription to the client. So an UNSUBSCRIBE message is similar to a SUBSCRIBE message in that it has a packetId and a topic list.
Confirm the unsubscribe
Unsubscription is also acknowledged by the broker, which sends an UNSUBACK message to the client. The UNSUBACK message is simple and has only one packetId data identifier.
The procedure for unsubscribing and confirming unsubscribing is as follows.
When the client receives an UNSUBACK message from the broker, it considers the UNSUBSCRIBE message deleted.
Talk about the Topic
With all this talk about MQTT, we haven’t really talked about Topic yet. In MQTT, Topic is the UTF-8 string that the broker filters messages for each connected client. A Topic is a hierarchical structure that can be composed of one or more topics. Each Topic is divided by /.
Compared to traditional message queues, MQTT topics are very lightweight. The client does not need to create the required Topic before publishing or subscrienting, and the broker does not need to initialize each Topic before receiving it.
The wildcard
When a client subscribing to a Topic, it can subscribe to the exact Topic of the published message, or it can use wildcards to subscribe to multiple topics simultaneously. There are two types of wildcards: single-level and multi-level.
Single-level wildcard
Single-level wildcards can replace a level of a Topic, and the + sign represents a single-level wildcard in a Topic.
Any Topic can match a single level wildcard if it contains any string other than a wildcard. For example,
Myhome /groundfloor/+/temperature has the following matches.
Multi-level wildcards
Multilevel wildcards cover multiple topics, and # represents a multilevel wildcard in a Topic. In order for the broker to determine which topics to match, the multi-level wildcard must be placed as the last character in the Topic and begin with a /.
Here are some examples of myHome /groundfloor/#
When a client subscribs to a Topic with multiple levels of wildcards, it receives all messages from any Topic before the wildcard, no matter how deep the Topic is. If you only define Topic as #, then you will receive all the messages.
I have collected six PDFS by myself, which spread more than 10W on the whole network. After searching the public account of “programmer Cxuan” on wechat, I replied to Cxuan in the background and got all PDFS. These PDFS are as follows
Six PDF links