In the previous article, we stress-tested MQTT message servers EMQ X and RabbitMQ using the same hardware resources. The results show that EMQ X is not much different from RabbitMQ in many-to-one scenarios. In a one-to-many scenario RabbitMQ is significantly different from EMQ X.

This result will be further analyzed in this installment.

There are three main reasons for the gap: the way nodes communicate, the way message flows are structured, and the use of queues.

Communication between nodes

RabbitMQ – Delegate architecture

RabbitMQ uses the Erlang language of distributed connections, where two nodes are connected to each other, each with a single link to the other. In the case of the figure, three nodes are connected in sequence; When the nodes need to communicate, a message needs to be sent from one node to another over this single link.

In the fan-out example, you normally push messages to queues on all nodes. RabbitMQ is optimized to send your message once, and then its built-in proxy framework will dispatch the message and send it to queues on other nodes. In this process, messages are sent in order, ensuring that they are in the same order in different queues.

This solution is not perfect, however, because you send all messages only once and rely on the same delegate process for distribution. And RabbitMQ’s strategy for selecting this proxy process is based on the publisher’s hash algorithm. So, if you have only one publisher, all messages are pushed all the way to a single delegate agent process.

EMQ X – Gen_RPC

There is a neat design in EMQ X: there is not only distributed connectivity but also Gen_RPC. Distribution joins and Gen_RPC serve their respective functions, the former being used to exchange Mnesia’s data and the latter being used only to forward messages. Instead of automatically generating new inter-node links (1 by default) whenever you need to publish a message from one node to another, EMQ X handles the task of pushing a message from one node to another through these new connections. Instead, it relies on a proprietary Gen_RPC connection designed specifically for this scenario to handle the push work. So in the fan out (one to many) example, these links are fully utilized.

However, this design can suffer in a partitioned environment, as RabbitMQ nodes have only one distributed connection between them, so it is much easier to heal and repair when the connection is broken.

The message flow

MQTT plug-in

RabbitMQ listens for messages published using the MQTT protocol after using the MQTT plug-in. Once the message is received, it is parsed and converted through the AMQP protocol before it is sent to RabbitMQ.

If you want to send a message, you need to go through the socket into MQTT_reader, and then through all the procedures shown in the following figure. However, to receive the message just sent in the same channel at the same time, all the processes shown above need to be repeated in reverse, including MQTT_reader. Mqtt_reader is responsible for both reading and writing.

AMQP

The AMQP scenario is different, where each message is read by a reader and written by a writer. These two channels are independent of each other. Reader only reads the content, while writer only writes the content. The only channel, a channel, is the main Erlang process, which is responsible for exchanging messages.

The obvious design issues with RabbitMQ in MQTT scenarios can lead to performance degradation, so what if the RabbitMQ test cases in AMQP mode were introduced? Comparing EMQ X stress tests with RabbitMQ in MQTT mode and AMQP mode alone, EMQ X still performed better in all tests. But on the whole RabbitMQ in AMQP mode performs better than its own.

For one more

RabbitMQ and EMQ X have similar performance in this scenario.

More than a pair of

However, in fan-out scenarios EMQ X still has a significant advantage, but RabbitMQ (AMQP) has significantly narrowed the gap.

The queue

The above tests all used QoS 1 messages. When QoS 1 messages are sent, they are saved on hard disk each time as a persistent backup. So the use of queue space is particularly important.

RabbitMQ

RabbitMQ maturely uses a default queue space execution (which can be replaced with other queues). The mutable queue is balanced between message persistence and the time it takes to send a message to the client. But in the worst case, a message can be stored in memory. However, it also helps that RabbitMQ can be restarted after a crash and the server can come online again, and all clients can be reconnected and receive persistent messages.

EMQ X

EMQ X implements queues in a very simple way, using priority queues in memory. If an incoming message cannot be pushed into the recipient’s queue, the message is discarded. In EMQ X, messages can be persisted only with some other persistent plug-in, which is provided in the commercial version.

EMQ X is designed to keep the access layer separate, leaving message persistence to the back end. This issue will be addressed in future versions of persistent sessions.

The throttle

The RabbitMQ – curtains

RabbitMQ uses a well-known flow control mechanism that assigns a credit value to each process, as shown below. Suppose our server receives a message that is read by a Reader and sent to a channel. This process consumes the corresponding credits for reader and Channel. In this way, it is possible to keep the two parties’ credit values in sync with the matching method to achieve not excessive sending.

This is actually a good solution. Imagine that we have many users, that is, many queues, and each message sent means that message will be distributed to many queues, which can seriously affect RabbitMQ instances. However, this process prevents RabbitMQ from reading any more messages from the send buffer – the send buffer is almost full!

EMQ X – Current limiting

EMQ X throttling is implemented mainly by limiting the traffic on the reading side. First, by default, 200 messages are read from the socket at a time. Messages are processed one by one after they have been fully received. Once the socket reports that it has reached the maximum limit on the read side, it checks the number of publishers and the number of bytes that have been read and goes to sleep for a period of time based on that number. The receive buffer will eventually fill up and the publisher will not post any more content as required by TCP’s flying window.

conclusion

These are the results and analysis of this horizontal evaluation. The winner is hard to say for sure, but when it comes to server performance, EMQ X is definitely better. But RabbitMQ has its own unique advantages.

Design principles of EMQ X

In terms of design, EMQ X separates FrontEnd from Backend and Flow Plane from Monitor/Control Plane.

  1. The EMQ X core solves the problem of handling massive amounts of concurrent MQTT connections and routed messages.
  2. Take full advantage of Erlang/OTP platform soft real-time, low latency, high concurrency, distributed fault tolerance.
  3. Connection, Session, Router, and Cluster are layered.
  4. The Flow Plane is separated from the Control Plane.
  5. Supports back-end databases or NoSQL to implement data persistence, disaster recovery, backup, and application integration.

System layering for EMQ X

  1. Connection Layer: responsible for TCP Connection processing and MQTT protocol encoding and decoding.
  2. Session Layer: Processes the MQTT protocol publish subscribe message interaction process.
  3. Route Layer: Routes within nodes send MQTT messages.
  4. Distributed Layer: MQTT messages are routed between Distributed nodes.
  5. Authentication and access Control (ACL) : The connection layer supports extensible authentication and access control modules.
  6. Hooks and Plugins: each layer of the system provides extensible Hooks that support the extension of the server through plug-ins.

RabbitMQ is more similar to Kafka’s message queue cache design. It is recommended to use both in IoT projects.

Copyright: EMQ

Original link: www.emqx.com/zh/blog/emq…

Technical support: If you have any questions about this article or EMQ related products, please visit askemq.com to ask questions. We will respond to you in time.

More technical dry goods, welcome to our official account [EMQ Chinese community].