RocketMQ is an excellent message queue developed by Alibaba. RocketMQ has many features that other message queues do not have. More importantly, RocketMQ is developed in Java and has a low learning cost. Kafka was originally developed by LinkedIn and opened source in early 2011. Kafka is a distributed message queue. Kafka saves messages according to Topic. The sender is Producer and the receiver is Consumer. In addition, a Kafka cluster consists of multiple Kafka instances, each server being a broker.
Need JAVA Spring Cloud large enterprise distributed micro service Cloud to build B2B2C e-commerce platform source code 1038774626
RocketMQ vs. Kafka is a comparison document for your convenience. If there are any errors, please correct them.
Data reliability
RocketMQ supports asynchronous real-time flush, synchronous flush, synchronous Replication, and asynchronous Replication
Kafka uses asynchronous flush mode and asynchronous Replication
Summary: RocketMQ’s synchronous flush is more reliable than Kafka’s on a single machine, without data loss due to OS crashes. Synchronous Replication is also more reliable than Kafka asynchronous Replication with no single point of data at all. In addition, Kafka Replication is based on topic, and can be switched over automatically when the host is down. However, there is a problem here. Because it is asynchronous Replication, data will be lost after the switchover, and if the Leader restarts, data conflicts will occur with the Leader. The RocketMQ open source version does not support Master downtime. The Slave automatically switches to Master. The RocketMQ Ali Cloud version supports automatic switchover.
The performance comparison
Kafka single-machine write TPS is about 1m/SEC and message size is 10 bytes
A RocketMQ single machine can write about 70,000 TPS single instances per second, with three brokers deployed on a single machine, running up to 120,000 TPS single instances per second, and message sizes of 10 bytes
Bottom line: Kafka’s TPS runs to millions on a single machine mainly because the Producer side merges small messages and sends them in batches to the Broker.
Why didn’t RocketMQ do that?
Producer usually uses the Java language, caches too many messages, and GC is a serious problem
When the Producer invokes the interface for sending messages, the message is not sent to the Broker and returns a success message to the service. In this case, the Producer breaks down, resulting in message loss and service errors
Since the Producer is usually a distributed system and each machine is sending multiple threads, we believe that an online system with a single Producer can produce only a limited amount of data per second, not tens of thousands.
The function of caching can be completed by the upper-layer business.
Number of queues supported by a single machine
When a Kafka single machine has more than 64 queues/partitions, the Load increases significantly. The more queues, the higher the Load, and the longer the response time of sending messages
A RocketMQ single machine supports up to 50,000 queues, and Load does not change significantly
What’s so good about queues?
More topics can be created in a single machine because each Topic is made up of a batch of queues
The cluster size of consumers is proportional to the number of queues. The larger the queues, the larger the Consumer cluster can be
Real-time message delivery
Kafka uses short polling, and the real time depends on the polling interval
RocketMQ uses long polling, which is as real-time as Push, with delivery delays of messages typically within milliseconds.
Consumption failure retry
Kafka consumption failure does not support retry
RocketMQ consumption failure Periodic retry is supported. The retry interval is extended
Conclusion: For example, for the recharge application, if the operator gateway is called at the current moment and the recharge fails, it may be because the other party is under too much pressure, and the call will be successful later. For example, alipay to bank deduction is also a similar demand.
Retries here require reliable retries, meaning that failed retries are not lost due to a Consumer outage.
Strict message order
Kafka supports message ordering, but when a Broker goes down, messages are out of order
RocketMQ supports strict message ordering. In sequential message scenarios, when a Broker goes down, sending messages fails, but Mysql Binlog distribution requires strict message ordering
Timing of the message
Kafka does not support timed messages
RocketMQ supports two types of timing messages: RocketMQ supports timing Level only, and Aliyun ONS supports timing Level and a specified millisecond delay time
Distributed transaction messages
Kafka does not support distributed transaction messages
Aliyun ONS supports distributed timed messaging, and a future open source version of RocketMQ has plans to support distributed transaction messaging
Information query
Kafka does not support message query
RocketMQ supports querying messages by Message Id as well as by Message content (specifying a Message Key when sending a Message, an arbitrary string, such as an order Id)
Conclusion: Message query is very helpful in locating message loss problems, such as an order processing failure, whether the message was not received or received incorrectly.
Message back
Kafka can theoretically trace messages back to Offset
RocketMQ supports the ability to trace messages back in time to milliseconds, such as re-consuming messages from a certain hour or a certain second earlier in the day
Conclusion: In a typical business scenario, for example, a consumer does order analysis, but due to the failure of program logic or dependent system, all the messages consumed today are invalid and they need to be consumed again from midnight yesterday. Then the message replay function based on time is very helpful to the business.
Consumption parallelism
The consumption parallelism of Kafka depends on the number of partitions configured for the Topic. For example, if the number of partitions is 10, a maximum of 10 machines can consume in parallel (each machine can only open one thread), or one machine can consume in parallel (10 threads can consume in parallel). That is, the parallelism of consumption is consistent with the number of partitions.
The parallelism of RocketMQ consumption is exactly the same as that of Kafka. The parallelism of out-of-order consumption depends on the number of Consumer threads. For example, a Topic has 10 queues, 10 machines consume, and each machine has 100 threads, so the parallelism is 1000.
Message trajectory
Kafka does not support message traces
Ali Cloud ONS supports message trails
Develop language friendliness
Kafka is written in Scala
RocketMQ is written in the Java language
Broker message filtering
Kafka does not support message filtering on the Broker side
RocketMQ supports two types of Broker side message filtering
Filter by Message Tag, equivalent to subtopic concepts. Upload Java code to the server to do any form of filtering on messages, even Message Body filtering.
Message stacking capability
Theoretically Kafka is more stackable than RocketMQ, but RocketMQ can also support hundreds of millions of messages on a single machine, which we believe is sufficient for the business. Java B2B2C Springcloud emulates Taobao electronic mall system