There are a variety of solutions and products for message delivery in the industry. In this article, two representative MQ(rabbitMQ, Kafka) are described and compared.

1. Application scenarios

  • RabbitMQ, based on the AMQP protocol, is developed by the inherently high concurrency erlanng language for real-time, reliable messaging.

  • Kafka is an open-source messaging and subscription system that Linkedin launched in December 2010. Kafka is designed to handle active streaming data, large amounts of data processing.

2. In the architecture model

  • RabbitMQ follows the AMQP protocol. A RabbitMQ broker consists of Exchange,Binding, and queue, where Exchange and Binding form the routing keys. The Producer communicates with the server by connecting the Channel and the server, and the Consumer receives messages from the Queue for consumption. (For a long connection, messages from the Queue are pushed to the Consumer, and the Consumer loop reads data from the input stream.) RabbitMQ is broker centric; There is a mechanism for confirming messages.

  • Kafka follows the general MQ structure, with producer, broker, consumer as the center. The consumer saves the consumption information of the message on the client. The consumer pulls data in batches from the broker according to the consumption point. No message acknowledgement mechanism exists.

3. The throughput

  • Kafka has high throughput, internal use of message batch processing, zero-copy mechanism, data storage and acquisition is the local disk sequential batch operation, with O(1) complexity, message processing efficiency is very high.

  • RabbitMQ is not as good as Kafka in terms of throughput. They start from a different point of view. RabbitMQ supports reliable delivery of messages, transactions, not batch operations. Based on storage reliability requirements, storage can be memory or hard disk.

4. Availability

  • RabbitMQ supports a Miror queue. The primary queue fails and the Miror queue takes over.

  • Kafka’s broker supports active/standby mode.

5. Load balancing in the cluster

  • Kafka uses ZooKeeper to manage brokers and consumers in a cluster. You can register topics on ZooKeeper. Through the coordination mechanism of ZooKeeper, producer saves broker information of corresponding topics, which can be sent to the broker randomly or in polling. In addition, producer can specify shards based on semantics, and messages are sent to a shard of the broker.
  • RabbitMQ load balancing requires a separate loadbalancer.