There is no best message queue middleware (message-oriented middleware for short), only the most appropriate message-oriented middleware. Common usage scenarios of message queues are as follows:

  • The real time: When you don’t need immediate results, but the amount of concurrency needs to be controlled, this is almost the time to use message queues. It mainly solves the problems of application coupling, asynchronous processing and flow cutting.
  • Application of the coupling: Multiple applications process the same message through message queues to avoid interface invocation failures. (e.g. Order -> Stock)
  • Asynchronous processing: Multiple applications process the same message in the message queue and process messages concurrently between applications, which reduces the processing time compared with serial processing. (P2p scenario, broadcast scenario (register SMS, email), etc.)
  • Current limiting peak clipping: Used in the activity of seckilling or buying, to avoid the application system hanging due to heavy traffic; (The queue size is set according to the service tolerance. If the queue size exceeds the threshold, it will return to the end of the activity. We often kill each mall in seconds, and there is no point B in the heart.)
  • Message-driven systems: The system is divided into message queues, message producers and message consumers. Producers are responsible for generating messages, and consumers (there may be more than one) are responsible for processing messages. (Division of labor processing (each corresponding queue), flexible application (received processing/timed processing))

Message queue is one of the main means of asynchronous RPC

Two modes:

  1. Point to point: Each message has only one Consumer and cannot be re-consumed (once consumed, the message is no longer in the message queue)
  2. Publish/subscribe: wechat public number (Topic), everyone (subscribers) subscribe to pay attention to, wechat public number operation platform (publishers) after the release of information, everyone wechat will receive information, here is actually divided into pull/push. One is active push, one is passive pull

    Scaling based on publish/subscribe is scaling horizontally, with multiple queues and consumption groups subscribing (increasing consumption power)
  • pull: The initiative lies in the consumer, the advantage is on-demand consumption (eat buffet, can eat how much take), and the server queue accumulation of message processing is relatively simple (do not record the state ah, state are consumer); The disadvantage is that the message delay (do not know when to pull the update), at this time some friends will ask, then why not call the server notification (there is a saying that is not in its place, the server notification must record the notification status and increase the communication bandwidth between; Of course, it can also be combined with push according to the actual situation to improve the real-time information.
  • push: The initiative is in the service side, the advantage is high real-time, the server can be unified management to carry out the load, but it is easy to lead to slow consumption (we have to consider whether the consumer can bear, after all, you say you know, but only the other side knows how much you know); The disadvantage is that the state of sending messages is centrally managed, which is very stressful.

    For sequential messages, this is a very limited and expensive approach to the scenario. For globally ordered but small error scenarios (log push), pull mode is very suitable. See for yourself

In actual development, the selection of message-oriented middleware is based on the following aspects:

  1. Function: Priority queues, delay queues, dead-letter queues, pull/push consumption patterns, broadcast consumption, message backtracking (traceable, Otherwise be sold didn’t know who) accumulation, message + persistence, message tracking (article links, convenient location), message filter (filter according to the rules, different category message is sent to a different topic), multi-protocol support (universal), cross-language support (popularity), flow control (gnome male – “, it has), order of message (again?) , security mechanism (identity authentication, permission authentication (read and write)), message idempotence (promise to know, promise to do), transactional message (do not want to say), etc
  2. Performance: Generally refers to its throughput (the capacity to produce and consume message bodies of uniform size and different sizes). Performance and functionality are often at odds.
  3. High reliability, high availability: first of all, reliability mainly lies in the persistence of messages (messages will be consumed as long as they are written, and data will not be lost due to failures (this is a good test). If it is from the perspective of the system, it has to be measured from the overall dimension (not just by message-oriented middleware itself, but from the production side, the service side, the consumption side of three dimensions to ensure). In addition, it is mainly due to the dependence on external services (such as Kafka’s dependence on ZooKeeper), which can be divided into strong and weak dependencies. The other is the guarantee brought by its own backup mechanism (such as master-slave replication backup, adding multiple slaves to strengthen the guarantee, and there is also a waste of resources. The Slave may be idle most of the time).
  4. Operation and maintenance: there are usually audit evaluation, monitoring, alarm reminding, disaster recovery, capacity expansion, upgrade deployment and so on. On the one hand, it depends on the dimension supported by middleware, and on the other hand, it depends on the difficulty of integrating automatic operation and maintenance
  5. Community strength and ecological development: This is easy to understand, the use of open source framework at the beginning of the basically pleasant run, but from time to time will always fall into the pit, can climb out of one depends on their own strength, on the one hand depends on the community strength
  6. Cost: It’s much easier for a C-stack team to dig deep into zeroMQ than it is to write Kafka in Scala

First post a map (online Q to), some functions are not supported mainly depends on the mode it uses, read the above detailed instructions should be more clear

RabbitMQ kafka (rabbitMQ, Kafka)

Application:

  • RabbitMQ, based on the AMQP protocol, is developed by the inherently high concurrency erlanng language for real-time, reliable messaging.
  • Kafka is mainly used to deal with active streaming data, large amount of data processing.

Architectural model:

  • The RabbitMQ followCloser agreementA RabbitMQ broker consists of Exchange,Binding, and queue, where Exchange and Binding form the routing keys. The Producer communicates with the server by connecting the Channel and the server, and the Consumer receives messages from the Queue for consumption. (For a long connection, messages from the Queue are pushed to the Consumer, and the Consumer loop reads data from the input stream.) RabbitMQ is broker centric; There is a mechanism for confirming messages.
  • Kafka follows the general MQ structure, with producer, broker, consumer as the center. The consumer saves the consumption information of the message on the client. The consumer pulls data in batches from the broker according to the consumption point. No message acknowledgement mechanism exists.

Throughput:

  • RabbitMQ is not as good as Kafka in terms of throughput. They start from a different point of view. RabbitMQ supports reliable delivery of messages, transactions, not batch operations. Based on storage reliability requirements, storage can be memory or hard disk.
  • Kafka has high throughput, internal use of message batch processing, zero-copy mechanism, data storage and acquisition is the local disk sequential batch operation, with O(1) complexity, message processing efficiency is very high.

Usability:

  • RabbitMQ supports miror(mirrored) queues. The primary queue fails and the Miror queue takes over.
  • Kafka’s broker supports active/standby mode.

Cluster load balancing:

  • RabbitMQ load balancing requires a separate loadbalancer.
  • Kafka uses ZooKeeper to manage brokers and consumers in a cluster. You can register topics on ZooKeeper. Through the coordination mechanism of ZooKeeper, producer saves broker information of corresponding topics, which can be sent to the broker randomly or in polling. In addition, producer can specify shards based on semantics, and messages are sent to a shard of the broker.

rabbitMQ

Development based on Erlang is a message middleware based on AMQP protocol implemented by Erlang language. It originated from financial system and is used to store and forward messages in distributed system. RabbitMQ has become more and more popular today due to its outstanding performance in reliability, usability, extensibility and functionality. Advantages:

  • Due to the nature of the Erlang language, MQ performance is high and concurrency is high;
  • Robust, stable, easy to use, cross-platform, multi-language support, complete documentation;
  • With message confirmation mechanism and persistence mechanism, high reliability;
  • Highly customizable routing;
  • The management interface is rich, and it is also widely used in Internet companies.
  • High community activity;

Disadvantages:

  • Although combined with the concurrency advantages of Erlang language itself, the performance is good, but it is not conducive to secondary development and maintenance;
  • The broker architecture is implemented, meaning messages can be queued on the central node before being sent to the client. This feature makes RabbitMQ easy to use and deploy, but slow to run because of the added latency of the central node and the large size of message encapsulation;
  • Complex interfaces and protocols need to be learned, which is costly to learn and maintain.

activeMQ

Java-based development is a message-oriented middleware developed by Apache and written by Java language, which is completely based on JMS1.1 specification. It provides efficient, extensible, stable and secure enterprise-level message communication for applications. However, due to historical reasons, its market share is not as large as the following three message-oriented middleware. Its latest architecture is named Apollo (Jingdong’s message-oriented middleware is developed based on activeMQ).

  • Cross-platform (JAVA written platform independent, ActiveMQ can run on almost any JVM)
  • You can use JDBC: You can persist data to a database
  • JMS support: Supports the JMS unified interface;
  • Support automatic reconnection;
  • Security mechanism: Supports various security configuration mechanisms, such as Shiro and JAAS, to authenticate and authorize Queue/Topic
  • Complete monitoring: Complete monitoring, including Web Console, JMX, Shell command line, Jolokia REST API;
  • User-friendly: The Web Console is available for most cases, and there are many third-party components available, such as Hawtio;

Disadvantages:

  • Community activity is not as high as RabbitMQ;
  • Things go wrong, messages get lost;
  • Not suitable for thousands of queues;

zeroMQ

C-based development claims to be the fastest message queue in history, developed in C language. ZeroMQ is a messaging queue library, can be in a multi-threaded, elastic scaling between the kernel and host, although most of the time we used to go into the family of the message queue, but in front of it and a few the distinction that having essence, ZeroMQ itself is not a message queue server, more like a set of underlying network communication library, Add a layer of encapsulation to the existing Socket API. Advantages:

  • Known as the fastest message queuing system, especially for high-throughput demand scenarios
  • There is no need to install and run a message server or middleware because your application will play this service role
  • Advanced/complex queues can be implemented, but developers need to assemble multiple technical frameworks themselves
  • Cross-platform, multi-language support
  • Can be used as a Socket communication library

Disadvantages:

  • Only non-persistent queues are provided, meaning that data will be lost if the machine goes down

rocketMQ

Java-based development (Ali Message-oriented Middleware) is ali’s open source message-oriented middleware, which has been donated to the Apache Foundation. It is developed by Java language, featuring high throughput, high availability and suitable for large-scale distributed system applications. Having experienced the baptism of Double 11, its strength cannot be ignored. Advantages:

  • A single machine supports more than 10,000 persistent queues
  • All RocketMQ messages are persistent, written to the system’s Pagecache and then flushed to ensure that both memory and disk have a copy of the data that, when accessed, is read directly from memory.
  • Simple model, easy to use interface (JMS interface is not very practical in many cases)
  • The performance is good for piling messages into brokers (a cluster of one or more servers called brokers) in large numbers;
  • Supports multiple consumption, including cluster consumption and broadcast consumption.
  • Distributed expansion design for each link, master-slave HA(high availability cluster);
  • The development degree is relatively active, the version updates quickly.

Disadvantages:

  • There are not many client languages supported, currently Java and c++, among which c++ is not mature;
  • The RocketMQ community is also less focused and mature than the first two;
  • There is no Web management interface, but a COMMAND-LINE interface (CLI) management tool to query, manage, and diagnose problems.
  • Interfaces such as JMS are not implemented in the MQ core;

kafka

Scala based and Java Development was originally developed by LinkedIn in Scala as a distributed, multi-partitioned, multi-replica zooKeeper-coordinated distributed messaging system, which has been donated to the Apache Foundation. It is a high throughput distributed publish and subscribe messaging system widely used for its horizontal scalability and high throughput. More and more open source distributed processing systems such as Cloudera, Apache Storm, Spark and Flink support Kafka integration. Advantages:

  • Rich client language, support Java,.NET, PHP, Ruby, Python, go and other languages;
  • Excellent performance, single-machine write TPS of about 1,000,000 / SEC, message size of 10 bytes;
  • It provides a completely distributed architecture and replica mechanism, which has high availability and reliability and theoretically supports unlimited accumulation of messages.
  • Batch operations are supported.
  • Consumers use Pull mode to obtain messages, messages are orderly, through control can ensure that all messages are consumed and only once;
  • There are excellent third-party Kafka Web management interface Kafka-Manager;
  • It is mature in the logging field and is used by multiple companies and open source projects;

Disadvantages:

  • When a Kafka single machine has more than 64 queues/partitions, the Load increases significantly. The more queues, the higher the Load, and the longer the response time of sending messages
  • In short polling mode, real-time performance depends on polling interval time.
  • Retry is not supported when consumption fails.
  • Message ordering is supported, but when an agent goes down, messages are out of order.
  • Community updates are slow;

redis

Redis’ PUB/SUB, publish-subscribe model. Make use of Redis’ lists data structure. A good usage pattern is producer Lpush messages, consumer BRPOP messages, and set timeout, can reduce the pressure of Redis. Only if Redis is down and the data is not persisted, high reliability can be guaranteed by AOF and shortening the persistence interval according to the business, and consumption speed can be increased through multiple clients. However, compared with the professional message queue, the state of the message in this scheme is too simple (no state), and there is no ACK mechanism. After the message is taken out, the consumption failure depends on the client to log or push the message to the queue again.

Redis doesn’t support grouping (which is important, because it’s a disadvantage when it comes to load balancing), but it can be used completely as a lightweight queue, but Redis made a Disque, so give it a try.

The deployment of installation

rabbitMQ

A few key concepts:

  • Broker: Simple message queue server entity.
  • Exchange: message switch, which specifies the rules by which messages are routed to which queue.
  • Queue: Message Queue carrier, each message is put to one or more queues.
  • Binding exchanges and queues according to routing rules.
  • Routing Key: The Key used by Exchange to deliver messages.
  • Vhost: Virtual host. Multiple vhosts can be set up within a broker to separate permissions between different users.
  • Producer: A program that delivers messages.
  • A consumer is a program that receives messages.
  • Channel: Message channel. In each connection of the client, multiple channels can be established. Each channel represents a session task.

Use process:

  1. The client connects to the message queue server and opens a channel.
  2. The client declares an Exchange and sets the related properties.
  3. The client declares a queue and sets its properties.
  4. The client uses the routing key to establish a binding relationship between the Exchange and queue.
  5. The client posts a message to exchange.
  6. When exchange receives a message, it routes the message to one or more queues based on the key and the binding that has been set.

Install Erlang for RabbitMQ and install Erlang for RabbitMQ. Install Erlang for RabbitMQ and install RabbitMQ.

The default user name and password for rabbitmq are guest. This user can only be accessed from the local computer by default. You are advised to delete the guest user

Configuring External Access

The # rabbitmq.config file is not available by default
vi rabbitmq.config
# Add the following
[{rabbit, [{loopback_users, []}]}].
Copy the code

Deleting a Guest user

rabbitmqctl  delete_user guest
Copy the code

Add user

rabbitmqctl add_user <username> <newpassword>
Copy the code

Grant super administrator rights to the new user

rabbitmqctl set_user_tags <username> administrator
Copy the code

Start the service (background mode is recommended)

service rabbitmq-server start &
Copy the code

Enable the management UI(background mode is recommended)

rabbitmq-plugins enable rabbitmq_management &
Copy the code

Access: IP :15672, log in as the newly added user

kafka

A few key concepts:

  • Broker: A Kafka cluster contains one or more servers, which are called brokers
  • Topic: Every message published to a Kafka cluster has a category called Topic.
  • Partition: Parition is a physical concept that each Topic contains one or more partitions.
  • Producer: Publishes messages to Kafka Broker
  • Consumer: Message Consumer, the client that reads messages to Kafka Broker.
  • Consumer Group: Each Consumer belongs to a specific Consumer Group (you can specify a Group name for each Consumer, or the default Group if you do not specify a Group name).

Zookeeper is written in Java, so you need to install JDK first

  • KafkaOffsetMonitor (JDK 1.8 or later)
  • Zookeeper: The new version of Kafka comes with ZooKeeper

Download address: kafka.apache.org/downloads

#Download decompressionThe mkdir kafka && CD kafka wget tar - XZF at http://mirrors.shuosc.org/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz Kafka_2. 11-1.0.0. TGZ CD kafka_2. 11-1.0.1#Start the zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
#Start the kafka
bin/kafka-server-start.sh config/server.properties
#Create a Topictest
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
#Parameter Description:
#Create: create a Topic
#Zookeeper: Indicates the ZooKeeper cluster information. Multiple zooKeeper clusters are used separately
#Replication-factor: indicates the replication factor
#Partitions: indicates partition information
#Topic: topic name

#Send messages to topics
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>Simply type

#Get messages from topic
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
#You can see the message you entered earlier
Copy the code

Kafka monitoring tool Kafka does not have its own Web UI. KafkaOffsetMonitor is used here. The program is run in the form of a JAR package, which is convenient to deploy. Only monitoring function, relatively safe to use.

KafkaOffsetMonitor is hosted on Github and can be downloaded from Github. Download: github.com/Morningstar…

#Once you've downloaded itcdGo to the KafkaOffsetMonitor directory and start it either directly or by writing shell scriptsJava - cp KafkaOffsetMonitor - assembly - 0.4.1 - the SNAPSHOT. Jar com. Quantifind. Kafka. Offsetapp. OffsetGetterWeb - port 8089 - zk 127.0.0.1:2181 --refresh 5.minutes --retain 1.dayCopy the code

Script startup

mkdir kafka-monitor.sh
chmod  +x kafka-monitor.sh
vi kafka-monitor.sh
#Copy in the previously launched command as follows:
#! /bin/bashJava -xms512m -xmx512m -xss1024k \ -cp KafkaOffsetMonitor-assembly 0.4.1 -snapshot.jar \ com.quantifind.kafka.offsetapp.OffsetGetterWeb \ --zk localhost:2181 \ --port 8089 \ --refresh 10.seconds \ --retain 5.days#Save, and you can start the script
Copy the code

Zk: IP addresses of zooKeeper hosts. If there are multiple IP addresses, separate them with commas (,). Port: application port (if this parameter is not set, random port numbers are displayed in the logs.) refresh: Frequency of refreshing and storing points in the database retain: Length of retention in the DB dbName: The saved database file name. Default is offsetApp

Github:

Topic: subscribed Topic Partition: Partition number Offest: indicates how many messages have been consumed by the parition. LogSize: indicates how many messages have been written to the Partition. Lag: indicates how many messages have not been consumed. Owner: indicates that consumers are Created. Last Seen: indicates the time when the consumption status is updated.Copy the code

If you can access it, but no information is displayed, only a black screen is displayed, because the UI page depends on ajax.googleapis.com and other public libraries are blocked

conclusion

To be continued…… . Personal blog ~ Jane book ~