Apache Pulsar messaging middleware has recently become very popular as the next generation of in-message middleware. Today, let’s see how awesome it is.

An overview of the

Apache Pulsar is a pub/ Sub messaging platform for persistence using Apache Bookkeeper. It is a messaging middleware for server-side, originally developed by Yahoo and opened source in 2016. It is currently being incubated under the Apache Foundation. It provides the following features:

  • Cross-region replication
  • multi-tenant
  • Zero data loss
  • Zero Rebalancing time
  • Unified queue and flow model
  • High scalability
  • High throughput
  • Pulsar Proxy
  • function

architecture

Pulsar uses a hierarchical structure to isolate the storage mechanism from the broker. This architecture provides the following benefits for Pulsar:

  • Independent extension Broker
  • Independent extended storage (Bookies)
  • Easier containerization of Zookeeper, Broker and Bookies
  • ZooKeeper provides configuration and status storage for clusters

In a Pulsar cluster, one or more agents process and load balance incoming messages from producers, dispatch messages to consumers, configure storage communication with Pulsar to handle various coordination tasks, store messages in BookKeeper instances (aka Bookies), Rely on cluster-specific ZooKeeper cluster tasks, and so on.

  • A BookKeeper cluster consisting of one or more BookIe handles persistent storage of messages.
  • The ZooKeeper cluster specific to this cluster handles coordination between the Pulsar clusters.

More about, please refer to the architecture of Pulsar pulsar.apache.org/docs/en/con…

Four subscription models

There are four subscription modes in Pulsar: EXCLUSIVE, shared, failover, and KEY_shared. These patterns are shown in the figure below.

Details refer to: pulsar.apache.org/docs/en/con…

Performance is better than Kafka

The best part about Pulsar is performance. Pulsar is much faster than Kafka, 2.5 times faster and 40 percent slower than Kafka.

Streaml.io/PDF /Gigaom-…

Note: Comparison is for 1 topic on 1 partition with 100 byte messages, Pulsar can send 220,000+ messages per second.

The installation

The binary version installs Pulsar

# download official binary package/root @ centos7 ~ # wget https://archive.apache.org/dist/pulsar/pulsar-2.8.0/apache-pulsar-2.8.0-bin.tar.gz # extract [root@centos7 ~]# tar ZXF apache-pulsar-2.8.0-bin.tar.gz [root@centos7 ~]# CD apache-pulsar-2.8.0 [root@centos7 Apache-pulsar-2.8.0]# ll total 72 drwxr-xr-x 3 root root 225 Jan 22 2020 bin drwxr-xr-x 5 root root 4096 Jan 22 2020 conf drwxr-xr-x 3 root root 132 Jul 6 11:47 examples drwxr-xr-x 4 root root 66 Jul 6 11:47 instances drwxr-xr-x 3 root root 16384 Jul 6 11:47 lib -rw-r--r-- 1 root root 31639 Jan 22 2020 LICENSE drwxr-xr-x 2 root root 4096 Jan 22 2020 Licenses -rw-r--r-- 1 root root 6612 Jan 22 2020 NOTICE -rw-r--r-- 1 root root 1269 Jan 22 2020 README #binCopy the code

Docker Installation (Emphasis)

[root@centos7 ~]# docker run -it \ -p 6650:6650 \ -p 8080:8080 \ --mount source=pulsardata,target=/pulsar/data \ --mount Pulsar source = pulsarconf, target = / / conf \ apachepulsar/pulsar: pulsar 2.8.0 \ bin/standaloneCopy the code

Port 8080 is used for HTTP access, and port 6650 is used for PulSAR (Java, Python and other clients) access.

The official visualization tool, Pulsar Manager, can manage multiple Pulsar visually. Pulsar.apache.org/docs/en/adm…

[root@centos7 ~]# docker pulsar/pulsar-manager:v0.2.0 [root@centos7 ~]# docker run-it \ -p 9527:9527 -p 7750:7750 \ -e SPRING_CONFIGURATION_FILE=/pulsar-manager/pulsar-manager/application.properties \ Apachepulsar/pulsar - manager: v0.2.0Copy the code

Set the administrator user and password

[root@centos7 ~]# CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token) curl \ -H 'X-XSRF-TOKEN: $CSRF_TOKEN' \ -H 'Cookie: XSRF-TOKEN=$CSRF_TOKEN; ' \ -H "Content-Type: application/json" \ -X PUT http://localhost:7750/pulsar-manager/users/superuser \ -d '{"name": "admin", "password": "admin123", "description": "test", "email": "[email protected]"}' {"message":"Add super user success, please login"}Copy the code

Enter http://server_ip:9527 to log in to the browser

Enter the newly created user and password to configure the managed server

The list of

Toptic list

Toptic details

Client Configuration

Java client

Here is an example of a Java consumer configuration using shared subscriptions:

import org.apache.pulsar.client.api.Consumer;
import org.apache.pulsar.client.api.PulsarClient;
import org.apache.pulsar.client.api.SubscriptionType;

String SERVICE_URL = "pulsar://localhost:6650";
String TOPIC = "persistent://public/default/mq-topic-1";
String subscription = "sub-1";

PulsarClient client = PulsarClient.builder()
        .serviceUrl(SERVICE_URL)
        .build();

Consumer consumer = client.newConsumer()
        .topic(TOPIC)
        .subscriptionName(subscription)
        .subscriptionType(SubscriptionType.Shared)
        // If you'd like to restrict the receiver queue size
        .receiverQueueSize(10)
        .subscribe();
Copy the code

The Python client

Here is an example of a Python consumer configuration using shared subscriptions:

from pulsar import Client, ConsumerType

SERVICE_URL = "pulsar://localhost:6650"
TOPIC = "persistent://public/default/mq-topic-1"
SUBSCRIPTION = "sub-1"

client = Client(SERVICE_URL)
consumer = client.subscribe(
    TOPIC,
    SUBSCRIPTION,
    # If you'd like to restrict the receiver queue size
    receiver_queue_size=10,
    consumer_type=ConsumerType.Shared)
Copy the code

C + + client

Here is an example of a C++ consumer configuration using shared subscriptions:

#include <pulsar/Client.h>

std::string serviceUrl = "pulsar://localhost:6650";
std::string topic = "persistent://public/defaultmq-topic-1";
std::string subscription = "sub-1";

Client client(serviceUrl);

ConsumerConfiguration consumerConfig;
consumerConfig.setConsumerType(ConsumerType.ConsumerShared);
// If you'd like to restrict the receiver queue size
consumerConfig.setReceiverQueueSize(10);

Consumer consumer;

Result result = client.subscribe(topic, subscription, consumerConfig, consumer);
Copy the code

More configuration and operation guide, the official document is very clear, the official document: pulsar.apache.org/docs/

conclusion

As the next generation of distributed message queues, Plusar has a number of attractive features that make up for some of the weaknesses of rival products, such as geographic replication, multi-tenant, scalability, read/write isolation, and so on.

Please like + follow support migrant brother!

If you have any mistakes or other questions, please leave a comment and make corrections. If it helps, please like + retweet and share. For more open source technology articles, please stay tuned to the migrant Worker brother Technology column