The authors are Penghui Li and Yu Liu, engineers of StreamNative.
Apache Pulsar is the top project of Apache Software Foundation. It is the next generation cloud native distributed message flow platform, integrating message, storage and lightweight functional computing. It adopts the architecture design of computing and storage separation. Supports multi-tenant, persistent storage, and cross-area data replication across multiple equipment rooms, and provides stream data storage features such as strong consistency, high throughput, low latency, and high scalability. GitHub address: github.com/apache/puls…
One of the major advantages of Apache Pulsar is that Pulsar’s multi-layer sharding architecture and layered resource management provide a solid foundation for isolation. Users can isolate resources in the required way to avoid resource competition and ensure system stability.
This series of blogs will discuss Pulsar Isolation in depth. This article, the first in a series, describes how to implement isolation in Pulsar by:
- Separate Pulsar cluster
- Share the BookKeeper cluster
- Single Pulsar cluster
Separate Pulsar cluster
In this environment, you need to create separate Pulsar clusters for the isolation units.
The working principle of
Figure 1 shows how isolation can be achieved by deploying a separate Pulsar cluster.
Key points of working principle:
- Each Pulsar cluster accesses the service through DNS Entry Point and ensures that clients can access the cluster through DNS Entry point. The client can use one or more Pulsar urls, which are the connection addresses of the Pulsar cluster.
- Each Pulsar cluster has one or more brokers and bookies.
- Each Pulsar cluster has a metadata store.
- The metadata store can be divided into Pulsar metadata store and BookKeeper metadata store. This article does not distinguish between the two types of metadata stores; the metadata stores contain both.
- Multiple Pulsar clusters share a configuration store.
- Pulsar’s layered resource management provides a solid foundation for isolation. If you want to isolate a namespace, you need to specify a cluster for the namespace. The cluster must belong to the list of clusters allowed by the tenant. Topics belonging to this namespace are also assigned to the cluster. For information on how to cluster a namespace, see Cluster for a Namespace. For information about managing a Pulsar cluster, see Managing a Pulsar Cluster.
Migrating namespaces
To migrate namespaces between clusters, you need to enable cross-region replication and disable it after all data is replicated to the target cluster. For information on how to set up cross-realm replication for a namespace, see Setting up cross-realm replication for a Namespace.
Expansion and contraction nodes
Scaling up the broker or bookie takes place in the corresponding cluster.
Share the BookKeeper cluster
In this way, you need to deploy a BookKeeper cluster shared by multiple broker clusters.
The working principle of
Figure 2 shows how isolation can be achieved by deploying a shared BookKeeper cluster.
Key points of working principle:
- Each Pulsar cluster accesses the service through DNS Entry Point and ensures that clients can access the cluster through DNS Entry point. Clients can use one or more Pulsar cluster connection addresses.
- Each Pulsar cluster has one or more brokers.
- Each Pulsar cluster has a metadata store.
- Multiple Pulsar clusters share a BookKeeper cluster.
- Pulsar’s layered resource management provides a solid foundation for isolation. If you want to isolate a namespace, you need to specify a cluster for the namespace. The cluster must belong to the list of clusters allowed by the tenant. Topics belonging to this namespace are also assigned to the cluster. For information on how to cluster a namespace, see Cluster for a Namespace. For information about managing a Pulsar cluster, see Managing a Pulsar Cluster.
- Storage isolation is achieved through different Bookie Affinity groups, as shown in Figure 3.
- All Bookie isolation groups share the BookKeeper cluster and metadata store.
- Each Bookie quarantine group has one or more BookIes.
- A user can specify one or more primary or secondary groups for a namespace. Create topics in the namespace on the bookie of the primary group and then create topics in the namespace on the bookie of the secondary group. For details on how to set up the Bookie Affinity Group, see Setting up the Bookie Affinity Group.
Migrating namespaces
Migrating a message service on a namespace to another broker cluster requires changing the cluster for the namespace. To migrate the namespace to another Bookie Affinity Group, you need to change the Bookie Affinity Group. For details on how to set up the Bookie Affinity Group, see Setting up the Bookie Affinity Group. Because all broker clusters share the same BookKeeper cluster, there is no need to copy data to a new BookKeeper cluster.
Expansion and contraction nodes
Broker
When scaling up the broker, note the following:
- When you expand the broker, specify a Broker isolation group for the new broker using the Primary group or secondary Group.
- When downsizing brokers, ensure that there are a sufficient number of brokers in the Broker isolation group.
Bookie
Note the following when expanding or shrinking a bookie:
- When expanding a BookIE, you need to specify the Bookie Affinity Group for the new bookie.
- To shrink bookIe, ensure that the Bookie Affinity Group has a sufficient number of BookIe. For details on how to set up the Bookie Affinity Group, see Setting up the Bookie Affinity Group.
Single Pulsar cluster
With this approach, isolation is achieved by managing a single Pulsar cluster rather than deploying multiple broker clusters and multiple Bookie clusters.
The working principle of
Figure 4 shows how isolation can be achieved by deploying a single Pulsar cluster.
Key points of working principle:
- Each Pulsar cluster accesses the service through DNS Entry Point and ensures that clients can access the cluster through DNS Entry point. The client can use the Pulsar cluster connection address.
- Broker isolation is implemented through different Broker isolation groups (Pulsar assigns topics to brokers in a particular broker isolation). For information on how to set up broker isolation groups, see Setting up Broker Isolation Groups.
- Storage isolation through different Bookie Affinity Groups. For details on how to set up the Bookie Affinity Group, see Setting up the Bookie Affinity Group.
Migrating namespaces
Migrating namespaces to other broker isolation groups requires a change in the namespace isolation policy. For information on how to set set namespace isolation policy, see Setting Namespace Isolation Policy.
Migrating namespaces to other Bookie Affinity Groups (without moving raw data to the target Bookie Affinity Group) requires changing the Bookie Affinity Group. For details on how to set up the Bookie Affinity Group, see Setting up the Bookie Affinity Group.
Expansion or shrinkage mode
Broker
When scaling up the broker, note the following:
- When you expand the broker, specify a Broker isolation group for the new broker using the Primary group or secondary Group.
- When downsizing brokers, ensure that there are a sufficient number of brokers in the Broker isolation group.
Bookie
Note the following when expanding or shrinking a bookie:
- When expanding a BookIE, you need to specify the Bookie Affinity Group for the new bookie.
- To shrink bookIe, ensure that the Bookie Affinity Group has a sufficient number of BookIe. For details on how to set up the Bookie Affinity Group, see Setting up the Bookie Affinity Group.
Reference
In a production environment, users can use all three Pulsar isolation methods mentioned in this article together, or they can choose other methods as required. In general, the following considerations should be taken into account when choosing the isolation method: Certain critical services (such as billing, advertising, etc.) use multiple small Pulsar clusters and do not share storage or local ZooKeeper with other clusters. Using multiple small Pulsar clusters provides the highest level of isolation for critical workloads. Large enterprises (with multiple teams) can deploy a large Pulsar cluster with different namespaces for different isolation groups, depending on capacity or workload. For example, there are a lot of fan-out scenarios that differ from the hardware used in scenarios with minimal end-to-end latency.
reading
- Post recommendation | article take you read the news of the Pulsar reserved and expiration policies
- Tencent experts in-depth analysis of Apache Pulsar five application scenarios
- Pulsar storage space does not free problem analysis and solutions
Click on thelink, get Apache Pulsar hardcore dry goods information!