August more text challenge | new play strange upgrade, prize stacking get soft 🎁

Recommend a very good Kafka management platform, Kafka soul mate Didi open source Logi-Kafkamanager one-stop Kafka monitoring and control platform

Technical communication

If you want to enter didi LogI open source user group, add my personal wechat: jjdLmn_ group (note: group). The group mainly exchanges Kakfa, ES, Agent, Logi-Kafka-Manager, and other related technologies; There are people in the group to answer your questions. There are people in the related technical field to answer your questions. Everything you ask will be answered

preface

Today’s new column:

To introduce the management operation and maintenance platform of Kafka by Didi open source; The logi-Kafkamanger, kafka’s soulmate, is better than any other kafka management platform you’ve ever used. You can read my columns one by one; If you have any questions, you can leave a message in the comment area or add my personal wechat. Or join the official group to ask questions!

If there are any errors in the article ❎ ~ please remind the blogger to correct ✅, thank you ~~~~

@[TOC]

Before reading this article, you can follow the documentation logi-Kafkamanager deployment. Build your own project;

So with the platform set up, what does the o&M manager do next?

The body of the

After reading this article, you will know the following

  • Access Cluster Procedure
  • Concepts such as Region, logical cluster, and physical cluster
  • Topic application process

The next sure first step is to plug into an existing Kafka cluster; This article is about cluster access;

Access to the cluster

Parameters to describe

parameter describe
The data center Default domestic; This is also just a dimension to divide data, which should be because there are multiple data centers inside didi
The cluster type Invalid. It will be removed later
Security protocols TODO
JMX certification JMX authentication is not required if Kafka does not have authentication enabled3. Solution — Certified JMX
Make sure the JMX port is open, otherwise a lot of data will be lostThe jMX-connection failure problem is resolved

After the physical cluster is successfully added, you can view the added physical cluster. The list shows some basic information, with a possible one minute delay

Basic Information Display

Click the newly added cluster to display a lot of cluster information. I’ll talk about that later, but I’ll focus on two modulesRegion information 和 Logical cluster Information

Create a Region and logical cluster

Before that, let’s take a look at the official concept of logi-Kafkamanager

In the face of large-scale clusters and complex business scenarios. The introduction of the concept of Region and logical cluster is also didi’s experience in long-term practice.

Region: Some brokers are divided into a Region and a Region is used to define resource division units to improve scalability and isolation. If some topics fail, a large number of brokers will not be affected. For example, in the figure on the right, some brokers are divided into regions. When a Topic sends an exception, the impact scope is limited to the Region to which it belongs.

Logical cluster: Some regions can be divided into logical clusters, facilitating the division of large-scale clusters based on services and support capabilities.

Let’s take a few examples of showing scenes;

For small clusters:

It is possible to have only one physical kafka cluster deployed; A Kafka cluster may consist of three to five brokers; All businesses share a common cluster; For such scenarios, there is no need to divide regions and logical clusters so complex; We can simply set Region = logical cluster = physical cluster;

Create a Region PS: Here the state drop-down also has a disk is full option; Ignore the function here, just fill in normal;

Creating a logical cluster (the logical cluster must have a Region configured) Logical cluster ID: uniquely identifies a logical cluster.RegionIdList: Select the Region list. For example, the newly created common_region is displayed in the drop-down list. We choose this Region;

Cluster type: Exclusive cluster: In a physical cluster, part of a Region is allocated for an application as an exclusive cluster. Only for this app. Independent cluster: An independent physical cluster is deployed for an application. Only for this app. Shared cluster: Allocates some regions as shared clusters. All applications are available.

For small clusters; We chose the shared cluster because the other patterns are bound to the application;

How to divide regions and logical clusters in medium – and large-scale clusters

Let’s imagine a scenario; An enterprise Kafka has N physical clusters; Where there is a shared physical cluster broker=20; All applications in the company’s R&D department are plugged into the cluster; Normally they do

After adding the concept of Region and logical cluster in KafkaManager; You can divide it up like this;After this division, when applying for the Topic in each line of business,Operation and maintenance administratorWhen the review is based onResearch and development personnelTo assign a Region.

There are several obvious benefits to this division:

  1. Isolation Topics for different applications are scattered to more centralized brokers; It also reduces communication between brokers; For example, all topics accessed by an application app-1 are partitioned within the 20 brokers. (For example, TopicA goes to 123, TopicB goes to 456…) ; In this case, does app-1 need to communicate with all 20 brokers when sending? If all topics of app-1 are created only in Brokers 1,2, and 3, then the number of connections will be reduced

  2. Cluster stability prevents a small number of exceptions from extending to a larger number of exceptions. For example, if a Broker is unavailable, the extent of the impact is controlled by the system in the Region to which the Broker belongs.

  3. O&m friendliness O&M has a clearer understanding and stronger control over the entire cluster. Logical clusters are divided according to business division and guarantee level. Then you can do some targeted expansion and expansion operations;

After dividing the region and logical cluster, the next operation; User management application application Topic application

Create a regular user

Operation and maintenance administratorCreate a newThe average userUser of the role;The average userYou can log in to the system to do thatApplication to apply for The Topic forOperations such as

A common user applies for an application

For more information on user roles, see Logi-Kafkamanger (5), Operation and maintenance Management — Platform Management (User Management and Platform Configuration).

The average userAfter logging into your account, apply for your own systemOperation and maintenance administratorAfter approval

.

A common user applies for a Topic

A common user will select a specific logical cluster when applying for a Topic. Which application does the Topic belong to? The o&M manager will help you determine which Region or Broker the Topic will fall into based on the logical cluster and the application.

Peak flow:Refers to the maximum rate of production/consumption of a Topic

This is really just a presentation; When a user creates a Topic, he or she estimates the maximum production rate. The operation and maintenance staff partition the Topic according to this rate. If the rate is higher, the number of partitions may be allocated. So how does the filling rate compare to the number of partitions? This is up to the user to decide, since each Broker has different performance configurations; For example, the average user creates a Topic with an estimate of 102KB per message. At its peak, 10,000 pieces of data may be generated per second; So the peak traffic is 102KB *10000/s which is about 1000Mb/s; If the operation and maintenance personnel know that the maximum write rate of a server configured with XX is 300MB/s based on their own experience or pressure test results, the maximum write rate of a server configured with XX is 300MB/s. The topic needs at least 4 brokers configured in this way.

Note: This is not where the actual flow limits are set; Just as a reference value to allocate the number of partitions; There is no quota information written to zK so where can I set the production/consumption limits for the Topic? Logi-kafkamanger (2) How to manage a topic-specific quota in Kafka

Operation and maintenance personnel review Topic

According to the division of the Kafka cluster, the operation and maintenance personnel select the corresponding Region or Broker. For example, this topic belongs to system A, and we just divided it into Region1. Then select the corresponding Region during the audit; You can also specify the Broker directly;

After the Topic is approved, create the corresponding Topic in the selected Broker; Thus Topic creation is restricted to a specific Region or Broker after being managed by KM;

PS: Only when the platform creates a Topic, it is restricted to a specific Broker; However, if you use kafka’s tools directly to create a Topic or enable auto-creation of a Topic within the Broker, you will not be restricted to our Broker. It is recommended to turn off automatic Topic creation within the Broker;

The article lists

Kafka’s soulmate logi-Kafkamanger (1) : cluster access and related concepts

Logi-kafkamanger (2) How to manage a topic-specific quota in Kafka

Logi-kafkamanger (3) operation and maintenance of Kafka’s soulmate — cluster list

O&m of Logi-Kafkamanger (4) — Cluster O&M (Task Migration and Cluster Online Upgrade)

The resources

Cluster access

Kafka quota management (speed limit) mechanism