@[TOC]
How can didi open source LogiKM one-stop Kafka monitoring and control platform be absent for troubleshooting daily operation and maintenance problems
TopicCommand
1. The Topic created
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 3 --topic test
This parameter is optional
| parameter | describe | example |
|---|---|---|
--bootstrap-server Specify the Kafka service |
Specifies the Kafka service to connect to. If there is this parameter, then--zookeeperYou don’t have to |
–bootstrap-server localhost:9092 |
--zookeeper |
Deprecated, connected to the Kafka cluster through zK connection mode; | — Zookeeper localhost:2181 or localhost:2181/kafka |
--replication-factor |
The number of copies should not be greater than the number of brokers. If it is not provided, the default configuration in the cluster is used | –replication-factor 3 |
--partitions |
Number of partitions. This is used to specify the number of partitions when creating or modifying a topic. If no parameter is provided at creation time, the default value in the cluster is used. Note that if the partition is smaller than before, it will be a problem | –partitions 3 |
--replica-assignment |
Copy partition allocation mode; When creating a topic, you can specify the replica allocation; | --replica-assignmentBrokerId-0:BrokerId-1:BrokerId-2,BrokerId-1:BrokerId-2:BrokerId-0,BrokerId-2:BrokerId-1:BrokerId-0 ; This means that there are three partitions and three copies, corresponding to the assigned Broker; Comma (,) separates identification partitions; Colons separate copies |
--config <String: name=value> |
Set topic-level configuration to override the default configuration;This function only works when –create and –bootstrap-server are used together; Please refer to the attachment at the end of the document for a list of parameters that can be configured | For example, overwrite two configurations--config retention.bytes=123455 --config retention.ms=600001 |
--command-config<String: command file path > |
Admin Client startup configuration,This parameter is valid only when –bootstrap-server is used simultaneously; | For example, set the timeout period of the request--command-config config/producer.proterties ; Then configure request.timeout.ms=300000 in the file |
2. Delete the Topic
bin/kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic test
To support regular expressions that match a Topic for deletion, just wrap the Topic in double quotes. For example, delete a Topic starting with create_topic_byhand_zk.
bin/kafka-topics.sh –bootstrap-server localhost:9092 –delete –topic “create_topic_byhand_zk.*”
. Matches any single character except the newline \n. To match a., use a.
·*· : Matches the previous subexpression zero or more times. To match the * character, use *. .* : Any character
Delete any Topic (with caution)
bin/kafka-topics.sh –bootstrap-server localhost:9092 –delete –topic “.*?”
See regular expressions for more usage
3. Expand the capacity of a Topic partition
Zk (not recommended)
>bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic topic1 --partitions 2
Kafka version >= 2.2 The following methods are supported (recommended)
Expanding capacity for a Single Topic
bin/kafka-topics.sh --bootstrap-server broker_host:port --alter --topic test_create_topic1 --partitions 4
Batch expansion (increase the number of Topic partitions matched by all regular expressions to 4)
sh bin/kafka-topics.sh --topic ".*?" --bootstrap-server 172.23.248.85:9092 --alter --partitions 4
“. *?” Regular expression means to match all; You can match on demand
PS: If a Topic has fewer than the specified number of partitions, it throws an exception. However, it will not affect the normal operation of other topics.
This parameter is optional
| parameter | describe | example |
|---|---|---|
--replica-assignment |
Copy partition allocation mode; When creating a topic, you can specify the replica allocation; | --replica-assignmentBrokerId-0:BrokerId-1:BrokerId-2,BrokerId-1:BrokerId-2:BrokerId-0,BrokerId-2:BrokerId-1:BrokerId-0 ; This means that there are three partitions and three copies, corresponding to the assigned Broker; Comma (,) separates identification partitions; Colons separate copies |
PS: Although this configuration is all partition copy allocation configuration, but is taking effect is the new partition; For example, the previous three partitions and one copy were like this
| Broker-1 | Broker-2 | Broker-3 | Broker-4 |
|---|---|---|---|
| 0 | 1 | 2 |
Replica-assignment 2,1,3,4; Change Broker from partition 0 to partition 1
| Broker-1 | Broker-2 | Broker-3 | Broker-4 |
|---|---|---|---|
| 1 | 0 | 2 | 3 |
But it doesn’t actually do that, the Controller will actually cut off the first three as it processes them; Only the new partition allocation mode, the original is still unchanged
| Broker-1 | Broker-2 | Broker-3 | Broker-4 |
|---|---|---|---|
| 0 | 1 | 2 | 3 |
4. Query Topic description
1. Query a single Topic
sh bin/kafka-topics.sh --topic test --bootstrap-server xxxx:9092 --describe --exclude-internal
2. Batch query Topic(regular expression match, below is query all Topic)
sh bin/kafka-topics.sh --topic ".*?" --bootstrap-server xxxx:9092 --describe --exclude-internal
To support regular expressions to match Topic, you simply wrap the Topic in double quotes
This parameter is optional
| parameter | describe | example |
|---|---|---|
--bootstrap-server Specify the Kafka service |
Specifies the Kafka service to connect to. If there is this parameter, then--zookeeperYou don’t have to |
–bootstrap-server localhost:9092 |
--at-min-isr-partitions |
Some statistics and configuration information are omitted during the query | --at-min-isr-partitions |
--exclude-internal |
Exclude kafka internal topics such as__consumer_offsets-* |
--exclude-internal |
--topics-with-overrides |
Only topics with overridden configurations are displayed, i.e. configurations that are specific to Topic Settings overwrite the default configuration; Partition information is not displayed | --topics-with-overrides |
5. Query the Topic list
1. Query the list of all topics
sh bin/kafka-topics.sh --bootstrap-server xxxxxx:9092 --list --exclude-internal
2. Query the list of matching topics (regular expression)
Sh bin/kafka-topics. Sh –bootstrap-server XXXXXX :9092 –list — exclud-internal — Topic “test_create_.*”
This parameter is optional
| parameter | describe | example |
|---|---|---|
--exclude-internal |
Exclude kafka internal topics such as__consumer_offsets-* |
--exclude-internal |
--topic |
You can match with a regular expression to show the topic name | --topic |
About the author: Shi Zhenzhen’s grocery store focuses on knowledge sharing in the Java field and the big data field. The content is mainly principle, source code, actual practice and so on. It insists on output dry goods, and the written content must be verified, and in-depth source code analysis to ensure the accuracy of the content. For a long time in CSDN, and the public number [Shi Zhenzhen’s grocery store] published original articles, welcome to pay attention! If you have questions related to the technical field, welcome to the group communication, each field has a person to answer, you ask, will get a response!
Didi Open source Logi-Kafkamanager one-stop Kafka monitoring and control platform
More
Kafka column continues to be updated… (source code, principle, actual combat, operation and maintenance, video, interview video)
【 Kafka operation 】Kafka network most complete most detailed operation and maintenance command collection (boutique strongly recommended collection!! _ Shi Zhenzhen’s grocery store -CSDN blog
[Kafka actual] partition redistribution may appear problems and troubleshooting problems (production environment actual combat, dry goods!! Very dry!!!!!! Suggested collection)
【 Kafka exception 】 Kafka common exception handling solution (continuously updated! Suggested collection)
【 Kafka operation and Maintenance 】 Partition from allocation, data migration, copy expansion (with video)
Kafka source 】 【 ReassignPartitionsCommand source code analysis (duplicates of scalability, data migration, and redistribution, copy migration across the path
To learn more, go to….