Kafka provides several command line tools for managing cluster changes. These command line tools can perform some basic operations, including the following six parts:

Take a closer look at the command line tools provided by Kafka (kafka-2.8.0, kafka-2.8.0-src/bin)

1. kafka-topics.sh

1) Create a theme

kafka-topics.sh --zookeeper <zookeeper connect> --create --topic <string>
     --replication-factor <integer> --partitions <integer>
Copy the code
  • Disable-rack -aware: no rack-based allocation policy is required and parameters can be specified
  • –if-not-exists: Ignores an error in creating a topic repeatedly

2) Add partitions

kafka-topics.sh --zookeeper <zookeeper connect>
     --alter --topic <string> --partitions <integer>
Copy the code

Purpose: Topics scale and replicate based on partitions. Partitions are added to expand topic capacity or reduce the throughput of a single partition.

  • Depending on the key topic, if you change the number of partitions, the key-to-partition mapping will also change, so set the number of partitions in the first place to avoid changes
  • –if-exists parameter. If the topic does not exist, errors are ignored. This parameter is not recommended

3) Reduce partitions

There is no way to reduce the number of theme partitions, and if you do, you can only delete the entire theme and then recreate it

4) Delete the theme

kafka-topics.sh --zookeeper <zookeeper connect>
     --delete --topic <integer>
Copy the code

Prerequisite: The delete.topic.enable parameter of the broker must be set to true. If false, deletion requests will be ignored

  • If a topic is no longer in use, as long as it exists in the cluster, it takes up a certain amount of disk space and file handles, and deleting it frees up the occupied resources.
  • Delete a theme and discard all data in the theme. The operation is irreversible

5) List topics

kafka-topics.sh --zookeeper <zookeeper connect>
     --list
Copy the code

6) List subject details

kafka-topics.sh --zookeeper <zookeeper connect> --describe
Copy the code

Describe also provides parameters to filter the output (do not specify –topic, list for these parameters)

  • –topic-with-overrides to find all topics that contain override configurations, which only lists topics that contain configurations that are different from the cluster
  • –under-replicated — lists all partitions that contain unsynchronized copies
  • Unavailable-partitions lists all partitions without bosses that are already offline and unavailable to producers and consumers

7) List the asynchronous partitions

kafka-topics.sh --zookeeper <zookeeper connect> --describe
      --under-replicated
Copy the code

If multiple brokers have asynchronous partitions, a single broker or cluster may have problems

2. kafka-consumer-groups.sh

In Kafka, there are two places to keep information about consumer groups

  • Old version: Saved to ZooKeeper, using the — zooKeeper parameter to specify the ZooKeeper address (the new version is still available)
  • New version: Saves to the broker, specifying the broker hostname and port number with the –bootstrap-server argument

The kafka-consumer-groups.sh tool can be used to list both of these consumer groups. On older versions, you can also remove consumer groups and offset information.

1) List and describe groups

  • The old version
kafka-consumer-groups.sh --zookeeper <zookeeper connect> --list
Copy the code
  • The new version
kafka-consumer-groups.sh --new-consumer --bootstrap-server
    <kafka connect> --list
Copy the code

For any of the groups listed, use –describe instead of –list and specify a specific group by –group to obtain details of that group, which lists information on all the topics in the group and offsets for each partition

2) Delete the group

Only older versions of the consumer client support the delete group operation, which removes the entire group from Zookeeper, including all the saved offsets. Before performing this operation, all consumers must be shut down.

kafka-consumer-groups.sh --zookeeper
     <kafka connect> --delete --group <string>
Copy the code

It is also possible to delete offsets for individual topics without deleting the entire group (consumers must also be turned off before they can read the topic to be deleted)

kafka-consumer-groups.sh --zookeeper
     <kafka connect> --delete --group <string>
     --topic <string>
Copy the code

3. kafka-run-class.sh

Currently, the management tool is only available for offsets submitted to Zookeeper. There is no tool to manage offsets submitted to Kafka by the consumer client. To manage offsets submitted to Kafka by the consumer group, you need to use the corresponding API on the client to submit the group offsets

1) Export offset

Kafka does not provide a ready-made script for exporting offsets, but the underlying Java class implementation can be invoked using the kafka-run-class.sh script. When you export offsets, a file is generated that contains information about the partitions and offsets. The offsets are stored in a file in a format that the import tool can recognize, and each partition occupies a line in the file.

2) Import offsets

Before importing offsets, all consumers must be closed. If the consumer group is active, they will not read new offsets and may overwrite the imported offsets.

4. kafka-reassign-partitions.sh