In the previous article “Installing Kafka on Linux”, we covered how to install Kafka on Linux, how to start/stop Kafka, and how to create posts and generate messages and consume messages. This article introduces some of the common Kafka commands.

The start-stop/create topic/message generation and consumption commands for Kafka were pointed out in the previous article “Installing Kafka for Linux” and will not be covered here. Just a few other common commands.

♛ 1 View consumer status and consumption details

Sometimes we need to care about the state of consumer applications. Generally, consumer applications will know which topic, which partition and which offset of the current consumption through logs. However, when there is a problem with consumers, or for monitoring reasons, we need to know the state and details of consumers. Use the commands provided by Kafka.

Command format:

bin/kafka-consumer-groups.sh --bootstrap-server BORKER_HOST1:PORT1,BORKER_HSOT2:PORT2 --list

bin/kafka-consumer-groups.sh --bootstrap-server BORKER_HOST1:PORT1,BORKER_HSOT2:PORT2 --group GROUP_NAME --describe

Tips:

  • BROKER_HOST is the IP address of Kafka Server and PORT is the server’s listening PORT. Multiple host ports are separated by commas
  • The first command is to get the list of groups. Generally speaking, the application is aware of the consumer group, usually in the application configuration, if known, this step can be omitted
  • The second command is to view the details of a specific consumer group, which needs to be named

Example:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group console-consumer-68682 --describe

Effect:

  • TOPIC: The name of the TOPIC consumed in the group
  • PARTITION: indicates the PARTITION number
  • Current-offset: indicates the OFFSET currently consumed by the partition
  • Log-end-offset: indicates the latest OFFSET of the partition
  • LAG: the consumption LAG interval is log-end-offset-current-offset. The specific LAG depends on the application consumption speed and producer speed. Generally, if the consumption is too large, it may fall behind and the application needs to pay attention to it
  • Consumer-id: indicates the CONSUMER ID assigned to the partition by the server
  • HOST: indicates the HOST where the consumer resides
  • Client-id: indicates the consumer ID, usually specified by an application

♛ 2 Query the offset range of topic

The minimum offset for topic:demo Broker :localhost:9092 can be found using the following command:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic demo --time -2

Query the maximum value of offset:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 -topic demo --time -1

Effect:

♛ 3 Reset the consumer offset

A tool to reset offset can be useful in scenarios where you want to modify the position of the consumer’s offset in order to re-consume or skip a portion of the message.

Command format:

bin/kafka-consumer-groups.sh --bootstrap-server BORKER_HOST1:PORT1,BORKER_HSOT2:PORT2 --group GROUP_NAME --reset-offsets --execute --to-offset NEW_OFFSET --topic TOPIC_NAME

bin/kafka-consumer-groups.sh --bootstrap-server BORKER_HOST1:PORT1,BORKER_HSOT2:PORT2 --group GROUP_NAME --reset-offsets --execute --to-earliest/--to-latest --topic TOPIC_NAME

Tips:

  • BROKER_HOST is the IP address of Kafka Server and PORT is the server’s listening PORT. Multiple host ports are separated by commas
  • The first command is to change the specified GROUP_NAME and topic offset to NEW_OFFSET. After restarting the consumer, the consumption will be consumed from the specified offset. Note that only one value can be set for NEW_OFFSET, that is, all partitions will use this value, if the partition message load is unbalanced, you need to consider whether this is applicable.
  • The second command is to change the offset specified for GROUP_NAME and topic to the earliest or latest position, so that the consumer can consume from the beginning or from the end.

Example:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group console-consumer-68682 --reset-offsets --execute --to-offset 3 --topic demo

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group console-consumer-68682 --reset-offsets --execute --to-latest --topic demo

Effect:

Similar effects can be achieved by directly changing the group ID of consumers in accordance with the default consumption strategy of consumers, but it is simpler, more efficient and safer.

♛ 4 View the topic status and partition load

When the broker goes down and recovers, we can check whether the topic leader is load balanced. Load balancing is especially important in a production environment because kafka sends all read and write requests to the Partition leader.

Command format:

bin/kafka-topics.sh --zookeeper ZOOKEEPER_HOST1:PORT1,ZOOKEEPER_HOST2:PORT2 --describe --topic TOPIC_NAME

Tips:

  • ZOOKEEPER_HOST is the IP address of ZooKeeper used by Kafka, and PORT is the PORT on which ZooKeeper listens. Multiple host ports are separated by commas
  • Similarly, a ZooKeeper cluster does not need to provide an available ZK address and port on all columns

Example:

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic demo

Effect:

Kafka is abnormal if the following symptoms are found:

  • The number of synchronized replicas for each partition of a topic is inconsistent with the set number of replicas
  • For each partition of a topic, the leader ID is -1 or None

♛ 5 Consumer messages

Spending from scratch:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo --from-beginning

Effect:

From the tail:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo --offset latest --partition 0

Effect:

Specify partition:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo --offset latest --partition 0

Take the specified number:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic demo --offset latest --partition 0 --max-messages 2

As shown in the example section, when two messages are fetched, the call is automatically terminated.

♛ Comes with pressure measurement tools

The test uses the built-in test script of Kafka to send requests to Kafka for writing MQ messages and consuming MQ messages through commands. Simulation of different orders of magnitude of MQ message writing and MQ message consumption scenarios, based on the processing results of Kafka, evaluate whether Kafka can handle more than 100 million messages.

Command format:

bin/kafka-producer-perf-test.sh --topic demo --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=localhost:9092

Example:
The test item Number of pressure measurement messages (unit :W) The test command
Write MQ messages 10  bin/kafka-producer-perf-test.sh --topic demo --num-records 100000 --record-size 1000  --throughput 2000 --producer-props bootstrap.servers=localhost:9092
  100  bin/kafka-producer-perf-test.sh --topic demo --num-records 1000000 --record-size 2000 --throughput 5000 --producer-props bootstrap.servers=localhost:9092
  1000 bin/kafka-producer-perf-test.sh --topic demo --num-records 10000000 --record-size 2000  --throughput 5000 --producer-props bootstrap.servers=localhost:9092
Consuming MQ messages 10  bin/kafka-consumer-perf-test.sh --broker-list localhost:9092 --topic demo --fetch-size 1048576 --messages 100000 --threads 1
  100  bin/kafka-consumer-perf-test.sh --broker-list localhost:9092 --topic demo --fetch-size 1048576 --messages 1000000 --threads 1
  1000  bin/kafka-consumer-perf-test.sh --broker-list localhost:9092 --topic demo --fetch-size 1048576 --messages 10000000 --threads 1

Some of the contents of this paper are translated or referenced from the following learning materials, special thanks:

  • Using Apache Kafka Command-line Tools
  • Kafka offset reset
  • Kafka Broker | Command-line Options and Procedure
  • Kafka common o&M commands
  • Kafka command

About the author: Focus on basic platform project development. If you have any questions or suggestions, please feel free to comment! Copyright notice: The copyright of this article belongs to the author and the blog garden, welcome to reprint, but without the consent of the author must retain this statement, and give the original text link in a prominent place on the page of the article. For the record: all comments and messages will be answered as soon as possible. You are welcome to correct your mistakes and make progress together. Or direct private message I support the blogger: if you think the article is helpful to you, you can click on the lower right corner of the article [recommendation]. Your encouragement is the author to adhere to the original and continuous writing of the biggest power! \