This is the 18th day of my participation in the August Genwen Challenge.More challenges in August

This exception occurs when a Consumer commits a shift, and our Consumer will receive the exception message if an unrecoverable condition occurs. We all know that error retries can be performed on the Consumer side, but error retries cannot be performed if this exception occurs. So try to avoid this exception during development

CommitFailedException cause

The server was rebalanced before submitting the shift, and the partition of the submitted shift has been handed over to another consumer for management, so the current consumer failed to submit the shift. This problem usually occurs when your consumer instance calls the poll method twice in a row, and the interval exceeds the expected value of the max.poll.interval.ms parameter. This is usually a sign that your consumer instance is taking too long to process messages and delay calling the poll method.

The solution

Configure Parameters properly

You can avoid this exception by configuring the maximum interval between polls and reducing the number of poll messages per poll. That is,

  1. Increases the maximum amount of time that the Consumer side allows downstream systems to consume a batch of messages

    Configure the max.poll.interval.ms parameter to increase the length of time consumers consume messages. The default value is 5 minutes.

  2. Reduce the total number of messages consumed at a time by downstream systems

    Max.poll. records is configured to determine the number of pull messages per time. The default value is 500. Reducing the number of pull messages per time can prevent CommitFailedException. This will increase the pressure for consumber to interact with the broker.

Optimize business processing

  1. Shorten the processing time of a single message

    If the average processing time before each message is 100ms, after optimization, the processing time of each message is optimized to 50ms, so that we can double the TPS of the program and shorten the time interval between two polls.

  2. Downstream systems use multithreading to speed up consumption.

    Kafka can have multiple producers submit messages to the broker at the same time. However, each Producer, consumber, processes messages in a single thread, one after the other. Therefore, we can improve the mode of consuming messages and introduce multiple threads to process messages in parallel to speed up message processing.

    For more information about multithreaded development for Consumer: juejin.cn/post/698887… , but everything has advantages and disadvantages, the introduction of multi-threading, we need to manage the communication between threads, displacement submission convention, as well as the life cycle of multiple threads and thread number management, so, before the introduction of multi-threading, we must confirm whether to introduce multi-threading.