How does Kafka keep messages from being lost

Ps: This article feels to say a lot of nonsense! I hope you found it interesting.

If you don’t know Kafka, take a look at the following articles. Read the first one and learn the rest on demand.

  1. Introductory article! Kafka in plain English!
  2. Five minutes to experience a Kafka
  3. Kafka series 3! Learn how to use Kafka as a message queue in Spring Boot programs.

The condition in which a producer loses a message

After a Producer invokes the send method to send messages, the messages may not be sent due to network problems.

Therefore, we cannot default to sending a message after calling send. To determine whether the message was sent successfully, we need to determine the result of the message sent. Kafka Producer sends messages asynchronously. We can use get() to get the result of the call, but this also makes it a synchronous operation.

See my article for detailed code:Kafka series 3! Learn how to use Kafka as a message queue in Spring Boot programs.

SendResult<String, Object> sendResult = kafkaTemplate.send(topic, o).get();
if(sendResult.getRecordMetadata() ! =null) {
  logger.info("Producer successfully sent message to + sendResult.getProducerRecord().topic() + "-> " + sendRe
              sult.getProducerRecord().value().toString());
}
Copy the code

But it’s not recommended! This can take the form of adding a callback function to it, with the following example code:

        ListenableFuture<SendResult<String, Object>> future = kafkaTemplate.send(topic, o);
        future.addCallback(result -> logger.info("The producer successfully sent a message to topic:{} partition:{}", result.getRecordMetadata().topic(), result.getRecordMetadata().partition()),
                ex -> logger.error("Producer send failed, cause: {}", ex.getMessage()));
Copy the code

If the message fails to be sent, we can check the cause of the failure and send it again.

In addition, it is recommended to be produced by Producerretries(Retry times) Set a reasonable value, usually 3, but a higher value to ensure that messages are not lost. After the configuration is complete, messages can be automatically retried to avoid message loss when network problems occur. In addition, it is recommended to set the retry interval, because if the interval is too small, the retry effect is not obvious, the network fluctuation once you three times a sudden retry finished

A situation in which a consumer loses messages

We know that messages are assigned a specific offset when appended to a Partition. The offset represents the Partition to which the Consumer is currently consuming. Kafka uses offsets to ensure that messages are ordered within a partition.

When the consumer pulls a message from the partition, the consumer automatically submits the offset. One problem with auto-submission is that the consumer just gets the message and is ready to actually consume it, but suddenly hangs up. The message is not actually consumed, but the offset is automatically submitted.

The solution is to manually turn off the automatic submission of offset, and then manually submit the offset after the actual consumption of the message. However, careful friends will certainly find that this will bring about the problem of the message being re-consumed. For example, if you have just consumed the message and have not submitted the offset, then the message will theoretically be consumed twice.

Kafka lost the message

We know that Kafka introduces the Replica mechanism for partitions. Between copies of a Partition there is a guy called the leader, and the other copies are called followers. The messages we send are sent to the Leader replica, and the follower replica can then pull messages from the Leader replica for synchronization. Producers and consumers interact only with the Leader replica. You can understand that the other replicas are just copies of the Leader replicas and exist only to keep the message store secure.

Imagine a situation where a new leader needs to be selected from the follower copy if the broker that the leader copy belongs to suddenly dies. However, if the leader’s data is not synchronized with the follower copy, the message will be lost.

Set acks = all

The solution is to set acks = all. Acks are an important parameter for Kafka producers.

The default value of acks is 1, indicating that our message is successfully sent after being received by the leader copy. When acks = all is configured to represent, the message is not successfully sent until all replicas receive it.

Replication. factor >= 3

To ensure that the leader has followers to synchronize messages, we usually set replication. Factor >= 3 for the topic. This ensures that each partition has at least three copies. Although it causes data redundancy, it brings data security.

Set min.insync.replicas > 1

Typically, you also need to set min.insync.replicas> 1 so that the configuration represents that the message must be written to at least 2 replicas before it can be successfully sent. The default value of min.insync.replicas is 1, which should be avoided in actual production.

However, to ensure high availability of the entire Kafka service, you need to ensure replica.factor > min.insync.replicas. Why is that? Imagine adding equality. If only one copy died, the partition wouldn’t work. This is a clear violation of high availability! Replic. factor = min.insync.replicas + 1 is recommended.

Set the unclean. Leader. Election. The enable = false

Kafka 0.11.0.0 versions are unclean. Leader. Election. Enable parameters with default value of true or false instead

We also said at the beginning that the messages we send are sent to the Leader replica, and the follower replica then pulls the messages from the Leader replica for synchronization. Multiple followers copies of messages between the synchronization condition is different, when we configure unclean. Leader. Election. Enable = false words, If the leader copy fails, the leader will not be selected from the copies whose synchronization degree between the follower copy and the leader does not meet the requirements, thus reducing the possibility of message loss.

Reference

  • Kafka’s official documents: kafka.apache.org/documentati…
  • Geek Time — Kafka Core Technology and Combat, Section 11: How to implement a no-message loss configuration?

Recommended reading

70K Star Java Open Source project available for PDF reading! .

Github 70K Star project JavaGuide is the author of the project. Every week will update some of their own original dry goods in the public account. Public hao background reply “1” to receive Java engineers essential learning materials + interview surprise PDF.