Welcome to follow our wechat official account: Shishan100

directory

Let me write the PI of 0 out front

(1) How to ensure that data is not lost during downtime?

(2) High availability mechanism of multi-copy redundancy

(3) How to synchronize data between multiple copies?

(4) What exactly does ISR stand for?

(5) What is the meaning of the acks parameter?

(6) Final thoughts

Personal public account: Architecture Notes of Huishania (ID: Shishan100)

Let me write the PI of 0 out front

Once Kafka is on your resume, you’ll almost certainly get one question: What about the impact of the acks parameter on message persistence?

This acks parameter is a very core and critical parameter in kafka usage, determining a lot of things.

So it’s worth checking out this article’s analysis of Kafka’s acks parameters and the rationale behind them, both for interviews and for actual projects.

(1) How to ensure that the data is not lost during downtime?

To understand what this acks parameter means, you need to understand kafka’s highly available architecture.

For example, the figure below shows that for each Topic, we can set it to contain several partitions, and each Partition is responsible for storing part of the data for that Topic.

In A Kafka Broker cluster, partitions are stored on each machine, which stores a portion of the Topic data. This enables distributed storage of Topic data on a single Broker cluster.

But there is a problem. If a Kafka Broker goes down, isn’t the data stored on it lost?

Yes, this is a relatively big problem, distributed system data loss problem, is the first thing he must solve, once said that any machine down, at this time will lead to the loss of data.

(2) High availability mechanism of multi-copy redundancy

Zookeeper, Kafka, Redis Cluster, ElasticSearch, HDFS, etc. All distributed systems have their own multi-copy redundancy mechanism. Multi-copy redundancy is a common feature of almost any good distributed system today.

In a Kafka cluster, each Partition has multiple copies, one of which is called leader and the others are called followers.

As shown in the figure above, suppose a Topic is split into three partitions, namely Partition0, Partiton1, and Partition2, where each Partition has two copies.

For example, one copy of Partition0 is Leader and the other is Follower. The Leader and Follower copies are distributed on different machines.

This multi-copy redundancy mechanism can ensure that the failure of any machine will not lead to complete data loss, because at least there are copies on other machines.

(3) How to synchronize data between multiple copies?

Then we’ll look at how data is synchronized across multiple replicas. In fact, only the Leader of a Partition provides read and write services externally

That is, if a client writes data to a Partition, it is generally writing to the Leader copy of that Partition.

After the Leader copy receives the data, the Follower copy keeps sending him requests to try to pull the latest data and write it to disk. As shown below:

(4) What exactly does ISR mean?

Now that you know how Partiton’s multiple replicas synchronize data, it’s time to see what ISR is.

ISR stands for “in-sync Replicas”, which means the number of followers that are constantly In Sync with the Leader.

For example, if the Broker of a Follower is stuck due to JVM FullGC or other problems, it is unable to pull synchronous data from the Leader in a timely manner.

At this time, it means that the Follower is no longer in a synchronous relationship with the Leader. However, as long as the followers are constantly synchronizing data from the Leader, they are guaranteed to be in a synchronous relationship.

Therefore, each Partition has an ISR. In this ISR, there must be the Leader himself, because the Leader is sure that the data is up to date, and followers who keep in sync with the Leader will also be in the ISR.

(5) The meanings of the acks parameters

With all that stuff in the way, it’s time to get down to business and talk about the meaning of the acks parameter.

If you don’t understand the duplicate mechanism, synchronization mechanism, and ISR mechanism, you won’t be able to fully understand the acks parameter, which actually determines a lot of important things.

The acks parameter is set in the KafkaProducer client

That is, when you write data to Kafka, you can set the acks parameter. However, this parameter actually has three common values that can be set: 0, 1, and all.

The first option is to set the acks parameter to 0, which means that if my KafkaProducer is on the client, I simply send the message, regardless of whether the data falls to disk on the Partition Leader, and I consider the message sent successfully.

If you use this setting, you must be aware that you may be sending a message halfway through. As a result, the Broker where the Partition Leader is located hangs, and as a result your client thinks the message was sent successfully, causing the message to be lost.

The second option is to set acks = 1, which means that as long as the Partition Leader receives the message and writes it to the local disk, it is considered successful, regardless of whether his other followers have synchronized the message.

This is the default setting for Kafka. This is the default setting

That is, by default, if you ignore the acks parameter, the Partition Leader will write successfully.

However, there is a problem here. If the Partition Leader has just received the message and the followers have not had time to synchronize, the Leader’s broker goes down and the message is lost because the other client considers the message sent successfully.

In the last case, the Partition Leader sets acks=all. This means that after receiving the message, the Partition Leader must require all the followers in the ISR list that are synchronized with the Leader to synchronize the message before the message is considered to have been successfully written.

If the Partition Leader has just received the message, but the Follower does not receive the message, the Leader breaks down. The client senses that the message is not sent successfully and tries to send the message again.

At this point, the Follower of Partition 2 May become the Leader, and only the latest Follower in the ISR list becomes the Leader. If the new Leader receives the message, the success is considered.

(6) Final thoughts

Acks =all indicates that the data will not be lost.

If your Partition has only one copy, that is, a Leader, and no followers, do you think acks=all is useful?

Of course, it is useless because there is only one Leader in the ISR, and the Leader breaks down after receiving messages, which also leads to data loss.

Therefore, this acks=all parameter must be used with at least two replicas (one Leader and one Follower) in the ISR list.

In this way, data can be successfully written only when more than two copies are received. In this case, the breakdown of any copy does not cause data loss.

So I hope you understand this article well, for everyone to go out to interview, or work with Kafka is a very good help.

Welcome to our official account: Notes on the architecture of Huoia

Architecture notes, BAT architecture experience taught each other