​

The opening is introduced

Hello everyone, I will sort out some Java high frequency interview questions to share with partners, and I hope to see partners in the process of finding a job can be enough to use! This chapter will share some frequent interview questions about Java messaging middleware.

Q1:

What are messages and batches?

Messages, units of data in Kafka, are the concept of messages in our general messaging middleware. The message consists of an array of bytes. Messages can also contain keys to select partitions for messages.

For efficiency, messages are written to Kafka in batches.

The ** batch is a group of messages that belong to the same topic and partition. If only a single message is delivered, there is a lot of network overhead, which can be reduced by sending messages in batches. However, there is a trade-off; the more messages contained in a batch, the more messages are processed per unit of time, and the longer the transmission time of a single message. If compressed, data transmission and storage capacity can be improved, but more computing is required.

Q2:

What are themes and partitions?

Kafka’s messages are categorized by topics, which can be divided into several partitions. A partition is essentially a commit log, where new messages are appended to the partition and then read in a first-in, first-out order.

​

But because topics have multiple partitions, the order of messages cannot be guaranteed across the scope of a topic, as can a single partition.

Kafka implements data redundancy and scalability through partitions, which can be distributed on different servers, meaning that a topic can span multiple servers.

We said that Kafka can be viewed as a streaming platform, and most of the time we treat a topic’s data as a stream, no matter how many partitions there are.

Q3:

What do ISR and AR stand for in Kafka? What does scaling for ISR mean?

  • ISR: in-sync Replicas synchronization queue

  • AR: Assigned Replicas All Replicas

The ISR is maintained by the leader, and the followers synchronize data from the leader with some delays (including the delay time replica.lag.time.max.ms and the delay number replica.lag.max.message). X only supports replica.lag.time.max.ms.). Any follower that exceeds the threshold is removed from the ISR and stored in the OUTof-Sync Replicas (OSR) list. New followers are also stored in the OSR first.

Note: AR = ISR + OSR

Q4:

The Broker and the cluster

A separate Kafka server is called the Broker. The main tasks of the broker are to receive messages from producers, set offsets, and commit messages to disk for storage. Provides services to consumers, responds to requests, and returns messages. With the right hardware, a single broker can handle thousands of partitions and millions of messages per second.

Multiple brokers can form a cluster. The broker elects a cluster controller for each cluster. The controller manages, including assigning partitions to and monitoring brokers.

In a cluster, a partition is subordinate to a broker called a boss. But partitions can be assigned to multiple brokers, where partition replication occurs.

​

The benefit of partition replication is that it provides message redundancy. Once the leader broker fails, other brokers can take over leadership. Of course the relevant consumers and producers are reconnected to the new leader.

Q5:

What does ZooKeeper do in Kafka?

Zookeeper is a distributed coordination component. Earlier versions of Kafaka used ZK to store meta information, consumer consumption status, group management, and offset values.

Considering some factors of ZK itself and the probability of single point problems in the whole architecture, the role of ZooKeeper is gradually weakened in the new version. The new Consumer uses the group Coordination protocol within Kafka and also reduces the dependency on ZooKeeper.

Q6:

How do Kafka followers synchronize with leader data?

Kafka’s replication mechanism is neither fully synchronous nor purely asynchronous.

Fully synchronous replication requires that All Alive followers have replicated the message before it is considered a commit. This replication mode greatly affects the throughput rate.

In one-step replication mode, the followers asynchronously copy data from the Leader. If the Leader writes logs to the data, the data is considered to have been committed. In this case, if the Leader fails, the data is lost.

Kafka uses ISR in a very balanced way to ensure data loss and throughput. The followers can copy data from the Leader in batches. The Leader makes full use of sequential disk reads and the Send File (Zero Copy) mechanism, which greatly improves the replication performance. The Leader writes data to disks in batches internally, greatly reducing the message volume difference between the followers and the Leader.

Q7:

Are messages lost and re-consumed in Kafka?

Message sending:

There are two ways to send kafka messages: sync and async.

The default mode is synchronous, which can be configured using the producer.type attribute.

Kafka confirms the production of messages by configuring the request.required. Acks attribute.

  • 0: does not confirm whether the message is received successfully.

  • 1: indicates the confirmation when the Leader receives the packet successfully.

  • -1: indicates that the packet is received successfully by the Leader and Follower.

To sum up, there are six scenarios of message generation and message loss:

  • Acks =0, if the kafka cluster does not receive messages for confirmation, the message may be lost when the network is abnormal or the buffer is full.

  • Acks =1. In synchronous mode, only the Leader confirms the receiving success but hangs up. The copy is not synchronized and data may be lost.

Message consumption:

Kafka message consumption has two consumer interfaces, the low-level API and the high-level API:

  • Low-level API: Consumers maintain their own offset values to achieve full control over Kafka;

  • High-level API: encapsulates the management of parition and offset, easy to use;

If the high-level API is used, there may be a problem that when the message consumer takes out the message from the cluster and submits the new message offset value, the message hangs before consuming. Then the message that did not consume successfully before the next consumption will “weird” disappear.

Solution:

1 For message loss: In synchronous mode, the confirmation mechanism is set to -1, that is, the message is written to the Leader and Follower before the message is successfully sent. In asynchronous mode, to prevent the buffer from being full, you can set an unlimited blocking timeout in the configuration file and keep the producer blocked when the buffer is full.

2 For message duplication: The unique identifier of the message is saved in the external media, and each consumption determines whether the message has been processed.

Q8:

Why doesn’t Kafka support read/write separation?

In Kafka, the producer writes messages and the consumer reads messages by interacting with the Leader copy, thus realizing a production-consumption model of master write and master read.

Kafka does not support primary write slave read because primary write slave read has two obvious disadvantages:

  • Data consistency problem: Data must be transferred from the primary node to the secondary node in a delayed time window, which may cause data inconsistency between the primary and secondary nodes. At some point, the value of A data in both the master node and the slave node is X, and then the value of A data in the master node is changed to Y. Then, before the change is notified to the slave node, the application reads the value of A data in the slave node is not the latest Y value, thus causing the data inconsistency problem.

  • Delay problem: For components like Redis, the process of data writing from the master node to synchronization from the slave node needs to go through the following stages: network → master node memory → network → slave node memory, and the whole process will take a certain amount of time. In Kafka, master/slave synchronization is more time-consuming than in Redis. It goes through the network → master memory → master disk → network → slave memory → slave disk. For delay-sensitive applications, master write/slave read scenarios are not suitable.

Focus, don’t get lost

If you think the article is good, please pay attention to it, like it, and save it. Your support is the motivation for my creation. Thank you all.

If there is a problem with the article, please don’t be stingy, welcome to point out the message, I will check and modify in time.

If you want to know more about me, you can send me a message. Every 8:00 on time push technical articles, so that your way to work is not lonely, and there are monthly book activities, to help you improve hard power!