Welcome everyone to pay attention to my wechat public number [old week chat architecture], Java back-end mainstream technology stack principle, source code analysis, architecture and a variety of Internet high concurrency, high performance, high availability of solutions.

One, MY story with the Courier boy

A very normal working day, the old week is busy tapping code, the office seems quiet only knock code sound. Suddenly, my phone rang, breaking the silence.

Me: Hello, who is this? Courier: I’m from SF Express. You have a package. Are you at home now? Me: Oh, I am not at home now, you can help me to bring it over in the evening. Courier: How about I drop it off at the rookie station for you? Me: Yes, thank you.

Fortunately, there is a rookie station, otherwise work overtime to go home very late on weekdays, the evening Courier boy is off work again, I have to wait until the weekend at home Courier boy to help me send. If there is no rookie station, let’s take a look at the interaction diagram between the Courier and me:

If there is a newbies station, let’s look at the interaction diagram again:

The rookie station in the story above is the message queue, which is Kafka for this story; The Courier is the producer, and zhou is the consumer. Lao Zhou has been too busy to pick up the Courier from the rookie station, which is a backlog of information. I sent a message to the Courier to confirm that the Courier had been picked up, which is the ACK mechanism.

You guys may have found the benefit of the rookie station. It smells good.

Here are some of the benefits of message queues and usage scenarios:

  • The application of decouplingWhen it came to 618 and Double 11, the Courier boy would be crazy. He had to make every phone call, and then ask the customer whether he was at home and when he had time. So he totally relied on the consignee, and the Courier boy would be crazy. With cainiao station, the Courier will directly put the express in the cainiao Station and then inform the consignee to pick it up, which enables the consignee and Courier to realize decoupling. In the software industry, application decoupling.
  • Asynchronous processingIf there is no rookie Courier station, the Courier boy has to wait for you downstairs to pick up the Courier, and then he can go, this is called synchronous processing. With the rookie station, the Courier puts your delivery at the rookie station, informs you to pick it up, and then he continues to do something else. This is called asynchronous processing. Asynchronous processing has significantly improved the work efficiency of the Courier, and in software, writing asynchronous code can also improve the efficiency of code execution.
  • Flow cutting frontDouble tenth one Lao zhou has bought a lot of things, different store express, along abundant, YunDa, zhongtong, shentong, all in my Sunday morning at ten o ‘clock this time down to take, make this time, I go downstairs and good frequently with novice station, can I have lunch and then go down with Courier, by the way, this to reach ten flow cutting edge effect of that time.

I and the story of Courier brother is true ha, just I have a reader some time ago when the interview sf was asked about Kafka, Kafka to me out of the content, so there is inspiration to write this article.

Two, Kafka introduction

Kafka, originally developed by Linkedin, is a distributed, partitioned, multi-replica, multi-producer, multi-subscriber, Distributed messaging system coordinated by ZooKeeper. Commonly used for Web/Nginx logs, access logs, message services, etc.

Linkedin was donated to the Apache Foundation in 2010 and became a top open source project.

2.1 Coordination Based on ZooKeeper

It should be mentioned that Kafka 2.8.0 implements the Raft distributed consistency mechanism, which means it can run independently of ZooKeeper.

ZooKeeper plays an important role in Kafka by storing Kafka’s metadata. ZooKeeper stores metadata for the Partition and Broker, and is responsible for the election of the Kafka Controller.

ZooKeeper is an external system for Kafka. To deploy a Kafka cluster, ZooKeeper must be deployed, managed, and monitored. ZooKeeper has its own configuration and management tools, which are completely different from Kafka, so having two distributed systems together naturally increases complexity and is more prone to problems. Sometimes the workload can be doubled, such as to enable some security features that need to be configured in Both Kafka and ZooKeeper. In addition to complexity, external storage also reduces system efficiency.

Such as:

  • Every time a Kafka cluster starts, the Controller must load the cluster status information from ZooKeeper.
  • The election of a new Controller can also be troublesome because the amount of metadata that needs to be loaded may already be very large, creating efficiency problems.

As a result, the complexity and efficiency of ZooKeeper have become pain points for Kafka, and the Kafka team has been working hard to eliminate the dependency on ZooKeeper. Kafka 2.8.0 has finally been implemented.

With Raft mode, metadata and configuration information are stored in the @metadata Topic and automatically replicated across the cluster. This makes Kafka a lot simpler and lighter.

It is important to note that ZooKeeper-less Kafka is an early release and is not fully developed, so do not use it in the production environment now.

2.2 Main Application Scenarios

  • Log collection System
  • The messaging system

2.3 Kafka’s main design goals

  • The time complexity isO(1)Provides message persistence capability even forTBData above level can also guarantee constant time access performance.
  • High throughput. Even on very cheap business machines it can be done on a single machine per second100KThe transmission of a message.
  • supportKafka ServerBetween message partitions, and distributed consumption, while ensuring eachpartitionTransmission.
  • Support both offline data processing and real-time data processing
  • Supports online horizontal expansion

2.4 Two main messaging modes

2.4.1 Point-to-point mode

The point-to-point pattern is typically based on a pull or poll messaging model, which is characterized by messages sent to a queue being processed by one and only one consumer. After the producer puts the message into the message queue, the consumer takes the initiative to pull the message for consumption. The advantage of the point-to-point model is that consumers can control how often they pull messages. However, whether the message queue needs to be consumed cannot be sensed by the consumer side, so additional threads are needed to monitor the message queue on the consumer side.

2.4.2 Publish/Subscribe Mode

The publish-subscribe pattern is a message-based messaging model that can have a number of different subscribers. After a producer puts a message on a message queue, the queue pushes the message to consumers who have subscribed to the message. Because the consumer is passively receiving the push, there is no need to sense whether the message queue is waiting to be consumed! Consumer1, Consumer2, and Consumer3 will process messages differently because of their machine performance, but the message queue won’t be able to sense the speed of consumer consumption! So the speed of push becomes an issue with the publish-subscribe model! Assuming the three consumer processing speeds are 8M/s, 5M/s, and 2M/s, if the queue push speed is 5M/s, consumer3 can’t afford it! If the queue push speed is 2M/s, consumer1 and Consumer2 will have a huge waste of resources!

Most messaging systems use the publish-subscribe model. Kafka is a publish-subscribe model.

2.5 Four core Kafka apis

  • Producer API: allows applications to publish streams of records to one or more Kafka topics.
  • Consumer API: Allows an application to subscribe to one or more topics and process the stream of records generated for them.
  • Streams API: Allows an application to effectively convert an input stream to an output stream by acting as a stream processor, using input streams of one or more topics and generating output streams of one or more output topics.
  • Connector API: Allows to build and run reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector for a relational database might capture all changes to a table.

Three, The advantages of Kafka

  • High throughput: The single machine processes tens of millions of messages per second. It maintains stable performance even with many messages stored.
  • A high performance: A single node supports thousands of clients with zero downtime and zero data loss.
  • Persistent data stores: persists messages to disk. By persisting data to disk and preventing data loss. (Zero copy, sequential read, sequential write, utilized page cache)
  • Distributed system, easy to scale out. All producers, brokers and consumers will have multiple producers, brokers and consumers, all of which are distributed.

Machines can be extended without downtime. Multiple producers and consumers may be different applications.

  • reliabilityKafka is distributed, partitioned, replicated, and fault tolerant.
  • Client status maintenance: The state of the message being processed is maintained on the Consumer side, not the server side. Auto-balance when failure occurs.
  • The online and offline scenarios are supported
  • Support for multiple client languages. Kafka supports Java,.NET, PHP, Python and many other languages.

Application scenarios of Kafka

4.1 Collecting Logs

Kafka can collect the Log of various services, through Kafka in a unified interface service open to a variety of consumers.

4.2 Message System

Decouple producers and consumers, cache messages, and so on.

4.3 User activity tracking

  • Kafka is often used to record the activities of Web users or App users, such as browsing, searching, clicking, and so on.
  • This activity information is published by various servers to Kafka’s topics, and consumers subscribe to these topics for real-time monitoring and analysis, which can also be stored in a database.

4.4 Operating Specifications

Kafka is also often used to record operational monitoring data. This includes collecting data from a variety of distributed applications and producing centralized feedback on various operations, such as alarms and reports.

4.5 Streaming processing

Spark Streaming and Storm, for example.

5. Infrastructure

5.1 Kafka Architecture Diagram

5.2 Messages and Batches

  • Kafka’s unit of data is called a message. You can think of a message as a “row” or a “record” in a database. Messages consist of byte arrays.
  • The message has a key, which is also an array of bytes. Keys are used when messages are written to different partitions in a controlled manner.
  • For efficiency, messages are written to Kafka in batches. A batch is a group of messages that belong to the same topic and partition.
  • Splitting messages into batches reduces network overhead. The larger the batch, the more messages processed per unit of time, and the longer the transmission time of individual messages. Batch data is compressed, which improves data transfer and storage, but requires more computational processing.

5.3 model

  • Message schemas have a number of options available for easy understanding. Like JSON and XML, but they lack strong typing capabilities. Many Kafka developers like to use Apache Avro. Avro provides a compact serialization format that separates the schema from the message body. There is no need to regenerate code when schemas change, and it supports strong typing and schema evolution with versions that are both forward and backward compatible.
  • Data format consistency is important to Kafka because it removes the coupling between message read and write operations.

5.4 Topics and Partitions

  • Kafka’s messages are sorted by topic. The topic is comparable toTable of databaseorA folder on a file system.
  • Topics can be divided into partitions, and a topic is distributed across a Kafka cluster through partitions, providing the ability to scale out.

5.5 Producers and consumers

  • The producer creates the message. Consumer spending news.

  • A message is posted to a specific topic.

  • By default, the producer distributes messages evenly across all partitions of the topic:

    • Specifies the partition of the message directly
    • Partition is modulo based on the key hash of the message
    • Polling the specified partition

  • Consumers consume messages by distinguishing between messages that have already been read by an offset.

  • Consumers are part of the consumer group. The consumer group ensures that each partition can only be used by one consumer to avoid repeated consumption.

5.6 Brokers and clusters

  • A separate Kafka server is called the Broker.
  • The broker receives the message from the producer, sets the offset for the message, and submits the message to disk for saving.
  • The broker serves the consumer and responds to the partition read request by returning a message that has been committed to disk.
  • A single broker can easily handle thousands of partitions and millions of messages per second.

Each cluster has one broker that is the cluster controller (automatically elected from the active members of the cluster).

The controller is responsible for the management work:

  • Assign partitions to the broker
  • Monitoring the broker

A partition in a cluster belongs to a broker, called the partition leader.

Partition replication occurs when a partition can be assigned to multiple brokers. Replication of partitions provides message redundancy and high availability. The copy partition is not responsible for reading or writing messages.

6. Core concepts

6.1 Producer

The producer creates the message.

This role publishes messages to Kafka’s topic. When the broker receives a message from the producer, it appends the message to the segment file currently used to append data.

Typically, a message is posted to a specific topic.

  • By default, messages are evenly distributed across all partitions of a topic by polling.
  • In some cases, the producer will write the message directly to the specified partition. This is typically done through a message key and a divider, which generates a hash value for the key and maps it to the specified partition. This ensures that messages containing the same key will be written to the same partition.
  • Producers can also use custom partiers to map messages to partitions based on different business rules.

6.2 Consumer

The consumer reads the message.

  • Consumers subscribe to one or more topics and read the messages in the order in which they were generated.
  • Consumers distinguish between messages that have been read by examining their offsets. An offset is another type of metadata, which is an incrementing integer value that Kafka adds to a message when it is created. In a given partition, the offset for each message is unique. The consumer keeps the offset of the last message read for each partition on Zookeeper or Kafka, and its read status is not lost if the consumer shuts down or restarts.
  • Consumers are part of the consumer group. The group ensures that each partition can only be used by one consumer.
  • If a consumer fails, other consumers in the consumer group can take over the failed consumer’s work, rebalancing, and partitioning reallocation.

6.3 the Broker

A separate Kafka server is called a broker.

The broker serves the consumer by responding to a partition read request and returning a message that has been committed to disk.

  • If a topic has N partitions and a cluster has N brokers, then each broker stores one partition for that topic.
  • If a topic has N partitions and a cluster has (N+M) brokers, then N brokers store a partition for that topic. The remaining M brokers do not store partition data for this topic.
  • If a topic has N partitions and the number of brokers in the cluster is less than N, then one broker stores one or more partitions for that topic. In a production environment, try to avoid this situation, which can cause data imbalance in the Kafka cluster.

Brokers are part of clusters. Each cluster has a broker that also acts as the cluster controller (automatically elected from the active members of the cluster).

The controller is responsible for administration, including assigning partitions to brokers and monitoring brokers.

In a cluster, a partition is subordinate to a broker, who is referred to as the leader of the partition.

6.4 the Topic

Every message published to a Kafka cluster has a category called Topic.

Physically, messages for different topics are stored separately.

Topics are like tables in a database, especially the logical tables behind the database tables.

6.5 Partition

  • Topics can be divided into partitions, and a partition is a commit log.
  • Messages are appended to the partition and then read in first-in, first-out order.
  • The order of messages cannot be guaranteed across the topic scope, but the order of messages within a single partition can be guaranteed.
  • Kafka uses partitioning to achieve data redundancy and scalability.
  • In scenarios where the order of message consumption needs to be strictly guaranteed, the number of partitions needs to be set to 1.

6.6 Replicas

Kafka uses topics to organize data, and each topic is divided into partitions with multiple copies of each partition. Those copies are held on brokers, each of which can hold hundreds or thousands of copies belonging to different topics and partitions.

There are two types of copies:

  • Chieftain duplicate Each partition has a chieftain duplicate. To ensure consistency, all producer and consumer requests go through this copy.

  • All copies other than the follower copy leader are follower copies. Follower replicas do not process requests from clients; their only task is to copy messages from the leader, keeping them in the same state as the leader. If a leader collapses, one of the followers is promoted to become the new leader.

6.7 Offset

6.7.1 Producer Offset

When a message is written, each partition has an offset. This offset is the producer offset and is also the latest and largest offset for the partition.

Sometimes an offset for a partition is not specified. Kafka does this for us.

6.7.2 Consumer Offset

This is A partition offset situation where the producer writes the latest maximum value of offset, which is 12, and when Consumer A consumes from 0 to 9, the Consumer offset is recorded at 9. Consumer B is at 11. The next time they come back, they can pick up where they left off, or they can start from scratch, or they can jump to the most recent record and start “now.”

6.8 copy,

Kafka ensures high availability through replicas. Copies are divided into Leader copies and Follower copies.

Follower replicas include both synchronous and asynchronous replicas, and only synchronous replicas can be switched to leader replicas in the event of a leader replica switch.

6.8.1 AR

All copies in a partition are collectively called Assigned Repllicas (AR).

AR=ISR+OSR

6.8.2 ISR

In-sync Replicas (ISR) are all Replicas (including the leader) that maintain a certain degree of synchronization with the leader Replicas. The ISR set is a subset of the AR set. Messages are sent to the leader copy before the follower copy can pull messages from the leader copy for synchronization. During synchronization, the follower copy lags behind the leader copy to some extent. By “some degree,” I mean the tolerable lag range, which can be configured with parameters.

6.8.3 OSR

Out-of-sync Relipcas (OSR) are copies that lag too much in synchronization with the leader replica (excluding the leader). Under normal conditions, all follower copies should be in some degree of synchronization with the leader copy, i.e. AR=ISR and the OSR set is empty.

6.8.4 HW

HW is short for High Watermak, commonly known as High water level. It represents the offset of a particular message, and consumers can only pull messages up to this offset.

6.8.5 LEO

LEO is short for Log End Offset, which represents the Offset of the next message to be written in the current Log file.