Mau big share big data directly connected KAFKA business directly used to create a proxy

  • Hundreds of billions of behavioral messages a day 200 meters per second hundreds of Kafka machines

The middleware

  • rocketmq
    • Ideas and ideas borrowed from KAFKA
    • Strong order is hard to do
  • Pulsar is used by fewer people
  • kafka
  • rabbitmq erlang
  • ZEROMQ
  • active MQ
  • datahub(ali)
  • AWS

Basic concepts of Kafka

Behavioral flow data is the most common part of the data that almost all sites use to report their site usage (learn something about behavioral analytics such as funnel analytics common business metrics and focus on learning something about the business)

  • Is a distributed, publish/subscribe based messaging system
  • Kafka has been used by many different types of companies as a data pipeline and messaging system of many types. Distributed, high-throughput communities are active
  • Including page view PV(Page view), page exposure, page Click and other behavior events
    • Use FLINK to do some real-time calculations SOURce is KAFKA
  • Kafka Source, Dataflow Pipeline in real time calculation
  • Business messaging systems decouple groups of microservices by publishing and subscribing messages to eliminate spikes

Kafka is a distributed messaging system developed and open source by LinkedIn. Widely used for its distributed and high throughput, Kafka is now integrated with Cloudera Hadoop, Apache Storm, and Apache Spark.

Kafka profile

Kafka design goals

  • The message persistence capability is provided in O(1) time complexity mode, and the access performance of constant time complexity can be guaranteed even for data above TB level.
    • Each message is a very fast sequential IO read and write operation appended to each TOPIC
    • KAFKA can drop disks
  • High throughput.
    • Low machine requirements even in the very cheap commercial machine can also support the transmission of more than 100K messages per second;
    • Mechanical disk unit price is cheap
    • HDD why slow can read the big talk storage can understand some hard disk basic knowledge 7200 RPM can reach 100MB
  • Support message partitioning between Kafka Servers
    • And distributed consumption, while ensuring the sequential transmission of messages within each Partition;
    • A TOPIC is created with multiple files spread across multiple disks on multiple machines
  • Support offline data processing and real-time data processing;
    • You can consume it right away
    • Offline consumption such as model training scenario offset reset to the earliest, replay, so that the data does not have to rebuild or re-pull from a channel
    • A lot of MQ is producer–customer–ACK and then the data is lost and there is no way to go back
  • Scale out: supports online horizontal scaling;
    • Increase or decrease BROKER partitions at any time
  • History messages can be played back indefinitely

Why not use REDIS as message bussiness?

  • The authors do not recommend Posting as a message queue
  • Business flow scenarios use Redis downstream consumption when there is a problem/or when slow, memory unlimited rise easy OOM
  • Redis Cache Volatile Restart may cause data loss
    • REDIS RDS AOF can make data persistent, even if persistent but OOM problem cannot
    • REDIS data is stored in memory
    • REDIS uses a two-threaded model where traffic intensive scenarios are prone to CPU overloads
    • KAFKA’s data is on disk

Why use a messaging system?

The decoupling

  • Subscription-based publishing can link multiple downstream processes to ensure data consistency between the master system and subsystems
  • The messaging system inserts an implicit data-based interface layer in the middle of the process, which is implemented by both processes. This allows you to extend or modify both processes independently, as long as they adhere to the same interface constraints.
  • The mechanism based on message publishing and subscription can link multiple downstream subsystems of business, and can be arranged and developed step by step without intrusion to ensure data consistency.
  • A TOPIC should have a business to which it belongs, a manager to whom it knows who is using it, and lifecycle management
    • I wonder what the contents are
    • Have a schema registry and go through the definition and implementation yourself
    • TOPIC is a table with uniform constraints on the definition of the underlying PB Protocol buffer
    • Determine whether the input data conforms to the definition or is legal
      • For example: upstream add and subtract fields cause downstream consumption problems themselves exit and cause a failure
    • This is where you have to have a gentleman’s agreement what is the format of the data what is the robustness of the downstream like doing some legitimacy of the data, catching exceptions, monitoring alarms

redundant

  • In some cases, the process of processing data will fail.
    • Unless data is persisted, it will be lost.
    • Message queues persist data until it has been fully processed, thus avoiding the risk of data loss.
    • The ground for
  • In the insert-retrieve-delete paradigm used by many message queues, your processing system needs to make it clear that a message has been processed before it can be removed from the queue, ensuring that your data is stored safely until you are done using it.
    • If deletion is processed before the system explicitly indicates that the message has been consumed, it is safe to delete it
      • This mechanism does not facilitate data backtracking
    • If message processing fails, you can proceed directly without confirmation

scalability

  • Message queues decouple your processing, so it’s easy to increase the frequency with which messages are queued and processed, simply by adding additional processing.
    • Similar to the design idea of computing storage separation
    • The speed of joining can be very fast and the speed of spending can be adjusted and expanded by itself
  • No code changes, no parameters adjustments. Scaling is as simple as turning up the power button.

Flexibility & peak processing capability

  • The application still needs to continue to play a role in the case of a surge in traffic, but such sudden traffic is not common.
  • It would be a huge waste to invest resources in being able to handle these spikes.
  • Using message queues enables key components to withstand sudden access pressures without completely collapsing under sudden overload of requests.
  • Writing directly to the database is not as complex as message queue O1

recoverability

  • The failure of a component does not affect the entire system.
    • Some of the asynchronous logic can be unhooked
  • Message queuing reduces coupling between processes, so that even if a process that processes messages dies, messages that are queued can still be processed after the system recovers.

In order to ensure

  • In most usage scenarios, the order in which data is processed is important.
    • The business logic of A “like” and B “like” is different
  • Most message queues are inherently sorted and ensure that the data will be processed in a particular order.
  • Kafka guarantees the ordering of messages within a TOPIC and Partition.

The buffer

  • In any significant system, there will be elements that require different processing times.
  • Message queues use a buffer layer to help the task execute most efficiently — write queue processing is as fast as possible.
  • This buffering helps control and optimize the speed at which data flows through the system.

Asynchronous communication

  • Many times, users do not want or need to process messages immediately.
    • Trunk and branch processes
    • For example, system notifications can be slowly pulled out of the subsystem
  • Message queues provide asynchronous processing, allowing users to queue a message without processing it immediately.
  • Put as many messages on the queue as you want, and then process them as needed.

Topic & Partition Introduction

Topic

  • Logically, a Topic can be thought of as a queue, and each consumption must specify its Topic, which can be simply interpreted as having to specify which queue to put the message in
  • We classify a class of messages by topic, sort of like tables in a database. It is understandable that a mySQL table must have a scheme specification

Partition

  • To increase Kafka throughput linearly, the Topic is physically divided into one or more partitions.
  • Corresponding to the system is one or more directories
  • You can scale multiple disks on multiple machines

Broker

  • A Kafka cluster consists of one or more servers, each server node called a Broker
  • The Broker stores data for a Topic.
  • If a Topic has N partitions and a cluster has N brokers, then each Broker stores one Partition of that Topic.
    • A machine can have multiple disks. Each disk is responsible for mapping a Partition
  • From the perspective of scale out performance, more nodes of Broker Kafka Server, with more storage, create more partitions to load I/O to more physical nodes, improve the total THROUGHPUT IOPS.

From the perspective of scale up, the more Physical disks a Node has, the more partitions it can load, improving the total IOPS.

Unbalanced Scenario 1

If a Topic has N partitions and a cluster has (N+M) brokers, then N brokers store one Partition for that Topic. The remaining M brokers do not store Partition data for this Topic.

Unbalanced Scenario 2

If a Topic has N partitions and there are fewer than N brokers in the cluster, then a Broker stores one or more partitions for that Topic.

The solution

  • A Topic is just a logical concept, a truly distributed Partition between brokers. Each message sent to the Broker selects which Partition to store according to Partition rules. (Default HASH)
  • Sharding’s rules can be configured by themselves. If Partition rules are set properly, all messages can be evenly distributed to different partitions
  • The Partition where the client configuration data is stored can also be configured by the client. Data in different scenarios can be allocated to the Partition that cannot communicate with the client

Pressure the best ratio of Broker to Partition

The number of partitions on the horizontal axis and TPS on the vertical axis

Experimental conditions: 3 brokers, 1 Topic, no Replication, asynchronous mode, 3 Producer, message Payload 100 bytes:

The test results

  • When the number of partitions is smaller than the number of brokers, the throughput rate increases linearly as the number of partitions increases.
  • When the number of partitions is greater than the number of brokers, throughput does not increase linearly but shows performance jitters

Explanation:

When the number of partitions is smaller than the number of brokers

  • Kafka distributes all partitions evenly across all brokers, so that when there are only two partitions, there are two brokers serving the Topic. When there are three partitions, there are three brokers serving the Topic.
  • At this time, I/O resources are maximized

When the number of partitions exceeds the number of brokers

  • The overall performance of the Broker is limited
    • Disk I/O capacity reached a bottleneck
    • A disk degrades to random I/OS when there are many partitions
  • When there were more partitions than brokers, the total throughput did not increase, and even decreased.
  • The possible reason is that when the number of partitions is 4 or 5, the number of partitions on different brokers is different, and the Producer sends data evenly to each Partition. As a result, the load of each Broker is different and the cluster throughput cannot be maximized.

Storage principle

Kafka benefits from high storage

  • Kafka messages exist on top of the file system. Kafka relies heavily on the file system to store and cache messages, and the common perception is that “disk is slow.”

    • The recommended file system is XFS or EXT4
    • For sequential 7200 RPM I/O HDDS, 100-200MB/s

  • The operating system also uses any freed memory space (freed cached buffer in free-m) remaining in the main memory as a disk cache. All read and write operations go through a unified disk cache (except direct I/O which bypasses the disk cache).

  • Kafka takes advantage of sequential I/O and the accelerated features of the Page Cache (Linux kernel prefetch and flush) to achieve ultra-high throughput.

    • If there is a read, it can hit the cache
    • This place can refer to chu Ba’s article non-amateur study alone
    • Page Cache contamination can also cause performance degradation

  • Any messages published to a Partition are appended to the end of the Partition data file. This sequential write to disk makes Kafka very efficient.

  • Write back Writes data to the memory and then writes data to the kernel

Kafka uses comback to collate empty files. Kafka uses comback to collate empty files. Kafka uses comback to collate empty files

  • The Kafka cluster keeps all published messages, regardless of whether the message is consumed or not
    • Kafka provides configurable retention policies to delete old data (there is also a policy to delete data based on partition size).
    • For example, if the retention policy is set to two days, a message is available for consumption for two days after it is written, after which it is discarded to make room.
    • Kafka’s performance is independent of the amount of data it stores, so it’s okay to store data for a long time.
  • This piece of data can be set to file scrolling like MYSQLbinlog

What is the Offset

  • Offset: indicates the Offset.
  • Each message has a 64-byte Offset unique to the current Partition, which is the Offset equivalent to the first message in the current Partition, i.e. the number of messages.
  • Consumers can specify the location of consumption information, when the consumer hangs up and then resumes, can continue consumption from the consumption location.
  • This is where you can modify the auto_offset_reset parameter to specify the consumption policy

Suppose we have only one Broker in our Kafka cluster. We create two topics named “Topic1” and “Topic2”, with 1 and 2 partitions respectively. The following three folders will be created under our root directory:

In Kafka’s file store,

  • Each Topic has multiple distinct partitions, each of which is a directory.
  • Each Partition is evenly divided into multiple Segment files of equal size
  • Each Segment File consists of an index File and a data File. They are always paired with the suffix “.index” and “.log” to denote the Segment index File and the data File.

Format of the index file

The source of the offest

For example, the metadata <3, 497> in the index file represents the third Message in the data file (368769 + 3 = 368772 Message in the global Partition) and the physical offset address of the Message is 497.

Kafka index

  • Index files do not start at zero and are not incremented by one at a time. This is because Kafka uses sparse Index storage, creating an Index every certain byte of data.

  • It reduces the Index file size and allows indexes to be mapped to memory, reducing disk I/O overhead while not consuming too much query time.

  • Because the file name is the Offset of the last Message in the previous Segment, when we need to find a -message with the Offset specified, we can find the Segment to which it belongs by binary search in the filename of all segments.

  • Find its physical location on the file in its Index file and retrieve the Message.

Suppose we have only one Broker in our Kafka cluster. We create two topics named “Topic1” and “Topic2”, with 1 and 2 partitions respectively.

The following three folders will be created under our root directory:

In Kafka, there are multiple partitions under the same Topic, and each Partition is a directory. Each directory is evenly divided into multiple Segment files of equal size. Segment files are composed of index files and data files, which always appear in pairs. The suffix “.index” and “.log” subtables denote Segment index files and data files.

  • For example, the metadata <3, 497> in the index file represents the third Message in the data file (368769 + 3 = 368772 Message in the global Partition) and the physical offset address of the Message is 497.
  • Note that the Index file does not start at 0 and is not incremented by one at a time. This is because Kafka uses sparse Index storage, creating an Index every certain byte of data.
  • It reduces the Index file size and allows indexes to be mapped to memory, reducing disk I/O overhead while not consuming too much query time.
  • Because the file name is the Offset of the last Message in the previous Segment, when we need to find a Message with the Offset specified, we can find the Segment to which it belongs by binary search in the filename of all segments.
  • Find its physical location on the file in its Index file and retrieve the Message.

How does Kafka know exactly what Message offset is? This is because Kafka defines a standard data store structure, where each Message in a Partition contains the following three properties: Offset: The offset of Message in the current Partition is a logical value that uniquely identifies a Message in the Partition, which can be simply considered as an ID. MessageSize: indicates the size of the Message content Data. Data: Specifies the content of the Message.

Kafka has added a.timeIndex index file to shard log files since version 0.10.0.0, which locates messages based on time stamps. You can also run the kafka-dump-log.sh script to check the contents of the time index file.

  • First locate the shard, compare 1570793423501 with the maximum timestamp of each shard (the maximum timestamp takes the last record time of the time index file, if the time is 0, take the last modified time of the log segment), until find the log segment greater than or equal to 1570793423501. So time to locate indexed file 00000000000003257573. Timeindex, its biggest timestamp for 1570793423505.
  • Repeat offset to find the log file.

Producer & Consumer

Producer

When a Producer sends a message to the Broker, it selects the Partition to store it according to the Paritition mechanism. If the Partition mechanism is set properly, all messages can be evenly distributed to different partitions, thus achieving load balancing.

  • If a Partition is specified, the specified Value is directly used as the Partition Value.
  • If the Partition is not specified but contains a Key, the Hash value of the Key is mod to the number of partitions to obtain the Partition value.
  • If there is no Partition Key or Partition Key, the system generates a random integer during the first call (the integer is incremented in each subsequent call) and modulates this value with the number of partitions available to obtain the Partition value. This is also known as the round-robin Round Robin algorithm.

To ensure that the data sent by Producer can be reliably sent to the specified Topic, each Partition of a Topic needs to send an ACK to Producer after receiving the data sent by Producer. If the Producer receives an ACK, it sends the next round, or resends the data otherwise.

  • Once the partition is selected, the producer knows which topic and partition the message belongs to, and it adds this record to the batch messages of the same topic and partition, which another thread is responsible for sending to the corresponding Kafka Broker.
  • When the Broker receives a message, it returns a RecordMetadata object containing the message’s subject, partition, and displacement if it is successfully written, otherwise an exception is returned.
  • Once the producer receives the result, it may retry the exception.

Poor Kafka Broker performance Finds a large number of network connection disconnections through nmon context switches

Producer Exactly Once

Version 0.11 of Kafka introduced idempotency: No matter how many duplicate data producers send to the Server, the Server persists only one.

  • To enable idempotency, simply set enable.idompotence to true in the Producer argument.

A Producer that enables idempotent is assigned a PID during initialization, and messages sent to the same Partition carry Sequence numbers.

  • Borker caches , and the Broker persists only one message when it is submitted with the same primary key. ,partition,seqnumber>
  • However, PID changes after restart, and different partitions have different primary keys, so idempotent cannot guarantee Exactly Once across Partition sessions.

Cannot achieve Exactly Once across sessions

Consumer

Consider a scenario where we read a message from Kafka, examine it, and produce the resulting data. We could create a consumer instance to do this, but what if producers can write messages faster than consumers can read them? So over time, the message pile gets worse and worse. For this scenario, we need to add multiple consumers to scale horizontally. A Kafka consumer is part of a consumer group, and when multiple consumers form a consumer group to consume a topic, each consumer receives a message from a different partition. Suppose you have a T1 topic that has four partitions; We also have a consumer group, G1, which has only one consumer, C1. Then consumer C1 will receive messages for these four partitions.

If we add a new consumer C2 to consumer group G1, then each consumer will receive messages for two separate partitions. The Partition within T1 Topic is equally divided among all consumers consuming G1, where C1 consumes P0 and P2 and C2 consumes P1 and P3.

If you increase to four consumers, each consumer will receive a message for one partition. At this point each consumer processes one of the partitions, running at full load.

But if we continue to add consumers to this consumer group, the remaining consumers will be idle and will not receive any messages.

  • In a word, we can expand the level and improve the consumption capacity by increasing the number of consumers in the consumer group.

This is why it is recommended to create topics with a large number of partitions to increase the number of consumers to improve performance under high consumption loads.

  • In addition, the number of consumers should not be more than the number of districts, because the extra consumers are idle and not helpful.
  • If we still have bottlenecks in C1 processing messages, how can we optimize and process them?

The message inside C1 was sharding twice, and multiple Goroutine workers were enabled for consumption. In order to ensure the correctness of offset submission, the watermark mechanism needed to be used to ensure the minimum offset saved before submitting to the Broker.

Consumer Group

An important feature of Kafka is that a message can be written once, allowing any number of applications to read the message. In other words, each application can read the full amount of messages. In order for each application to be able to read the full amount of messages, the application needs to have different consumer groups. For the example above, suppose we add a new consumer group, G2, that has two consumers as shown in the figure. In this scenario, consumer group G1 and consumer group G2 both receive full messages for T1 topics and logically belong to different applications. In the end, it boils down to this: If the application needs to read full messages, set a consumer group for the application; If the app doesn’t have enough spending power, consider adding customers to that group.

  • Consumer groups are isolated from each other and consume their own like one from the beginning and one from the end

Hit the pit

When a new consumer joins a consumer group, it consumes one or more partitions that were previously the responsibility of other consumers.

  • When a consumer leaves a consumer group (such as a reboot, outage, and so on), the partitions it consumes are allocated to other partitions. This phenomenon is called rebalancing.
  • Rebalancing is an important feature of Kafka, which ensures high availability and horizontal scaling. However, it is important to note that during rebalancing, no consumer can consume messages, thus making the entire consumer group temporarily unavailable.
  • Rebalancing partitions can also cause the original consumer state to expire, causing consumers to renew their state, which can also reduce consumption performance.
  • A consumer keeps alive within a consumer Group by periodically sending a Hearbeat to a Broker that acts as a Group Coordinator. The Broker is not fixed, and each consumer group may be different.
  • When a consumer pulls a message or submits, a heartbeat is sent. If the consumer does not send a heartbeat for more than a certain amount of time, its Session expires, the group coordinator assumes that the consumer is down and triggers a rebalance.

Whenever consumers re-add consumption information, reblance needs to be made, which will be unavailable for a short time

  • As you can see, there is a certain amount of time between when the consumer is down and when the session expires, during which time the consumer’s partition cannot consume messages.
  • In general, we can do a graceful shutdown so that the consumer sends an away message to the group coordinator, so that the group coordinator can rebalance immediately without waiting for the session to expire.
  • In version 0.10.1, Kafka changed the heartbeat mechanism to separate the sent heartbeat from the pull message, so that the frequency of the sent heartbeat is not affected by the pull frequency.
  • In addition, higher versions of Kafka allow you to configure how long a consumer does not pull a message but still stays alive, which avoids livelock. A live lock is an application that is not malfunctioning but cannot be consumed further for some reason.
  • However, live locking is also easy to lead to chain failure. When the performance of components downstream of the consumer end degrades, message consumption will become slow, and it is easy to turn to livelock’s rebalancing mechanism, instead of affecting throughput.

Leader & Follower

Data Loss Scenario

  • In the initial case, two messages have been written to primary copy A, corresponding to HW=1, LEO=2, LEOB=1, and one message has been written to primary copy B, corresponding to HW(High water mark)=1, LEO=1.
  • After receiving the request, the primary copy updates LEOB=1, indicating that copy B has received message 0, and then tries to update the HW value. Min (LEO,LEOB)=1, indicating that no update is required. Message 1 and the current partition HW=1 are then returned to slave copy B, and the response from slave COPY B is written to the log and LEO=2 is updated, then HW=1 is updated. Although two messages have been written, the HW value will not be updated to 2 until the next round of requests.
  • After the restart, log truncation is performed based on the HW value, that is, message 1 is deleted.
  • FetchOffset =1 request is sent from secondary copy B to primary copy A. If primary copy A is not abnormal at this time, then there is no problem as in step 2. If primary copy B is down at this time, then secondary copy B will become the primary copy.
  • When copy A recovers, it becomes A slave copy and logs truncation based on the HW value, that is, message 1 is lost permanently.

Data inconsistency scenario

  • In the initial state, two messages corresponding to HW=1, LEO=2 and LEOB=1 have been written to primary copy A, and two messages corresponding to HW=1 and LEO=2 have been synchronized to secondary copy B.
  • After receiving the request, primary copy A updates the partition HW=2 and returns the value to secondary copy B. If secondary copy B breaks down at this time, the HW value fails to be written.
  • Let’s assume that the master copy A also goes down, and the slave copy B first recovers and becomes the master copy. At this point, log truncation occurs, only message 0 is retained, and then services are provided externally. Let’s assume that A message 1 is written externally (this message is different from the previous message 1, and different messages are identified with different colors).
  • When copy A wakes up, it will become slave copy and log truncation will not occur because HW=2, but the message corresponding to shift 1 is actually inconsistent.

A mechanism was introduced to solve this problem

Leader epoch

The HW value is used to measure the success of copy backup and log truncation criteria in case of failure, which may lead to data loss and data inconsistency. Therefore, the leader epoch concept is introduced in the new Kafka (0.11.0.0).

Epoch actually means a version number. Leader epoch represents a key-value pair <epoch, offset>, where epoch represents the version number of the leader master copy. The code starts from 0, and when the Leader changes once, it will +1. Offset indicates where the first message was written to the master copy of the epoch version.

For example, <0,0> indicates that the first primary copy writes messages from offset 0, and <1,100> indicates that the second primary copy writes messages from offset 100 with version 1. The primary copy stores this information in the cache and periodically writes it to the checkpoint file. This information is queried from the cache every time a primary copy switchover occurs.

Data reliability

Producer required.acks

For some unimportant data, the reliability of the data is not very high and can tolerate a small amount of data loss. Therefore, there is no need to wait for the followers in the ISR to accept all the data successfully. Only the messages synchronized by all replicas in the ISR are committed. However, when the Producer publishes data, the Leader does not need all replicas in the ISR to confirm receiving the data.

  • 0: Producer does not wait for the Broker to ACK. This provides the lowest latency. The Broker returns data as soon as it receives it before writing it to disk.
  • 1: The Producer waits for the ACK from the Broker, and the Partition’s Leader returns an ACK after falling to the disk. If the Leader fails before the Follower synchronization succeeds, data will be lost.
  • -1 (all) : The Producer returns an ACK only after the Broker and the Partition’s Leader and Follower fall successfully. However, when the Broker sends an ACK, the Leader fails, causing data duplication.

Request. Required. Acks =-1 and the min.insync.replicas parameter (which can be set at the Broker or Topic level) should be used to maximize data reliability. Min.insync.replicas Specifies the minimum number of replicas in the ISR. The default value is 1. This parameter takes effect only when the request.required. If replications of ISR is less than min. Insync. When the number of the replicas, the client will return anomalies: org.apache.kafka.com mon. Errors. NotEnoughReplicasExceptoin: Messages are rejected since there are fewer in-sync replicas than required.

request.required.acks=1

The Producer sends data to the Leader. The Leader writes the local log successfully and returns the data to the client successfully. The Leader breaks down before the replica in the ISR can pull the message, and the sent message will be lost.

request.required.acks=-1

Replicas >=2 and min.insync.replicas>=2, data will not be lost.

There are two typical cases. If acks=-1, data is sent to the Leader. After all the followers of the ISR complete data synchronization, the Leader hangs up. A new Leader is elected and data is not lost.

If acks=-1, some ISR replicas are synchronized after data is sent to the Leader, and the Leader hangs up. For example, if follower1 and follower2 become the new Leader, the Producer will get an exception, and the Producer will re-send the data, and the data may be duplicated.

Kafka high-performance

Architecture level:

  • Partition level parallelism: Broker, Disk, and Consumer.
    • One machine can hang multiple disks at 7200 RPM of sequential IO
    • Ensure one-time consumption of data
  • ISR
    • Let’s say the one that can’t catch up with the 3 replicas is kicked out
  • The Broker level is load balanced

IO level:

  • Batch, speaking, reading and writing

    • Reducing network round-trips is like Redis pipline
    • Custormer is a long poll that can set how many pieces of data can be pulled at a time

  • Disk sequence IO

    • A large number of partion scenarios may degenerate into random IO

  • Page Cache

  • Zero Copy(Disk — kernel — Network)

    • Developer.ibm.com/languages/j…
    • Files are sent over the network without passing through the kernel
    • Normal disk — kernel user mode kernel network
    • www.usenix.org/conference/…

  • The compression

  • Catch a slow disk

    • MAD algorithm
    • Dbscan algorithm

References

https://mp.weixin.qq.com/s/fX26tCdYSMgwM54_2CpVrw https://www.jianshu.com/p/bde902c57e80 https://mp.weixin.qq.com/s?__biz=MzUxODkzNTQ3Nw==&mid=2247486202&idx=1&sn=23f249d3796eb53aff9cf41de6a41761&chksm=f9800c2 0cef785361afc55298d26e8dc799751a472be48eae6c02b508b7cb8c62ba3ac4eb99b&scene=132#wechat_redirect https://zhuanlan.zhihu.com/p/27551928 https://zhuanlan.zhihu.com/p/27587872 https://zhuanlan.zhihu.com/p/31322316 https://zhuanlan.zhihu.com/p/31322697 https://zhuanlan.zhihu.com/p/31322840 https://zhuanlan.zhihu.com/p/31322994 https://mp.weixin.qq.com/s/X301soSDWRfOemQhk9AuPw https://www.cnblogs.com/wxd0108/p/6519973.html https://tech.meituan.com/2015/01/13/kafka-fs-design-theory.html https://mp.weixin.qq.com/s/fX26tCdYSMgwM54_2CpVrw https://mp.weixin.qq.com/s/TUFNictt8XXLmmyWlfnj4g https://mp.weixin.qq.com/s/EY6-rA5DJr28-dyTh5BP8w https://mp.weixin.qq.com/s/ByIqEgKIdQ2CRsq4_rTPmA https://zhuanlan.zhihu.com/p/77677075?utm_source=wechat_timeline&utm_medium=social&utm_oi=670706646783889408&from=timeli ne https://mp.weixin.qq.com/s/LRM8GWFQbxQnKoq6HgCcwQ https://www.slidestalk.com/FlinkChina/ApacheKafka_in_Meituan https://tech.meituan.com/2021/01/14/kafka-ssd.html https://www.infoq.cn/article/eq3ecYUJSGgWVDGqg5oE?utm_source=related_read_bottom&utm_medium=article https://mp.weixin.qq.com/s/Zz35bvw7Sjdn3c8B12y8Mw https://tool.lu/deck/pw/detail?slide=20 https://www.jiqizhixin.com/articles/2019-07-23-11 https://www.jianshu.com/p/c987b5e055b0 https://blog.csdn.net/u013256816/article/details/71091774 https://zhuanlan.zhihu.com/p/107705346 https://www.cnblogs.com/huxi2b/p/7453543.html https://blog.csdn.net/qq_27384769/article/details/80115392 https://blog.csdn.net/u013256816/article/details/80865540 https://tech.meituan.com/2021/01/14/kafka-ssd.html https://www.infoq.cn/article/eq3ecYUJSGgWVDGqg5oE?utm_source=related_read_bottom&utm_medium=article https://mp.weixin.qq.com/s/Zz35bvw7Sjdn3c8B12y8Mw https://tool.lu/deck/pw/detail?slide=20 https://www.jiqizhixin.com/articles/2019-07-23-11 https://mp.weixin.qq.com/s/LRM8GWFQbxQnKoq6HgCcwQ https://mp.weixin.qq.com/s/EY6-rA5DJr28-dyTh5BP8wCopy the code

MAO big share

  • A wise man is impatient

  • The architecture is too perfect to land

  • The kernel knows how to write code that’s too abstract for anyone else to handle

  • Go back to your design every once in a while and find a lot of problems and add some improvements

    • Every time I look, I learn something new

  • A little coppying is better than a little dependency

  • R&d efficiency

    • Distributed compilation incremental compilation bazel
    • Automated testing

  • Pay issue

    • High performance high output is higher than the same level of output
    • Important and helpful to the team
    • Old cattle work hard and dirty work
    • It can be said that the salary of the previous company is not good, but I have enough ability

  • How to do technical recruitment well

    • Always find someone better than you
    • Start by talking about getting to know some great people. Keep an eye on great people. Don’t count on HR
      • You must be the first thing that bull thinks of when he wants to leave

  • If you want to do things well, follow people to do things, understand the business, and see the problem clearly

    • Put yourself in the other person’s shoes and see the pain points

  • To be liked by your boss and colleagues?

    • Don’t try to be everyone’s favorite person
    • Less talk, more work, more reward
      • The boss’s concern immediately came up with a plan for him to do
      • Under-promise, over-deliver, over-deliver and your boss will believe you
      • Work in a high profile low-key life
    • Just call the boss when making the decision. Don’t drag the boss when the details unfold. Just report the results to the boss
    • Understand things from the boss’s point of view

  • job-hopping

    • Consider personal development
    • Friendship in good gathering good coming days long
    • Treat the boys to dinner on the way out

  • How does team building work?

    • Capture flow capture result

    • Help brothers pave the way to serve brothers help brothers solve problems of cooperation, resources, needs sorting, personal growth