This is the 25th day of my participation in Gwen Challenge

1 Kafka Indicates the size of a single log transmission

The default Kafka message body size is 1 MB. However, in our application scenario, it is common for a message to be larger than 1 MB. If Kafka is not configured. Server.properties Replica.fetch. Max. Bytes: Server.properties replica.fetch. 1048576 Maximum number of bytes that the broker can copy messages. Default: 1M message.max.bytes: Bytes must be smaller than or equal to ‘replica.fetch. Max. Bytes. Otherwise, data synchronization between replicas fails.

2 Clear expired Kafka data

Log cleanup.policy = delete Enables the deletion policy log.cleanup.policy = compact Enables the compression policy

Kafka can consume data over time

Map<TopicPartition, OffsetAndTimestamp> startOffsetMap = KafkaUtil.fetchOffsetsWithTimestamp(topic, sTime, kafkaProp);

4 Kafka consumers consider whether to pull data or push data

Pull the data

5 Kafka partition allocation policy

There are two default partition allocation policies in Kafka: Range and RoundRobin.

Range is the default policy. Range is for each Topic (i.e. one Topic for one Topic). Firstly, it sorts the zones within the same Topic according to the serial number, and sorts the consumers in alphabetical order. Then divide the number of Partitions by the total number of consumer threads to determine how many Partitions each consumer thread consumes. If not, the first few consumer threads will consume one more partition.

For example: we have 10 partitions, two consumers (C1, C2), three consumer threads, 10/3 = 3 and can’t divide. C1-0 will consume partitions 0, 1, 2, 3 C2-0 will consume partitions 4, 5, 6 C2-1 will consume partitions 7, 8, 9

Step 1: Form all the topic partitions into a TopicAndPartition list, then sort the TopicAndPartition list according to hashCode, and finally send it to each consuming thread in a polling manner.

6 Kafka hang up

  1. The Flume record
  2. Log recorded
  3. Short-term ok

7 Kafka cannot lose data

  1. Ack = 0Is equivalent to asynchronous sending. When the message is sent, the offset increases and production continues.
  2. Ack = 1After receiving the ACK received by the leader Replica on a message, the leader adds the offset and continues production.
  3. Ack = -1, the leader adds the ack offset after receiving all the replicas’ acceptance of a message and continues production.

8 Duplicate Kafka data

  1. Idempotent + ACK -1 + transaction
  2. The DWD layer in SparkStreaming, Redis or Hive can be de-weighted by grouping, opening Windows according to id and only taking the first value.

9 Kafka message data backlog, Kafka consumption capacity is insufficient.

  1. If Kafka has insufficient consumption power, consider increasing the number of Topic partitions and simultaneously increasing the number of consumers in the consumer group. Number of consumers = number of partitions. (Both are indispensable)
  2. If downstream data processing is not timely: increase the number of pulls per lot. If the batch pull data is too small (pull data/processing time < production speed), the processed data is smaller than the production data, which also causes data backlog.

10 Kafka log retention time

The default value is 7 days. Three days is recommended for production environment

11 Data volume calculation in Kafka

Total data volume 100G per day, 100 million logs per day, 10,000,000/24/60/60 =1150 logs per second Average per second: 1150 valleys per second: 50 peaks per second: 1150 * (2-20 times) = 2,300-23000 logs per log size: 0.5K-2K (take 1K) How much data per second: 2.0m-20MB

12 Kafka hard disk size

Data volume per day 100G * 2 copies * 3 days / 70%

13 Kafka monitoring

Open source monitors developed by the company: KafkaManager, KafkaMonitor, KafkaEagle

14 Number of Kakfa partitions

  1. Create a topic with only 1 partition
  2. Test the Producer throughput and consumer throughput for this topic.
  3. Suppose they are Tp and Tc, respectively, in MB/s.
  4. Then assume that the total target throughput is Tt, so the number of partitions =Tt/min (Tp, Tc)

For example, producer throughput = 20m/s. Consumer throughput = 50m/s, expected throughput 100m/s; Number of partitions = 100/20 = 5 Number of partitions The value ranges from 3 to 10

15 How many topics

As usual: There are as many topics as there are log types. There are also merging log types.

16 Kafka ISR copy synchronization queue

In-sync Replicas (ISR) : specifies the replica synchronization queue. The ISR includes the Leader and Follower. If the Leader process fails, a service in the ISR queue is selected as the new Leader. There are two parameters replica.lag.max.messages (delay number) and replica.lag.time.max.ms (delay time) to determine whether a service can be added to the ISR replica queue. In version 0.10, the replica.lag.max.messages parameter was removed to prevent frequent queuing of services.

When any dimension exceeds the threshold, the followers are removed from the ISR and stored in the Outof-Sync Replicas (OSR) list. New followers are also stored in the OSR.

17 Number of Kafka machines

Number of Kafka machines = 2 * peak production speed * copies / 100 + 1

18 Set the number of copies

Usually set it to 2 or 3, and many companies set it to 2. Advantages of duplicates: improved reliability; Copy disadvantage: Increased network IO transport