The partition of this topic and the broker to which it belongs are as follows:

1. There are two brokers and two servers. 2. There are 6 partitions, which are distributed according to the hash modulus algorithm as shown in the figure. 3. There are 8 consumers, and they belong to the same consumer group.Copy the code

How do consumers in this consumer group access Kafka’s data if they follow this diagram?

Kafka’s consumer side has a balancing algorithm that looks like this:

1. A=(partition number/total number of consumers in the same group) 2. Calculate the patition set of the consumer pull data: Ci = [P(M* I)~P(I + 1) * m-1)]Copy the code

As shown in the figure, then here:

A = 6/8 = 0.75 M = 1 C0 = [P (1 * 0) - P (0 + 1) * (1-1)] = [P0 ~ P0] in the same way:  C1=[P(1*1)~P((1+1)*1-1)]=[P1~P1] C2=[P(1*2)~P((2+1)*1-1)]=[P2~P2] C3=[P(1*3)~P((3+1)*1-1)]=[P3~P3] C4=[P(1*4)~P((4+1)*1-1)]=[P4~P4] C5=[P(1*5)~P((5+1)*1-1)]=[P5~P5] C6=[P(1*6)~P((6+1)*1-1)]=[P6~P6] C7=[P(1*7)~P((7+1)*1-1)]=[P7~P7]Copy the code

Then follow the above algorithm:

C0 consumer consumes data P0, C1 consumer consumes data P1, C2 consumer consumes data P2, C3 consumer consumes data P3, C4 consumer consumes data P4, C5 consumer consumes data P5, C6 consumer consumes data P6, C7 consumer consumes data P7Copy the code

But partition only has P0-P5 and no P6 or P7 at all, so these two consumers will be idle, which is equivalent to occupying resources, but not useful, so what really plays a role here is C0-C5. As shown below:

What if the consumer group has fewer consumers than the partition (say 5)? 

A = 6/5 = 1.2 M = 2 C0 = [P (2 * 0) - P ((0 + 1) * 2-1)] = [P0 ~ P1] C1 = [P (2 * 1) - P ((1 + 1) * 2-1)] = [P2 and P3] C2 = [P (2 * 2) P ~ (2 + 1) * (2-1)] = [P4 ~ P5] C3=[P(2*3)~P((3+1)*2-1)]=[P6~P7] C4=[P(2*4)~P((4+1)*2-1)]=[P8~P9]Copy the code

And again, C3 and C4 don’t play any role.

As follows:

Conclusion:

1. If a kafka consumer group needs to increase the number of members to the same level as a partition, the number of additional members consumes resources.

2. The number of partitions in kafka must be greater than the number of consumer group members. The number of partitions must be 0 for consumer group members.

3. If you want to increase the number of members in the consumer group, adjust the number of partitions based on the preceding algorithm.

Reference source:Blog.csdn.net/qq_20641565…