Abstract:Server side of Kafka network module, introduces the process of Server side starting, receiving and processing requests.

This article is shared from Huawei Cloud Community “Kafka Network Module -Server Side”, the original author: middleware brother.

SocketServer is the module on the Kafka Server side that handles requests, created, initialized, and started during the Kafka startup process.

SocketServer startup process:

  • Initialize the acceptors in EndPoint order, one for each EndPoint Acceptor, and create processors for each Acceptor (the number is determined by the num.net World.Threads configuration item). An Acceptor will then listen for connections through a selector and pass the newly established connections to the Processor for processing (polling the Processor).
  • Initiate any Processor Acceptor that initiates or listens for connections:
  • After an Acceptor is started, a ServerSocketChannel is created to listen to the Acceptor’s endpoint, register OP_ACCEPT on the selector, and enter an infinite loop. Each time an Acceptor is executed, a ServerSocketChannel is created. Get the ready key (ServerSocketChannel) from the selector, indicate that a connection is coming, and then create a SocketChannel corresponding to that connection by accepting (). It then polls one of the processors responsible for the acceptor, assigning the SocketChannel to the chosen processor, and assigning the connection to the processor.
  • Acceptor sends a connection to the processor by adding a SocketChannel to the processor’s connection queue, NewConnection, which the processor will continuously retrieve and process in its run method.
  • After the Processor gets the SocketChannel from NewConnection, it registers OP_READ on the selector and creates the corresponding Kafkachannel.

The process of receiving and processing the request by the Server side:

  • When the Processor is ready to receive an OP_READ event, check and try to complete the SSL handshake and SASL verification. (The handshake may not be complete at this time, so if the Processor is ready to receive an OP_READ event, check and make sure the handshake is complete.) SSL/SASL Reference Section 9.4)
  • After the SSL handshake and SASL validation are complete, the data is read from the Channel, the NetworkReceive object is constructed, and the stagedReceives are merged
  • Remove the StagedReceives team head element (removed) and add the completedReceives
  • Take (not remove) the elements in the completedTransaction, construct the Request object, add the requestQueue, remove the event registration for OP_READ, and give the corresponding KafkaChannel as the fraternal, Buy for MUTED_AND_RESPONSE_PENDING again
  • KafKareQuessHandler removes the element from the RequestQueue and hands it to the KafkaAPI module to handle the request
  • After processing the request, KafkaAPI puts the response into the responseQueue and InflightResponses of the corresponding processor, and awakens its selector
  • The Processor fetches (removes) the response from the responseQueue. If the response needs to be sent back to the client, the Processor assigns the send of the response to Kafkachannel and registers the OP_WRITE event
  • When the channel is written, write send to the channel’s write buffer. When the send is written, remove the registration for the OP_WRITE event and add send to the completedSends
  • The corresponding responses were removed from the INFLIGHTResponses, the response callbacks were executed, Kafkachannel was given as the fraternal, then the Not_fraternal from the fraternal, and the OP_READ event registration was added again

Click on the attention, the first time to understand Huawei cloud fresh technology ~