Process of receiving network packets

  1. When a network frame arrives at the nic, the NIC puts the network frame into the packet receiving queue through DMA. A hard interrupt is then used to tell the interrupt handler that the network packet has been received.
  2. Next, the NIC interrupt handler allocates the kernel data structure (SK_buff) for the network frame and copies it into the SK_buff buffer; A soft interrupt notifies the kernel that a new network frame has been received.
  3. Next, the kernel protocol stack fetches the network frame from the buffer and processes the network frame layer by layer through the network protocol stack:
    • Check the validity of the packet at the link layer, find out the upper-layer protocol type (such as IPv4 or IPv6), remove the frame header and frame tail, and submit the packet to the network layer.
    • The network layer takes out the IP header and determines the next direction of the network packet, such as whether to hand it to the upper layer for processing or forward. When the network layer determines whether the packet is to be sent to the local computer, it fetches the type of the upper-layer protocol (such as TCP or UDP), removes the IP header, and passes it to the transport layer for processing.
    • After receiving the TCP or UDP header, the transport layer identifies the corresponding Socket based on the quad of Source IP address, Source port, Destination IP address, and destination port, and copies the data to the Socket receiving cache.
  4. Finally, the application can use the Socket interface to read the newly received data.

Process of sending network packets

After you understand the receiving process of network packets, you can easily understand the sending process of network packets. The flow of network packets is shown on the right side of the figure. It is easy to see that network packets are sent in the opposite direction from receiving.

First, the application calls the Socket API (such as SendMsg) to send the network packet.

Since this is a system call, it falls into the kernel-state socket layer. The Socket layer puts packets into the Socket send buffer.

Next, the network protocol stack fetches the packet from the Socket send buffer. The TCP/IP stack is then processed layer by layer from top to bottom. For example, at the transport layer and network layer, add the TCP header and IP header respectively, perform route lookup to determine the IP address of the next hop, and fragment the packets based on the MTU size.

The fragmented network packets are sent to the network interface layer for physical address addressing to find the MAC address of the next hop. Then add the frame header and the frame tail to the packet queue. When this is done, a soft interrupt notifies the driver that a new network frame needs to be sent in the outgoing packet queue.

Finally, the driver reads the network frame from the sending queue via DMA and sends it through the physical card.

Questions about the location of the send/receive queue and buffer

The process of receiving and sending network packets involves multiple queues and buffers, including:

  • The circular buffer that the nic interacts with through DMA when sending and receiving network packets.
  • The KERNEL data structure SK_buff buffer allocated by the NIC interrupt handler for network frames;
  • A socket buffer when an application interacts with the network protocol stack through a socket interface.

But correspondingly, there are two problems.

First of all, where are these buffers? Is it in the nic hardware, or in memory? The problem is easy to understand when you think about it — these buffers are in kernel managed memory.

The ring buffer, which requires DMA to interact with the nic, should be the domain of the nic device driver.

The SK_buff buffer is a bidirectional linked list that maintains the network frame structure. Each element in the linked list is a network frame (Packet). Although the TCP/IP stack has several layers, passing between the upper and lower layers actually requires manipulation of Pointers in the data structure without data replication.

Socket buffers allow applications to configure receive or send buffers of different sizes for each socket. The application sends data, essentially writing it to a buffer; When you receive data, you actually read it from the buffer. Further processing of the data in the buffer is done by TCP or UDP at the transport layer.

Second, do these buffers have anything to do with buffers and caches?

This question is not difficult to answer. According to the knowledge of memory modules, buffers mentioned in memory are directly related to block devices; Everything else is Cache.

In practice, SK_buffs, socket buffering, connection tracing, and so on are all managed by the SLAB allocator. You can check how much memory they occupy by going directly to /proc/slabinfo.