The author of this article, Zhang Yanfei, the original title “127.0.0.1 of the local network communication process know how much”, first published in the “development of internal skills training”, reprint please contact the author. There are some changes this time.
1, the introduction
Following “Do you really know the difference between 127.0.0.1 and 0.0.0.0?” After that, this is the second article I put together about the basics of network programming on native networks.
This article is originally shared by the author Zhang Yanfei. The reason for writing this article is that now the local network IO application is very wide. In PHP, Nginx and php-fpm generally communicate via 127.0.0.1; In microservices, due to the application of Side Car pattern, more and more native network requests are made. Therefore, if we can understand this problem in depth in a variety of network communication application technology practice will be very meaningful.
Today let’s make clear 127.0.0.1 local network communication related issues!
For the sake of discussion, I’ve divided this question into three questions:
1) 127.0.0.1 Do I need a network card for local network IO?
2) Compared with external network communication, what is the difference in the kernel transceiver flow?
3) Can 127.0.0.1 be faster than 192.168.x?
The above several problems, I believe that including the regular mixed with instant communication network instant communication veterans, are seemingly very familiar with, but in fact still can not thoroughly speak clearly about the topic. This time, let’s find out once and for all!
(synchronous published in this article: http://www.52im.net/thread-36.)
2. A series of articles
This is the 13th article in a series that Outlines the following:
“Unknown Network Programming (I) : A Brief Analysis of the Troubled Diseases in TCP Protocol (Part I)”
Network Programming Unknown (Part II) : A Brief Analysis of TCP’s Troubled Diseases (Part II)
Time_wait, Close_wait when closing a TCP connection
Network Programming Unknown (Part 4) : An In-depth Analysis of TCP Abnormal Shutdowns
Unknown Network Programming (5) : Connectivity and Load Balancing for UDP
Network Programming Unknown (Part 6) : Understanding the UDP Protocol in depth and using it well
Unknown Network Programming (7) : How to Make Unreliable UDP Reliable?
Network Programming Unknown (8) : Deep Decryption of HTTP from the Data Transport Layer
Unknown Network Programming (9) : Theory Linked to Practice, All-around and In-depth Understanding of DNS
Unknown Network Programming (10) : Deep into the Operating System, Understanding Network Packet Receiving from the Kernel (Linux)
Network Programming Unknown (XI) : From the Bottom Up, Deep Analysis of the Secret of TCP Connection Time
Unknown Network Programming (12) : thoroughly understand the KeepAlive mechanism of the TCP protocol layer
“Unknown Network Programming (13) : Deep into the operating system, thoroughly understand 127.0.0.1 native network communication” (* article)
3. For comparison, take a look at cross-machine network communication
Before we begin the native communication process, let’s take a look at cross-machine network communication (using the implementation in the Linux kernel as an example).
3.1 Cross-machine data transmission
From the send system call until the network card sends the data, the overall flow is as follows:
In the above diagram, we see that the user data is copied to the kernel state and then processed by the protocol stack into the RingBuffer. Then the NIC driver actually sends the data. When the send is complete, the CPU is notified via a hard interrupt and the ringBuffer is cleaned up.
However, the above image doesn’t do a good job of showing the kernel components and source code, so let’s look at it from a code perspective again.
After the network has sent. The network card notifies the CPU with a hard interrupt when it completes the transmission. Receiving this hard interrupt frees the memory used in the RingBuffer.
3.2 Cross-machine Data Receiving The process of receiving a Linux packet begins when the packet arrives on another machine. (For a more detailed explanation, see “Deep into the Operating System, Understanding Network Packet Receiving from the Kernel (Linux)”.)
▲ The above diagram is from “Diving into the Operating System, Understanding the Network Package Receiving Process from the Kernel (Linux)”.
When the network card receives the data, the CPU initiates an interrupt to inform the CPU that the data has arrived. When the CPU receives the interrupt request, it will call the network driver registered interrupt handler function to trigger the soft interrupt. Ksoftirqd detects the arrival of a soft interrupt request, starts polling for a packet, and sends it to all levels of the protocol stack for processing after receiving it. After the stack is processed and the data is placed on the receive queue, the user process is awakened (presumably in blocking mode).
Let’s look again from the kernel component and source point of view.
3.3 Cross-machine Network Communication Summary The understanding of cross-machine network communication can be summarized by the following picture:
4. The sending process of local network data
In the previous section, we saw the process of sending the entire network data across machines.
In the case of native network IO, the flow is somewhat different. For the sake of focus, this section will not cover the overall process, but only the points that differ from the cross-machine logic. There are two areas of difference, routing and drivers.
4.1 Network Layer Routing When sending data will enter the protocol stack to the network layer, the network layer entry function is IP_QUEUE_XMIT. Routing will be carried out in the network layer. After the routing is completed, some IP headers will be set, some netfilter filtering will be carried out, and the packet will be handed to the neighbor subsystem.
The special thing about native network IO is that the route entry is found in the LOCAL routing table, and the corresponding device will use the loopback card, which is also known as LO.
Let’s take a closer look at this routing-related process in the routing network layer. Start with the network layer entry function ip_queue_xmit.
//file: net/ipv4/ip_output.c
intip_queue_xmit(struct sk_buff skb, struct flowi fl)
{
// Check if there is a cached routing table in the socket
rt = (struct rtable *)__sk_dst_check(sk, 0);
if(rt == NULL) {
// Expand the lookup if there is no cache
// finds the route entry and caches it into the socket
rt = ip_route_output_ports(…) ;
sk_setup_caps(sk, &rt->dst);
}
The function to find the route entry is IP_ROUTE_OUTPUT_PORTS, which in turn calls IP_ROUTE_OUTPUT_FLOW, __IP_ROUTE_OUTPUT_KEY, FIB_LOOKUP. The call procedure is omitted and you look directly at the key code of FIB_LOOKUP.
//file:include/net/ip_fib.h
static inline int fib_lookup(struct net net, const struct flowi4 flp, struct fib_result *res)
{
struct fib_table *table;
table = fib_get_table(net, RT_TABLE_LOCAL);
if(! fib_table_lookup(table, flp, res, FIB_LOOKUP_NOREF))
return 0;
table = fib_get_table(net, RT_TABLE_MAIN);
if(! fib_table_lookup(table, flp, res, FIB_LOOKUP_NOREF))
return 0;
return -ENETUNREACH;
}
FIB_LOOKUP will expand the query on the LOCAL and MAIN routing tables, and the query will be local first and main second. We can view both routing tables on Linux using command names, but here we only look at the LOCAL routing table (because native network IO stops querying this table).
ip route list table local
Local10.143.x.y dev eth0 proto kernel scope host SRC 10.143.x.y
Local127.0.0.1 dev lo proto kernel scope host SRC 127.0.0.1
Routing for destination 127.0.0.1 can be found in the local routing table. FIB_LOOKUP completes and returns __ip_route_output_key to continue.
//file: net/ipv4/route.c
struct rtable __ip_route_output_key(struct net net, struct flowi4 *fl4)
{
if(fib_lookup(net, fl4, &res)) {
}
if(res.type == RTN_LOCAL) {
dev_out = net->loopback_dev;
.
}
rth = __mkroute_output(&res, fl4, orig_oif, dev_out, flags);
return rth;
}
For network requests that are native, the device will all use net->loopback_dev, which is the LO virtual network card.
The next network layer is still the same as the cross-machine network IO, eventually going past ip_finish_output and eventually into the neighbor subsystem’s entry function dst_neigh_output.
Is IP sharding required for native network IO? Because it goes through the ip_finish_output function as normal network layer processing. In this function, if SKB is larger than MTU, it will still be sharded. It’s just that LO’s MTU is much larger than Ethernet’s. Through ifconfig command can be found, the ordinary network card is generally 1500, and lO virtual interface can have 65535.
It is processed in the neighbor subsystem function to enter the network device subsystem (the entry function is dev_queue_xmit).
The entry function of the network device subsystem is dev_queue_xmit. Recall that for a physical device that actually has a queue, there is a complex queuing process in this function before calling dev_hard_start_xmit, from which the driver is sent.
During this process, it is even possible to trigger a soft interrupt for sending, as shown in the following diagram:
But for booted loop devices (q->enqueue evaluates to false), it’s much simpler: there’s no queue problem, just go straight to dev_hard_start_xmit. The SKB is then “sent” by going to loopback_xmit, the send callback function loopback_xmit in the “driver” of the loopback device.
Let’s look at the process in detail, starting with the network device subsystem entry dev_queue_xmit.
//file: net/core/dev.c
int dev_queue_xmit(struct sk_buff *skb)
{
q = rcu_dereference_bh(txq->qdisc);
If (q->enqueue) {// False for loop back devices
rc = __dev_xmit_skb(skb, q, dev, txq);
goto out;
}
// Start loop return device processing
if(dev->flags & IFF_UP) {
dev_hard_start_xmit(skb, dev, txq, …) ;
.
}
}
The device driver’s action functions are still called in dev_hard_start_xmit.
//file: net/core/dev.c
int dev_hard_start_xmit(struct sk_buff skb, struct net_device dev, struct netdev_queue *txq)
{
// Gets the device driver callback function set ops
const struct net_device_ops *ops = dev->netdev_ops;
// Call the driver ndo_start_xmit to send
rc = ops->ndo_start_xmit(skb, dev);
.
}
4.3 “drive” program For real igb card, its driver code are drivers/net/Ethernet/Intel/igb/igb_main c file. Following this path, I found the “driver” code location of the loopback device: drivers/net/loopback.c.
The drivers/net/loopback. C:
//file:drivers/net/loopback.c
static const struct net_device_ops loopback_ops = {
.ndo_init = loopback_dev_init,
.ndo_start_xmit = loopback_xmit,
.ndo_get_stats64 = loopback_get_stats64,
};
So the call to dev_hard_start_xmit actually executes loopback_xmit in the loopback “driver”.
The reason why I put the word “driver” in quotes is that Loopback is a software-only virtual interface that does not actually have a driver, and its workflow is roughly as follows.
Let’s look at the code in detail.
//file:drivers/net/loopback.c
static netdev_tx_t loopback_xmit(struct sk_buff skb, struct net_device dev)
{
// Remove the connection from the original socket
skb_orphan(skb);
/ / call netif_rx
if(likely(netif_rx(skb) == NET_RX_SUCCESS)) {
}
}
In skb_orphan, the socket Pointers on SKB are first removed (stripped out).
Note: During the local network IO transmission process, the SKB under the transport layer does not need to be released. It can be passed directly to the receiver. It saved me a little money. But unfortunately the SKB of the transport layer is also not economical, and still needs to be applied and released frequently.
It then calls netif_rx, which in this method is eventually executed in enqueue_to_backlog (netif_rx-> netif_rx_internal -> enqueue_to_backlog).
//file: net/core/dev.c
static int enqueue_to_backlog(struct sk_buff skb, int cpu, unsigned int qtail)
{
sd = &per_cpu(softnet_data, cpu);
.
__skb_queue_tail(&sd->input_pkt_queue, skb);
.
____napi_schedule(sd, &sd->backlog);
In ENQUEUE_TO_BACKLOG, insert the SKB to be sent into the softnet_data->input_pkt_queue and call ____ NAPI_SCHEDULE to trigger the soft interrupt.
//file:net/core/dev.c
static inline void ____napi_schedule(struct softnet_data sd, struct napi_struct napi)
{
list_add_tail(&napi->poll_list, &sd->poll_list);
__raise_softirq_irqoff(NET_RX_SOFTIRQ);
}
The sending process is complete only after the soft interrupt is triggered.
5. Receiving process of local network data
5.1 Main Process In the receiving process of network packets across machines, hard interrupts are needed before soft interrupts can be triggered.
In the process of local network IO, because the network card is not really passed, so the actual transmission of the network card, hard interrupts are omitted. It starts directly from the soft interrupt, goes through PROCESS_BACKLOG and is sent to the protocol stack. The general process is shown in the figure below.
5.2 Detailed Process Let’s move on to a more detailed process.
After the soft interrupt is triggered, it goes to the NET_RX_SOFTIRQ corresponding processing method net_rx_action (see section 4.2 of the article “Dig into the operating system and understand the network packet receiving process from the kernel” for details).
//file: net/core/dev.c
static void net_rx_action(struct softirq_action *h){
while(! list_empty(&sd->poll_list)) {
work = n->poll(n, weight);
}
}
Remember that for an IGB network card, poll actually calls the IGB_poll function.
So who is the poll function for the loopback card? Since poll_list is a struct softnet_data object, we can find something in net_dev_init.
//file:net/core/dev.c
static int __init net_dev_init(void)
{
for_each_possible_cpu(i) {
sd->backlog.poll = process_backlog;
}
}
The default poll for struct softnet_data is initialized with the process_backlog function, so let’s see what it does.
static int process_backlog(struct napi_struct *napi, int quota)
{
while(){
while((skb = __skb_dequeue(&sd->process_queue))) {
__netif_receive_skb(skb);
}
// The skb_queue_splice_tail_init() function connects list A to list B,
// Make a new list b and change the head of a to an empty list.
qlen = skb_queue_len(&sd->input_pkt_queue);
if(qlen)
skb_queue_splice_tail_init(&sd->input_pkt_queue, &sd->process_queue);
}
}
This time let’s look at the call to skb_queue_splice_tail_init. SD-> input_pkt_queue is linked to the sD-> process_queue list.
Then look at __skb_dequeue, __skb_dequeue is processed from the sd->process_queue. This corresponds to the previous end of the sending process. The sending process puts the packet into the input_pkt_queue, and the receiving process takes the SKB out of the queue.
Finally __netif_receive_skb is called to send the SKB (data) to the protocol stack. After that, the call process is consistent with cross-machine network IO again.
The call chain that goes to the protocol stack is __netif_receive_skb => __netif_receive_skb_core => Deliver_skb and sends the packet to IP_RCV (see “Deep into the Operating System,” Understanding the receiving process of network packets from the kernel (the Linux article), Section 4.3.
The network is then followed by the transport layer, and finally the wake up user process, which will not expand much here.
6. Summary of the local network communication process
Let’s summarize the kernel execution process of native network communication:
Recall the flow of cross-machine network IO as follows:
Well, back to business, we can finally answer the first three questions in a separate chapter.
7. Answer to the first three questions
1) Question 1:127.0.0.1 Does local network IO need a network card?
From the description of this article, we can definitely conclude that there is no need to go through the network card. Even if the network card is pulled out of the local network can also be used normally.
2) Question 2: What is the path of the packet in the kernel and what is the difference in the flow compared to the outgoing packet?
Overall, native network IO does save some overhead when compared to cross-machine IO. Instead of sending data into the driver queue of the RingBuffer, the SKB is sent directly to the Receive stack (via a soft interrupt).
But nothing is missing from the rest of the kernel: system calls, protocol stacks (transport layer, network layer, etc.), network device subsystems, and neighbor subsystems. Even the “driver” program is gone (though to the loop device it is only a pure software dummy). So even with native network IO, don’t be fooled into thinking there’s no overhead.
3) Question 3: Can 127.0.0.1 be faster than 192.168.x?
Conclusion first: I don’t think there’s a performance difference between the two usage methods.
I think a significant number of people will think that 127.0.0.1 is faster to access the native Server. The reason is that it is intuitively assumed that access to IP passes through the network card.
In fact, the kernel knows all the IP on the machine, so long as the destination address is found to be the local IP, it can loopback all the loopback devices. Other local IP and 127.0.0.1, is also not through the physical network card, so access to their performance costs are basically the same!
Appendix: More Network Programming Series
If you find this series too technical, you may want to read the “Lazy Introduction to Web Programming” series, which is cataloged as follows:
“A Lazy Introduction to Network Programming (Part 1) : Fast Understanding of Network Communication Protocols (Part 1)”
A Lazy Introduction to Network Programming (Part 2) : Quickly Understanding Network Communication Protocols (Part 2)
“A Lazy Introduction to Network Programming (III) : A Quick Understanding of TCP in One Paper”
A Lazy Introduction to Network Programming (Part 4) : Quickly Understanding the Difference between TCP and UDP
A Lazy Introduction to Network Programming (Part 5) : A Quick Understanding of Why UDP Sometimes Has an Advantage Over TCP
A Lazy Introduction to Network Programming (6) : The Most Popular Introduction to Hubs, Switches, and Routers in History
“A Lazy Introduction to Web Programming (7) : A Comprehensive Understanding of the HTTP Protocol.”
“Network programming lazy introduction (8) : hand by hand teach you to write TCP based Socket long connection”
“Network programming lazy introduction (9) : general explanation, have IP address, why still use MAC address?”
“The Lazy Intro to Network Programming (10) : Quick Reading of the QUIC Protocol in the Time of a Piss”
“A Lazy Introduction to Network Programming (11) : What IPv6 Is
“A Lazy Introduction to Network Programming (12) : A Fast Reading of the HTTP /3 Protocol, One Paper Enough!”
This site “brain-damaged network programming introduction” is also suitable for entry learning, this series outline is as follows:
The Insane Network Programming 101: Learn TCP Three Handshakes and Four Waves with Animation
Brainless Introduction to Network Programming (2) : What Are We Reading and Writing When We Read and Write Sockets?
“Brainy Introduction to Network Programming (Part 3) : What You Must Know About the HTTP Protocol”
“A Brainy Introduction to Network Programming: A Quick Understanding of HTTP/2 Server Push”
The Brainless Introduction to Network Programming (5) : The Ping command you use every day. What is it?
What is Public IP and Intranet IP and What is NAT Conversion?
“Brainy type network programming introduction (7) : face as necessary, the history of the most popular computer network stratification detail”
Do you really know the difference between 127.0.0.1 and 0.0.0.0?
“Brainy type network programming introduction (9) : interview required test, the history of the most popular small and small end byte order details”
The following information is from the “TCP/IP detail”, beginners must read:
Chapter 11 UDP: User Datagram Protocol
TCP: Transmission Control Protocol Chapter 17
TCP/IP Detail – Chapter 18: TCP Connection Creation and Termination
TCP/IP in detail – Chapter 21: TCP Timeouts and Retransmissions
The following series is suitable for server network programming developers:
“High Performance Network Programming (I) : How many concurrent TCP connections can a single server have”
“High Performance Network Programming (II) : The Notable C10K Concurrent Connection Problem of the Last Decade”
“High Performance Network Programming (III) : In the Next Decade, It’s Time to Consider C10M Concurrency”
“High Performance Network Programming (IV) : Theoretical Exploration of High Performance Network Applications from C10K to C10M”
“High Performance Network Programming (Part 5) : Understanding the I/O Model in High Performance Network Programming”
“High Performance Network Programming (Part 6) : Understanding Threading Models in High Performance Network Programming”
High Performance Network Programming (7) : What exactly is High Concurrency?
“Understanding the Roots of High Performance and High Concurrency (I) : Understanding Threads and Thread Pools at the Bottom of Computers”
“Understanding the Roots of High Performance and High Concurrency (Part 2) : Understanding I/O and Zero-Copy Technologies in Operating Systems”
“Understanding the Roots of High Performance and High Concurrency (III) : Diving Deep into Operating Systems for a Thorough Understanding of I/O Multiplexing”
“Understanding the Roots of High Performance and High Concurrency (Part 4) : A Deep Understanding of Operating Systems for a Thorough Understanding of Synchronization and Asynchrony”
“Understanding the Roots of High Performance and High Concurrency (5) : Understanding Coroutines in High Concurrency by Diving Deep into Operating Systems”
“Understanding the Roots of High Performance and High Concurrency (6) : In Plain English, How High Performance Servers Are Implementing”
“Understanding the Roots of High Performance and High Concurrency (7) : An In-depth Understanding of the Operating System, Processes, Threads and Coroutines”
The following series is suitable for senior mobile network communication developers to read:
IM Developers’ Introduction to Zero-Fundamentals (Part 1) : A Centennial History of Communication Switching (Part 1)
IM Developers’ Introduction to Zero-Fundamentals (Part 2) : A Centennial History of Communication Switching (Part 2)
IM Developers’ Introduction to Zero-based Communication Technology (III) : The Transformation of Chinese Communication Mode in a Century
IM Developers’ Introduction to Zero-based Communications (Part 4) : The Evolution of the Mobile Phone, the Most Complete History of Mobile Terminals
Introduction to Zero-based Communication Technologies for IM Developers (5) : A 30-year History of Mobile Communications Technology Evolution from 1G to 5G
Introduction to Zero-based Communication Technology for IM Developers (VI) : The Connector of Mobile Terminals — “Base Station” Technology
IM Developers’ Introduction to Zero-based Communication Technology (VII) : The Chollima of Mobile Terminals — “Electromagnetic Wave”
IM Developers’ Introduction to Zero-Based Communications (8) : Zero-based, Ever Strongest “Antenna” Principle Literacy
Introduction to Zero-based Communication Technology for IM Developers (9) : The “Core Network”, the Hub of Wireless Communication Networks
Zero Basics for IM Developers (10) : Zero Basics, the Best 5G Technology Literacy Ever
IM Developers’ Zero Fundamentals: Why WiFi Signal Is Bad?
IM Developer’s Zero Basics Communication Technology Introduction (XII) : Network Stuck? Network Down?
IM Developers’ Zero Basics in Communication Technology (13) : Why Is Cell Phone Signal Bad?
A Zero Basics Communication Technology Introduction for IM Developers (XIV) : How Hard Is Wireless Internet Access on High-Speed Rail?
“A Zero Basics Communication Technology Introduction for IM Developers (XV) : Understanding Location Technologies, One Paper Enough”
This article has been published simultaneously on the public account of “instant messaging technology circle”.
▲ The link to this article on the public number is: click here to enter. The synchronous publishing link is:http://www.52im.net/thread-36…
