While previous articles have discussed the impact of process context switches and system calls on system performance, today we’ll look at another CPU eater, soft interrupts.
When you look at system CPU consumption using VMstat or some other tool, there are two separate columns, hi and SI. They are hard interrupts and soft interrupts. Since VMstat lists the interrupt cost separately, there is no doubt that interrupts eat CPU.
It’s not necessary to understand everything about soft interrupts, but from a performance-oriented developer’s perspective, it’s important to understand the following:
- How much does a soft interrupt cost?
- How much CPU time is being eaten by soft interrupts on your server?
If you’re as curious as I am about the answers to the above questions, follow me!
Birth of soft interrupts
The CPU is normally focused on processing the user’s process. When external hardware or software wants to notify the CPU of a message, it does so through an interrupt Request (IRQ). For example, when you have a mouse click, or when the disk device has finished reading data, the CPU will be notified by interrupt that the work is complete.
However, when the interrupt mechanism is applied to network IO, there is a bit of a problem. After receiving the network packet, the processing work is not as simple as the mouse, keyboard and disk IO reading, but needs to go through a large number of kernel protocol stack processing, and finally put into the process of receiving cache. If only one interrupt (hard terminal) way to deal with network IO, because the hard interrupt priority is relatively high, so the CPU will be busy with a large number of network IO and can not respond to the keyboard and mouse in time, resulting in the real-time performance of the operating system is poor, you will feel the machine to card a card.
So modern Linux has invented software interrupts, with hard interrupts to handle network IO. Hard interrupts, as you can understand, are simply packets that can be collected, returned to the “home”, and completed quickly, without delaying the CPU’s response to other external high-priority interrupts. The soft interrupt has a lower priority and is responsible for processing packets from the driver layer to the network protocol stack, and finally putting the processed data into the Socker’s receive buffer.
Soft interrupts consume much more CPU cycles than hard interrupts, so we will focus on the overhead of soft interrupts in this article.
Soft interrupt cost estimation
The first overview of the soft interrupt context, good to jump directly into the topic of this article, how much soft interrupt overhead. Ok, please calculate with me:
1) Check the total time of soft interrupts. First, the cost proportion of soft interrupts on each core can be seen by using the top command, which is in the SI column
Top top-19:51:24 Up 78 days, 7:53, 2 Users, Load Average: 1.30, 1.35, 1.35 Tasks: 923 total, 2 running, 921 sleeping, 0 stopped, 0 zombie Cpu(s): Sy us 7.1%, 1.4%, 0.0% ni, 90.1% id, wa, 0.1%, 0.2% hi 1.2% si, 0.0% st Mem: 65872372k total, 64711668k used, 1160704k free, 339384k buffers Swap: 0k total, 0k used, 0k free, 55542632k cachedCopy the codeAs shown in the figure above, the CPU spends about 1.2% of its clock cycles on soft interrupts, which translates to 12ms per core.
2) Check the number of soft interrupts. Run the vmstat command to check the number of soft interrupts
$ vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------ r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 1231716 339244 55474204 0 0 6 496 0 0 7 3 90 0 0 2 0 0 1231352 339244 55474204 0 0 0 128 57402 24593 52 92 0 02 0 0 1230988 339244 55474528 0 0 140 55267 24213 52 93 0 02 0 0 1230988 339244 55474528 0 0 332 56328 23672 52 93 0 0Copy the codeThere are approximately 56,000 soft interrupts per second (this machine is a Web service, network IO intensive machine, and other interrupts are negligible).
This machine is a 16-core physical machine, so it can be concluded that the CPU time required by each soft interrupt is =12ms/(56000/16) times =3.428us
According to the experimental data, the CPU cost of a soft interrupt is about 3.4us
Context switching for soft interrupts
Above we have calculated a relatively accurate cost time. This time actually contains two parts, one is the overhead of context switch, the other is the overhead of soft interrupt kernel execution. Context switch has many similarities with system call and process context switch. Let’s put them in a simple comparison:
1. Compare with system call overhead
The opening sentence of “Understanding the Linux Kernel in Depth – Chapter 5” nicely links the unrelated concepts of interrupts and system calls, cleverly finding similarities between the two. “You can think of the kernel as a server that is constantly responding to requests, either from a process executing on the CPU or from an external device issuing an interrupt. A request from the boss is equivalent to an interrupt, while a request from the customer is equivalent to a system call from a user-mode process.
Soft interrupts, like system calls, are where the CPU stops the current user-mode context, saves the work scene, and then plunges into kernel mode to continue working. The only difference is that the system call switches to the kernel-mode context of the same process, while the soft interrupt switches to another kernel process, Ksoftirqd.
In fact, the early system calls were actually implemented with the assembly instruction int (interrupt). When a user-mode process issued int $0x80, the CPU switched to kernel mode and started to execute the system_call function. Later, it was decided that the system call was too slow because the int directive performed consistency and security checks. Later, the kernel used Intel’s sysenter instructions for “fast system calls” to become a bit less involved with interrupts. Soft interrupts must perform these checks, so the overhead of interrupts should be higher than the overhead of system calls at this point.
According to the experimental results above, the system call cost starts at 200ns. For those of you who haven’t read this article, follow me and look for it in the history article.
2. Compare with the process context switching cost
Compare this to A process context switch, which switches from user process A to user process B. The soft interrupt switch is from user process A to kernel thread Ksoftirqd. Ksoftirqd, as a kernel control path, has a lighter handler than a user process, so context switching is slightly more expensive than process switching. For those of you interested, you can continue to read chapter 5 of Understanding the Linux Kernel in Depth.
According to the experimental results above, the process context switch cost is 3US-5US. For those of you who haven’t read this article, follow me and look for it in the history article.
Related Linux commands
- The top: SI column shows the CPU overhead caused by soft interrupts
- Vmstat 1: in column displays the number of soft interrupts per second
- Cat /proc/softirqs: displays the total number of all soft interrupts, including TIMER, NET_TX, and NET_RX
Development of internal skills of the CPU
- 1. Do you think all your multicore cpus are eucore? Multicore “illusion”
- 2. I heard you only know memory, not cache? CPU says very sad!
- 3. What is TLB cache? How to check TLB miss?
- 4. What is the overhead of process/thread switching?
- 5. What makes coroutines better than threads?
- 6. How much CPU do soft interrupts eat you?
- 7. How much does a system call cost?
- 8. What is the overhead of a simple PHP request redis?
- 9. Do too many function calls cause performance problems?
My public account is “developing internal Skills and Practicing”. Here I am not simply introducing technical theories, nor only introducing practical experience. But to combine theory with practice, with practice to deepen the understanding of theory, with theory to improve your technical practice ability. Welcome you to follow my public number, also please share with your friends ~~~