Today’s tech topic: What are Soft interrupts? .

What is an interrupt?

So what is an interrupt? In computers, interrupts are a mechanism by which the system responds to a request from a hardware device. The operating system receives an interrupt request from the hardware, interrupts the running process, and then calls an interrupt handler in the kernel to respond to the request.

This explanation may be too academic and confusing, but LET me give you an example of takeout in daily life.

Xiao Lin moved bricks at noon, hungry, ordered a white cut chicken takeaway, this time I took flash, not by a group of big data cooked. Although the delivery progress will be displayed on the platform, I can’t keep staring at it. Time is precious, so OF course I have to do other things. When the delivery is delivered, the deliveryman will notify me through the “phone”.

When I don’t get a call, I can do something else. Only when I get a call, which is an interruption, can I stop doing something else, which is getting takeout.

From this example, we can see that interrupt is an asynchronous event processing mechanism, which can improve the concurrent processing capability of the system.

When the operating system receives an interrupt request, it will interrupt the running of other processes. Therefore, the response program to the interrupt request, that is, the interrupt handler, should be executed as soon as possible to reduce the impact on the normal process running schedule.

In addition, interrupt handlers may also “temporarily close interrupts” when responding to interrupts. This means that no other interrupt request in the system can be answered until the current interrupt handler has finished executing. In other words, interrupts can be lost, so interrupt handlers should be short and fast.

Or back to the example of the take-away, xiao Lin at night lit a take-away again, this time in order to reward yourself, total points two delivery, a crayfish and a milk tea, and agent from different distribution to distribution, then the problem comes, when the first delivery to the shipping agent called me long, said some miscellaneous things, such as to give a high praise, and so on, But if another delivery guy wants to call me at the same time.

Obviously, the second deliveryman couldn’t get through to me because I was in the middle of a call (effectively turning off interrupt response), so he probably tried a few times and then walked away (effectively missing an interrupt).

What is a soft interrupt?

We also mentioned above, the interrupt request handler should be short and fast, so as to reduce the impact on the normal process scheduling to run, but they may interrupt handlers closed temporarily interrupted, at this time if the interrupt handler execution time is too long, in has not been performed before the interrupt handlers, lose the current interrupt request of other equipment.

In order to solve the problem of long interrupt handlers and lost interrupts, The Linux system divides the interrupt process into two phases, called “upper half and lower half”.

  • The upper part is used to quickly handle interrupts, usually temporarily closing interrupt requests, and is mainly responsible for hardware related or time sensitive things.
  • The lower half is used to delay processing of unfinished work in the upper half and is typically run as a “kernel thread”.

In front of the take-away example, as a result of the first distribution member for a long time to talk with me, would have resulted in a second marki unable to dial my phone, in fact, when I got a call from the first delivery member, you can told marki, I go downstairs now, the rest of the things, wait until we meet (upper), and then you can hang up the phone, downstairs, in the delivery, And other things to say to the deliveryman (second part).

That way, the first delivery person won’t take up too much time on my phone, and there’s a good chance the second one will call me when it happens.

Take another example from a computer, a common example of a network card receiving a network packet.

After the nic receives the network packet, it will notify the kernel of the arrival of new data through hardware interrupt, and the kernel will call the corresponding interrupt handler to respond to the event, and the processing of this event will also be divided into the upper and lower parts.

The first part is to do fast processing, so as long as the network card data read into memory, and then update the hardware register state, such as the status update to indicate that the data has been read into memory state value.

Then, the kernel will trigger a softirq, put some processing is time consuming and complicated things, to “soft interrupt handler”, also is to interrupt the bottom half, mainly is the need to find the network data from memory, again according to network protocol stack, network data parsing and processing step by step, finally to send data to the application.

So, the upper and lower parts of the interrupt handler can be interpreted as:

  • The upper part directly deals with hardware requests, that is, hard interrupts, which are mainly responsible for time-consuming work and characterized by fast execution;
  • The lower part is triggered by the kernel, also known as soft interrupt, is mainly responsible for the unfinished work in the upper part, usually time-consuming things, is characterized by delayed execution;

Another difference is that hard interrupts (top half) interrupt the CPU’s ongoing task and immediately execute the interrupt handler, while soft interrupts (bottom half) execute as a kernel thread and each CPU has a soft interrupt kernel thread, usually named “Ksoftirqd /CPU number”. For example, the name of the soft interrupt kernel thread for CPU 0 is Ksoftirqd /0

However, soft interrupts include more than just the lower half of the hardware interrupt handler. Some kernel custom events are also soft interrupts, such as kernel scheduling, RCU locks, and so on.

What are the soft interrupts in the system?

On Linux, you can check the contents of /proc/softirqs to see how soft interrupts are running and /proc/interrupts to see how hard interrupts are running.

/proc/softirqs = /proc/softirqs = /proc/softirqs = /proc/softirqs = /proc/softirqs = /proc/softirqs = /proc/softirqs

As you can see, each CPU has its own total number of runs for different types of soft interrupts. There are three things to note.

First, pay attention to the contents of the first column, which represents the types of soft interrupts. In my system, there are 10 types of soft interrupts, respectively corresponding to different work types. For example, NET_RX stands for network receive interrupt. NET_TX: network transmission interruption; TIMER: timing interruption; RCU: RCU lock interruption; SCHED: kernel scheduling interruption.

Second, note the distribution of the same type of soft interrupt on different cpus. Under normal circumstances, the cumulative number of the same type of soft interrupt on different cpus is similar. For example, in my system, the number of NET_RX interrupts on CPU0, CPU1, CPU2, and CPU3 are roughly the same order of magnitude.

Third, these values are the cumulative number of interrupts since the system was running. The value is not of any reference value, but the change rate of the number of interrupts is what we should pay attention to. We can use the watch -d cat /proc/softirqs command to check the change rate of the number of interrupts.

As mentioned earlier, the soft interrupt is executed by the kernel thread, we can use the ps command to check the result of the soft interrupt kernel thread on my server:

As you can see, the name of a kernel thread is surrounded by brackets, indicating that PS cannot retrieve its command-line arguments. Therefore, the name of a kernel thread in brackets is generally considered to be a kernel thread.

Also, you can see that there are four KsoftirQD kernel threads because my server has a CPU of four cores, one for each CPU core.

How to locate the problem that the SOFT interrupt CPU usage is too high?

To know the current system soft interrupt status, we can use the top command to view, the following is a server top data:

Si, shown in yellow, is the CPU usage on soft interrupts, and you can see that each CPU usage is not high. Although the utilization of the two cpus is only about 3% and 4%, they are used for soft interrupts.

In addition, it can also be seen that the process with the highest CPU usage is also the soft interrupt KsoftirqD, so it can be considered that the system overhead at this time is mainly from the soft interrupt.

To know which soft interrupt type is responsible, we can use the watch -d cat /proc/softirqs command to view the rate at which the number of interrupts for each soft interrupt type changes.

Generally, for Web servers with high NETWORK I/O, the change rate of NET_RX network receiving interrupts is much faster than that of other interrupt types.

If the change rate of NET_RX network receive interrupts is too fast, you can use the SAR -n DEV command to check the network receive rate of the network adapter, and then analyze which network adapter has a large number of network packets.

Then, use tcpdump to capture packets and analyze the source of these packets. If the packets are invalid addresses, you can add firewalls. If the packets are normal, you need to upgrade the hardware.

conclusion

To prevent interrupt handlers from taking too long to schedule normal processes, Linux divides interrupt handlers into upper and lower parts:

  • The upper part corresponds to hard interrupt, which is triggered by hardware and used to deal with interrupt quickly.
  • The lower part, corresponding to soft interrupt, is triggered by the kernel to asynchronously process the unfinished work of the upper part;

Linux soft interrupts include network transceivers, timing, scheduling, and RCU locks. You can view the total number of soft interrupts by viewing /proc/softirqs. To view the change rate of the number of soft interrupts in real time, You can run the watch -d cat /proc/softirqs command.

Each CPU has its own soft interrupt kernel thread. We can also use the ps command to view the kernel thread, which is usually named in parentheses and considered to be the kernel thread.

If the top command finds that CPU usage is high on soft interrupts, and the process with the highest CPU usage is also ksoftirqd, it is generally considered that the system overhead is occupied by soft interrupts.

Generally, it is caused by network receiving soft interrupts. If yes, you can use the SAR command to check which network adapter receives a large number of network packets, and then use tcpdump to capture network packets to further analyze whether the source of the network packets is an illegal address. If yes, consider adding rules to the firewall, if not, consider hardware upgrades, etc.

Source:zhuanlan.zhihu.com/p/338075214

omg

Dong Ge has written a series of articles illustrating network and operating system in Zhihu, and I am very glad to receive recognition and support from many zhihu friends. Recently, there have been more than 20 articles illustrating network and operating system, which can be regarded as a system.

Therefore, in order to facilitate zhihu friends to read, Dongge sorted his original graphic network and graphic operating system into PDF. After sorting out, I did not expect that each graphic output 150,000 words + 500 pictures, the quality is also great, many friends specially private letter to me, read my graphic and got the offer of big factory.

Graphical system PDF Open source download:

**300 pages of illustrated web PDF download! **

Liver for a month, finally completed 240,000 words Java interview manual

Finally, I wish you a bright future in the coding of the road.