preface
As a backend developer, what do you do when you receive an alarm that the online server CPU load is too high? Restart the service and ignore the alarm? But in my opinion, a qualified engineer must locate the specific problem and fix it. The following is a record of the troubleshooting process for the high CPU load of the online server. The troubleshooting process is clear. In the future, problems will be located and solved quickly.
What are the scenarios that lead to high online CPU load?
Common scenarios at the code level are:
- The program is stuck in an endless loop, constantly consuming CPU
- Thread deadlocks, threads waiting for each other, causing a state of suspended animation, constantly consuming CPU
Program infinite loop scenario
Here we use the JAVA simple simulation of the system’s high load caused by an infinite loop, the code is as follows:
/ * * *@program: easywits
* @description: HashMap test in concurrent mode.... *@author: zhangshaolin
* @create: the 2018-12-19 and * * /
public class HashMapMultiThread {
static Map<String, String> map = new HashMap<>();
public static class AddThread implements Runnable {
int start = 0;
public AddThread(int start) {
this.start = start;
}
@Override
public void run(a) {
// An infinite loop is used to simulate the scenario of high CPU usage
while (true) {
for (int i = start; i < 100000; i += 4) { map.put(Integer.toString(i), Integer.toBinaryString(i)); }}}public static void main(String[] args) throws InterruptedException {
// The thread concurrently puts the HashMap, and if all goes well, map.size() is 100000
// Possible results:
//1. The program is normal and the result is 100000
//2. The procedure is normal and the result is less than 100000
Thread thread1 = new Thread(new AddThread(0), "myTask-1");
Thread thread2 = new Thread(new AddThread(1), "myTask-2");
Thread thread3 = new Thread(new AddThread(2), "myTask-3");
Thread thread4 = new Thread(new AddThread(3), "myTask-4"); thread1.start(); thread2.start(); thread3.start(); thread4.start(); thread1.join(); thread2.join(); thread3.join(); thread4.join(); System.out.println(map.size()); }}}Copy the codeThread deadlock scenario
Also use JAVA program to simply simulate the thread deadlock scenario, the code is as follows:
/ * * *@program: easywits
* @description: deadlock demo.... * 1. Two threads hold two Object objects: lock1 and lock2. These two locks act as locks for synchronized code blocks; Thread.sleep(XXX), thread.sleep (XXX), thread.sleep (XXX), thread.sleep (XXX) * This is mainly to prevent thread 1 from starting the lock1 and lock2 objects in a row * 3. * <p> * thread 1 is waiting for thread 1 to release the lock1 object lock. * < P > * thread 1 is waiting for thread 1 to release the lock1 object lock. Thread 2 has acquired the object lock of lock2, thread 1 attempts to acquire the object lock of lock2, and a deadlock is formed. *@author: zhangshaolin
* @create: the 2018-12-20 o * * /
public class DeadLock {
static Object lock1 = new Object();
static Object lock2 = new Object();
public static class Task1 implements Runnable {
@Override
public void run(a) {
synchronized (lock1) {
System.out.println(Thread.currentThread().getName() + "Got the first lock!!");
try {
Thread.sleep(50);
} catch (InterruptedException e) {
e.printStackTrace();
}
synchronized (lock2) {
System.out.println(Thread.currentThread().getName() + "Got the second lock!!"); }}}}public static class Task2 implements Runnable {
@Override
public void run(a) {
synchronized (lock2) {
System.out.println(Thread.currentThread().getName() + "Got the second lock!!");
synchronized (lock1) {
System.out.println(Thread.currentThread().getName() + "Got the first lock!!"); }}}}public static void main(String[] args) throws InterruptedException {
Thread thread1 = new Thread(new Task1(), "task-1");
Thread thread2 = new Thread(new Task2(), "task-2");
thread1.start();
thread2.start();
thread1.join();
thread2.join();
System.out.println(Thread.currentThread().getName() + "Execution complete!"); }}Copy the codeAfter the above two scenarios code execution, no accident, the system CPU load will soar, my machine, the 4-core CPU has obviously felt stuck, so the line should eliminate the emergence of dead loop code.
usetopCommand to monitor the current system load
Execute the first scenario test code.
After entering the top command in the Linux command line, the load information of the current system will be monitored in real time. The monitored load information is as follows:
From the monitoring information in the figure, you can quickly and roughly understand that the process with a PID of 17499 has a CPU load of 328+% and is a JAVA program. The following describes the monitoring information:
- PID: indicates the ID of a process
- USER: indicates the process owner
- PR: indicates the priority level of the process. The smaller the priority, the higher the execution priority
- VIRT: virtual memory occupied by a process
- RES: physical memory occupied by a process
- SHR: shared memory used by a process
- S: indicates the process status. S means sleep, R means running, Z means dead, and N means the process has a negative priority value
- %CPU: indicates the CPU usage of a process
- %MEM: The percentage of physical memory and total memory used by the process
- TIME+ : indicates the total CPU TIME occupied by the process after it is started
Press the keyboard number 1 on the monitoring page to see the load of each CPU, as shown below:
You can see that with four threads open and an infinite loop, I have four core cpus on my machine, each loaded close to 100 percent.
usetopCommand to monitor overloaded threads in a process
Top-h -p PID: Displays the CPU usage of each thread in a specified process. The monitoring result is as follows:
Monitor command output indicators for above is the threads in a process, seen from the graph can be quickly concluded that four JAVA thread CPU load is extremely high, thread ID is respectively: 17532175, 35175, 33175, 34, note printed thread ID here for decimal oh!
According to theThe process of pid&&Thread idView thread stack information
-
Jstack pid: check the specified thread stack information, in the process of this command will print out the thread stack of the specified process information, and the actual online happens, we should quickly remove the stack information output to the log in the text, retain log information quickly and then restart the service first, achieve the goal of temporarily alleviate the pressure of the server.
-
Jstack 17499 >./ threaddump. log: Outputs the thread stack information to the threaddump. log file in the current directory.
Note: Jstack prints the thread ID in hexadecimal format, while the top command prints the thread ID in decimal format. After conversion, locate the stack information of the specified thread
After analyzing the log file, the stack information of four threads is filtered as follows:
From the stack information executed by these four threads, it is clear that the program causing the CPU surge is performing a put operation on the HashMap.
Note: Test code should not be tested in the company’s online environment!
More original articles will be pushed in the public account at the first time, welcome to scan the code to follow Zhang Shaolin
