First article “Java services, MEMORY OOM problem how to quickly locate? After the release, there are friends in the comments, ask cpu100% performance problems, how to find relevant services, how to locate the problem code, also very test technical people’s skills, today simply say the idea.

Assume that several Java site services and Java microservices are deployed on the server, and a CPU exception alarm is received. How do I locate which server process is causing the CPU overload, which thread is causing the CPU overload, and which piece of code is causing the CPU overload?

The brief steps are as follows:

(1) Find the process that consumes the most CPU;

(2) Find the thread that consumes the most CPU;

(3) Look at the stack, locate what the thread is doing, locate the corresponding code;

Step 1: Find the process that consumes the most CPU

Tools: top

Methods:

  • Run the top -c command to display the process running list

  • Type P (capital P) to sort the processes by CPU usage

Here is:

As shown in the figure above, the process PID that consumes the most CPU is 10765.

Step 2: Find the threads that consume the most CPU

Tools: top

Methods:

  • Top-hp 10765, displays a list of thread running information for a process

  • Type P (capital P) and the threads are sorted by CPU usage

Here is:

As shown in the figure above, in process 10765, the thread PID that consumes the most CPU is 10804.

** First, convert the thread PID to hexadecimal.

Tools: printf

Printf “%x\n” 10804

Here is:

As shown in the figure above, 10804 corresponds to a hexadecimal 0x2a34. Of course, you can use a calculator for this step.

The reason to convert to hexadecimal is because in the stack, thread ids are represented in hexadecimal.

Next, look at the stack to find out what the thread is doing.

Tools: jstack

Methods: 10765 | jstack grep ‘0 x2a34 – C5 – color

  • Print process stack

  • The thread stack is filtered by the thread ID

Here is:

As shown in the figure above, we find the thread name “Asynclogger-1” for the high CPU consumption thread and see the stack of code executing on that thread. Finally, based on the stack information, find the corresponding code, and done!

I hope it will be helpful to students who often conduct online CPU troubleshooting. If there is a better practice, we also welcome to share. Related article: “Java services, MEMORY OOM problem how to quickly locate?” Research: Have you encountered the odd CPU100% problem?