JVM Optimization Essentials – Monitoring tools

This is the second in a series of JVM optimizations:

• JVM optimization – garbage collection

In our previous article on JVM garbage collection, we learned how the JVM handles memory allocation and garbage collection. Theory is the tool to guide practice. With the guidance of theory, knowledge and experience are the key basis for problem positioning, and data can provide us with the basis.

In common online problems, most of us will encounter the following:

• Memory leaks
• A process suddenly has a CPU surge
• The thread deadlock
• Slow response… And so on.

If you encounter the above problems, there are various local tools to support viewing offline, but there are not so many local debugging tools to support online, how should we locate the problem based on monitoring tools?

We typically base this on data collection, which requires monitoring tools such as run logs, exception stacks, GC logs, thread snapshots, heap snapshots, etc. Frequent use of the right analytics and monitoring tools can speed up our ability to analyze data and locate and solve problems. We will introduce it in detail below.

JVM common monitoring tools & instructions

JPS: JVM process health tool

jps [options] [hostid]Copy the code

If you do not specify hostid, the default value is the current host or server.

The command line parameters are described as follows:

-q Does not output the class name, Jar name, or parameters passed to main. -l Outputs the full name of the main class or Jar. -m Outputs parameters passed to mainCopy the code

Such as:

Jstat: JVM statistics monitoring tool

Jstat is a command line tool for viewing information about the running status of a virtual machine. It can display classloading, memory, garbage collection, JIT compilation, and other running data in local or remote virtual machine processes, and is the preferred tool for locating JVM performance online.

Command format:

Jstat [generalOption | outputOptions vmid [interval [s | ms] [count]]] generalOption - a single common command line options, Such as -help, -options, or -version. OutputOptions - one or more outputOptions, composed of a singlestatOption consists of the Option. It can be used with -t, -h, and -j options.Copy the code

Parameter Options:

Such as:

To check gc status, run jstat-gcutil 27777

Jinfo: Java configuration information

Command format:

jinfo[option] pidCopy the code

For example, get some JVM running and startup information for the current process.

4. Jmap: Java memory mapping tool

The jmap command is used to produce heap dump snapshots. Prints out all ‘objects’ in memory of a Java process (using pid) (e.g., which objects were generated, and how many).

Command format:

jmap [ option ] pid

jmap [ option ] executable core

jmap [ option ] [server-id@]remote-hostname-or-IPCopy the code

Parameter Options:

-dump:[live,]format=b,file=<filename> With hprof binary format, output JVM heap content to file=. Live suboption is optional, If the live option is specified, only live objects are output to the file. - FinalizerInfo Prints information about objects that are waiting for collection. Heap configuration and wise Heap usage -histo[:live] Displays the number of instances, memory usage, and full name of each class. If the live subparameter is added, only the number of live objects is counted. - Permstat Prints the classload and JVM heap persistent layers. Contains the name of each classloader, liveliness, address, parent ClassLoader, and number of classes loaded. In addition, the number of internal strings and memory usage are also printed. -f Force. -dump or -histo is used when the PID does not correspond. In this mode, the live child parameter is invalid. - h | - help printing auxiliary information - J pass parameters to jmap start the JVM.Copy the code

Such as:

Use jmap-heap PID to view process heap memory usage, including GC algorithm used, heap configuration parameters, and heap memory usage in each generation:

Use jmap-histo [:live] PID to view the statistical histogram of the number and size of objects in heap memory.

Jhat: JVM heap Snapshot analysis tool

The jhat command is used together with JAMP to analyze heap fast storage snapshots produced by MAP. Jhat has a mini HTTP /Html server built in, which you can look up in your browser. However, it is recommended not to use it as far as possible. Since there are Dumpt files, you can pull them down from the production environment and analyze them through local visualization tools, which not only reduces the pressure on the online server, but also provides enough detailed analysis (such as MAT/ JProfile /visualVm).

Jstack: Java stack trace tool

Jstack is used to generate a thread snapshot of the Java VIRTUAL machine at the current time. A thread snapshot is a collection of method stacks that are being executed by each thread in the Java VIRTUAL machine (JVM). The main purpose of a thread snapshot is to locate the cause of a long pause in a thread, such as deadlocks, loops, and long waits caused by requests for external resources.

Command format:

jstack [ option ] pid

jstack [ option ] executable core

jstack [ option ] [server-id@]remote-hostname-or-IPCopy the code

Parameters:

The -f when jstack ['-l] pid 'No corresponding time to force printing stack information-lThe long list. Print additional information about the lock, such as belong to Java. Util. Concurrent ownable synchronizers list. -m print Java and native c/c + + framework of all the stack information. - h | -help print help information pid The ID of the Java process whose configuration information needs to be printed can be queried using JPS.Copy the code

This will be used in the following example of finding the highest CPU consumption.

Second, visualization tools

Common visualization tools for JVM monitoring include Jconsole and visualVm provided by JDK itself, jProfilter, Perfino,Yourkit, Perf4j, JProbe, MAT, etc. These tools have greatly enriched the way we locate and optimize JVMS.

There are many tutorials on how to use these tools, but I won’t cover them here. For VisualVm, it is recommended to use, in addition to the RELATIVELY low invasion of the JVM, or the JDK team developed their own, I believe that the future features will be more rich and perfect. Jprofilter provides the most complete functionality and visualization for third-party monitoring tools. Most ides now support plug-ins for pre-launch debugging and performance tuning.

In addition, for online dump heap information, try to pull it offline for visual tool analysis, so that more detailed analysis. If online monitoring is necessary for some urgent problems, VisualVm’s remote functionality can be used to do this, which requires the MAT functionality under Tool.jar.

Three, the application

1. CPU surge

Sometimes when you’re online, at some point in time, you might have a problem with the CPU surge of your application. We should be familiar with some of the instructions, quickly troubleshoot the corresponding code.

1. Find the process that consumes the most CPU

Instructions: topCopy the code

2. Locate the thread that consumes the most CPU in the process

Command: top-HP PIDCopy the code

3. Convert to base

printf"%x\n" 15332 // Convert hexadecimal (convert to 0x3be4)Copy the code

4. Filter the specified thread and print stack information

Instructions: jstack pid | grep'threadPid'  -C5 --color 

jstack 13525 |grep '0x3be4'-c5 --color prints the process stack and filters the thread stack information by thread ID.Copy the code

It can be seen that it is a reporting program, which consumes too much CPU (the above example is only an example, and the CPU consumption is not high).

2. Thread deadlocks

Thread deadlocks sometimes occur in deployment scenarios, but they are not common. Let’s take a look at it using JStack. For example, we now have a thread deadlocked application, resulting in some operation waiting.

1. Search for the Java process ID

Command: TOP or JPSCopy the code

2. View the snapshot information about the Java process thread

Instructions: jstack-l pidCopy the code

As you can see from the output, a thread deadlock has occurred, and it indicates which line of code has occurred. This allows you to quickly troubleshoot problems.

3. OOM memory leaks

The OOM exception in the Java heap is a common memory overflow exception in practical applications. Generally, we first analyze heap dump snapshots through memory mapping analysis tools (such as MAT) to confirm whether objects in memory have problems.

Of course, there are many reasons for OOM, not a lack of application resources in the heap. In addition, too many resources may be applied and not released, or system resources may be exhausted due to frequent application. I need to go through all three.

OOM:

1. The requested resources (memory) are insufficient.

2. Too many applied resources are not released.

3. Too many application resources are used up. For example, too many threads or too large memory.

1. Troubleshoot application resource problems.

Instruction: jmap-heap 11869Copy the code

Check the size and usage of new generation and old generation heap memory allocation to see whether the allocation is too small.

From the above investigation, it is found that there is no problem with the memory applied for by the program.

2. The screen of gc

In particular, in the case of FGC, the memory of each generation.

Instruction :jstat -gcutil 11938 1000 Outputs generational memory allocation for gc per second and gc timeCopy the code

3. Find the object that consumes the most memory

Instruction: jmap - histo: 11869 | live moreCopy the code

In the preceding information, the maximum memory object is 161KB, which is within the normal range. If an object takes up a lot of space, such as more than 100Mb, you should analyze why it is not freed.

Note the above instructions:

Jmap - histo: live 11869 | more execution, will cause the JVM enforcing an FGC, online is not recommended, you can take the dump memory snapshot, offline analysis using visualization tools, more detailed. Jmap-dump :format=b,file=/ TMP /dump.dat 11869 Or use online o&M tools for automatic processing, which facilitates quick location and loss of error time.Copy the code

4. Check whether resources are used up

• Pstree Displays the number of process threads
• Netstat Displays the number of network connections

Or adopt:

• Ll/proc / ${PID} / fd | wc – l / / handle to open the number
• Ll/proc / ${PID} / task | wc -l (pstree effect equivalent – p | wc -l) / / open the number of threads

These are some common JVM command applications.

The application of one tool is not a panacea. To solve problems, it often requires the combination of multiple tools to better locate problems. No matter what analysis tools are used, the most important thing is to be familiar with the advantages and disadvantages of each tool. In this way, we can learn from each other and cooperate.

—————————————————————————–

For more interesting and original technical articles, scan our public account.

Focus on personal growth and game development to promote the growth and progress of the domestic gaming community.