0, background

For those of you who do a lot of back-end service development, you have encountered a very high CPU load. Especially on weekends or in the middle of the night, someone in the group suddenly reported that the load on the online machine is particularly high, and students who are not familiar with the positioning process and ideas may get on the server and get confused, and the positioning process will turn around.

In this regard, many students have organized the relevant process or methodology. For example, it takes several steps to put an elephant in the refrigerator. The traditional plan is generally four steps:

  1. Top oder by with P: 1040

  2. Top-hp process PID: 1073 // Find the associated load thread PID

  3. Printf “0x%x\n” Thread PID: 0x431 // Convert thread PID to hexadecimal in preparation for looking up jStack logs later

  4. Jstack process PID | vim + / hex threads PID / /, for example: 1040 | jstack vim + / 0 x431 –

However, every minute counts for online problem location, and the above four steps are still too tedious and time-consuming. Is it possible to encapsulate into a tool that can locate the problem with one click and find the problem line in the second level?

Of course you can! The maturity of toolchain not only reflects the operation and maintenance ability of a developer, but also reflects the efficiency consciousness of the developer. Oldratlee from Taobao encapsulated the above process into a tool: show-busy-Java-threads. sh, which can locate such problems on the line very easily. I will give two examples to see the actual effect.

Quick installation for use:

`source <(curl -fsSL https://raw.githubusercontent.com/oldratlee/useful-scripts/master/test-cases/self-installer.sh)`

Copy the code

Java regular expression backtracking causes 100% CPU

`import java.util.ArrayList; ` `import java.util.List; ` `import java.util.regex.Matcher; ` `import java.util.regex.Pattern; ` `public class RegexLoad {` `public static void main(String[] args) {` `String[] patternMatch = {"([\\w\\s]+)+([+\\-/*])+([\\w\\s]+)",` `"([\\w\\s]+)+([+\\-/*])+([\\w\\s]+)+([+\\-/*])+([\\w\\s]+)"}; ` `List<String> patternList = new ArrayList<String>(); ` `patternList.add("Avg Volume Units product A + Volume Units product A"); ` `patternList.add("Avg Volume Units / Volume Units product A"); ` `patternList.add("Avg retailer On Hand / Volume Units Plan / Store Count"); ` `patternList.add("Avg Hand Volume Units Plan Store Count"); ` `patternList.add("1 - Avg merchant Volume Units"); ` `patternList.add("Total retailer shipment Count"); ` `for (String s :patternList ){` `for(int i=0; i<patternMatch.length; i++){` `Pattern pattern = Pattern.compile(patternMatch[i]); ` `Matcher matcher = pattern.matcher(s); ` `System.out.println(s); ` `if (matcher.matches()) {` `System.out.println("Passed"); ` `}else` `System.out.println("Failed;" ); ' '} ' ' '} ' ' '} ' ' '} ' 'Copy the code

After compiling and running the above code, we can observe that the server has an additional Java process with 100% CPU:

How do you use it?

'show-busy-java-threads.sh' '# print the stack of threads that consume the most CPU from all Java processes (default 5). ' 'show-busy-java-threads.sh -c < number of threads to display >' 'show-busy-java-threads.sh -c < number of threads to display > -p < specified Java Process>' '# -f option: Execute jstack with -f option (force jstack), Sh -p < specified Java Process> -f ' 'show-busy-java-threads.sh -s < specify full path of jstack command >' '# For sudo, the JAVA_HOME environment variable cannot be passed to root. # show busy-java-threads.sh -a < output file > 'show busy-java-threads.sh -t < number of repetitions > -i < number of seconds between repeated execution > ' '# default execution once; To perform the default is 3 seconds interval ` ` # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # ` ` # note: ` ` # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # ` ` # if the user of the Java process and execution of the script for the current user is different, a jstack is the Java process. 'sudo show-busy-java-threads.sh' sudo show-busy-java-threads.shCopy the code

Example:

work@dev_zz_Master 10.48.186.32 23:45:50 ~/demo > ' 'bash show-busy-java-threads.sh' '[1] busy (96.2%) thread(8577/0x2181) stack of java process(8576) under user(work):` `"main" prio=10 tid=0x00007f0c64006800 nid=0x2181 runnable [0x00007f0c6a64a000]` `java.lang.Thread.State: RUNNABLE` `at java.util.regex.Pattern$GroupHead.match(Pattern.java:4168)` `at java.util.regex.Pattern$Loop.match(Pattern.java:4295)` `... ` `at java.util.regex.Matcher.match(Matcher.java:1127)` `at java.util.regex.Matcher.matches(Matcher.java:502)` `at Regexload.main (regexload.java :27) ' '[2] Busy(1.5%) Thread (8591/ 0x218F) Stack of Java process(8576) under user(work): `"C2 CompilerThread1" daemon prio=10 tid=0x00007f0c64095800 nid=0x218f waiting on condition [0x0000000000000000]` `java.lang.Thread.State: RUNNABLE ' '[3] Busy(0.8%) Thread (8590/ 0x218E) Stack of Java Process (8576) under User (Work): "C2 CompilerThread0" daemon prio=10 tid=0x00007f0c64093000 nid=0x218e waiting on condition [0x0000000000000000]` `java.lang.Thread.State: RUNNABLE ' '[4] Busy(0.2%) Thread (8593/0x2191) Stack of Java Process (8576) under User (Work):' 'VM Periodic Task thread" Prio =10 TID = 0x00007F0C640A2800 nID =0x2191 Waiting on condition ' '[5] Busy thread(0.1%) Stack of Java process(25137) under user(work):` `"VM Periodic Task Thread" prio=10 tid=0x00007f13340b4000 nid=0x6247 waiting on Condition ' 'work@dev_zz_Master 10.48.186.32 23:46:04 ~/demo >'Copy the code

As you can see, one click directly locate the exception line of code, is not very convenient?

2, thread deadlock, program hang

`import java.util.*; ` `public class SimpleDeadLock extends Thread {` `public static Object l1 = new Object(); ` `public static Object l2 = new Object(); ` `private int index; ` `public static void main(String[] a) {` `Thread t1 = new Thread1(); ` `Thread t2 = new Thread2(); ` `t1.start(); ` `t2.start(); ` `}` `private static class Thread1 extends Thread {` `public void run() {` `synchronized (l1) {` `System.out.println("Thread 1: Holding lock 1..." ); ` `try { Thread.sleep(10); }` `catch (InterruptedException e) {}` `System.out.println("Thread 1: Waiting for lock 2..." ); ` `synchronized (l2) {` `System.out.println("Thread 2: Holding lock 1 & 2..." ); ` `}` `}` `}` `}` `private static class Thread2 extends Thread {` `public void run() {` `synchronized (l2) {` `System.out.println("Thread 2: Holding lock 2..." ); ` `try { Thread.sleep(10); }` `catch (InterruptedException e) {}` `System.out.println("Thread 2: Waiting for lock 1..." ); ` `synchronized (l1) {` `System.out.println("Thread 2: Holding lock 2 & 1..." ); ` `} ` `} ` `} ` `} ` ` `}Copy the code

Effect after execution:

How to use the tool to locate:

One-click location: you can clearly see that threads are locking each other’s waiting resources, causing deadlocks. You can directly locate the code line and the specific cause.

From the above two examples, I think you should have a deep understanding of this tool and what the tool can solve the problem, when the CPU 100% problem can no longer panic. But more rely on your own practice, after all, the real knowledge comes from practice

3, free and practical script tools package

In addition to show-busy-Java-threads.sh mentioned in the main text, Oldratlee also integrated a lot of common scripting tools involved in development, operation and maintenance. I simply listed the ones I found particularly useful:

(1) show-duplicate-java-classes

Occasionally, I encounter a class exception that is inexplicable after the launch of the local development and testing, and the cause is a Jar conflict. This tool can find duplicate classes in either the Java Lib (Java libraries, or Jar files) or the Class directory (Class directories).

One of the troubling problems of Java development is Jar conflicts (that is, multiple versions of jars), or duplicate classes. NoSuchMethod and other problems will occur, not necessarily at that time. Look for jars that have duplicate classes to prevent this from happening.

'show-duplicate-java-classes'' show-duplicate-java-classes' 'show-duplicate-java-classes'' show-duplicate-java-classes' 'show-duplicate-java-classes'' show-duplicate-java-classes' Path /to/lib_dir1 /path/to/lib_dir2 ' ' Show -duplicate-java-classes -c path/to/class_dir1 -c /path/to/class_dir2 ' '# Find Jar ' 'show-duplicate-java-classes path/to/lib_dir1 /path/to/lib_dir2 -c path/to/class_dir1' in the specified Class directory and in all jars in the specified directory -c path/to/class_dir2`Copy the code

Such as:

'$MVN install' '$MVN install' '$MVN install' '$MVN install' ' $unzip target/*. War -d target/war... $unzip target/*. $show-duplicate-java-classes -c target/ WAR /WEB-INF/classes target/war/ web-INF /lib... `Copy the code

(2) find-in-jars

Look for class or resource files in all JAR files in the current directory.

Usage: Note that Pattern is an extended regular expression of grep.

`find-in-jars 'log4j\.properties'`
`find-in-jars 'log4j\.xml$' -d /path/to/find/directory`
`find-in-jars log4j\\.xml`
`find-in-jars 'log4j\.properties|log4j\.xml'` 

Copy the code

Example:

` $. / find - in - jars' Service. The class $` `. / WEB - INF/libs/spring - 2.5.6. SEC03. Jar! Org/springframework stereotype/Service. The class ` `. / RPC - benchmark - 0.0.1 - the SNAPSHOT. The jar! com/taobao/rpc/benchmark/service/HelloService.class`Copy the code

(3) housemd pid [java_home]

In the early days, we used BTrace to troubleshoot problems. In addition to marveling at the power of BTrace, we also failed the online system several times. In 2012, Jushi of Taobao wrote HouseMD, integrating several commonly used Btrace scripts to form an independent style application. Its core code is Scala. HouseMD is a diagnostic tool based on bytecode technology, so it can diagnose any language that ends up running on the JVM as bytecode other than Java, such as Clojure(thanks @killme2008 for getting started), Scala, Groovy, JRuby, Jython, Kotlin, etc.

Runtime tracking of Java programs using Housemd includes:

  • Viewing loaded classes

  • Tracking method

  • Viewing environment Variables

  • View object property values

  • For more information: github.com/CSUG/HouseM…

(4) jvm pid

The JVM debug tool allows you to check the status of the Java stack, heap, thread, gc, etc.

Thread ` = = = = = = = = = = = = = = = ` ` 1: see the highest CPU thread situation ` ` 2: print all threads ` ` 3: print the number of threads ` ` 4: according to the thread state statistics the number of threads ` ` = = = = = = = = GC related = = = = = = = ` ` 5: Garbage collection statistics (including reasons) can specify the interval and the number of times, the default is 1 second, 10 times. 6: shows the space of each generation in the heap. You can specify the interval and the number of times, the default is 1 second, 5 times. You can specify the interval and number of executions. The default is 1 second, 10 times. 8: Print the perm memory condition * will cause the program to pause the response * 9: check the directbuffer condition ======== heap object related ======= 10: Dump heap to file * will cause the program to pause the response * default save to 'PWD' /dump.bin, can specify other path ' '11: trigger full GC. 12: prints the JVM heap statistics * 13: prints the top20 objects in the JVM heap. Arguments: 1: sort by number of instances,2: sort by memory usage, default 1 14: prints the top20 objects in the JVM heap after triggering full gc. Arguments: 1: sort by number of instances,2: sort by memory usage, default 1 15: output all objects generated by class loaders in PERm. You can specify the interval and number of executions ======== other ======= 16: prints finalzer queues 17: displays classLoader statistics 18: displays JIT compilation statistics 19: deadlock detection 20: Displays classLoader statistics Wait X seconds. Default is 1 q: exitCopy the code

Enter the serial number to execute the corresponding command after entering the JVM tool

You can execute more than one command at a time using a semicolon (;). Separation, as in: 1; 3; 4. 5; 6

Each command can take arguments separated by a colon (:). Parameters of the same command are separated by commas (,), for example:

Enter command queue:1; Days of 00100; 10:/data1/output.bin

(5) greys [@IP:PORT]

PS: currently Greys only supports Java6+ on Linux/Unix/Mac, but not on Windows

Greys is an exception diagnostic tool for JVM processes that can easily troubleshoot problems without interrupting program execution. Like HouseMD, Greys-Anatomy was named after the eponymous TV series “Grey’s Anatomy” as a tribute to its predecessor. The code was written with reference to the ideas of two predecessors, BTrace and HouseMD.

Use Greys for runtime tracking of Java programs (no parameters, need to first greys -C PID, then Greys). The following operations are supported:

  • View loading class, method information

  • View the current basic information about the JVM

  • Method execution monitoring (call volume, failure rate, response time, etc.)

  • Methods perform data observation, recording, and playback (parameters, return results, exception information, etc.)

  • Method calls track rendering

  • For more information: github.com/oldmanpushc…

(6) sjk sjk –commands sjk –help

Use SJK for Java diagnostics, performance checks and optimization tools

  • Ttop: Monitors CPU usage for individual threads of a specified JVM process

  • JPS: improved version

  • Hh: jmap-histo enhanced version

  • Gc: Reports garbage collection information in real time

  • For more information, please refer to github.com/aragozin/jvm-tools

Refer:

  • oldratlee/useful-scripts

    github.com/oldratlee/useful-scripts

  • awesome-scripts

    github.com/superhj1987/awesome-scripts

  • Bit. ly/2xtukcb

  • Java tuning experience: Bit. ly/2xCIj2L

  • JVM screening toolbox JVM – tools segmentfault.com/a/1190000012658814

  • alibaba/arthas

    github.com/alibaba/arthas/blob/7f236219ddbd040764dd821cbcbd44899dd57c90/README.md

    From: my.oschina.net/leejun2005/blog/1524687