Small knowledge, big challenge! This article is participating in the creation activity of “Essential Tips for Programmers”.

JMH, or Java Microbenchmark Harness, is a suite of tools dedicated to Java code Microbenchmark testing. What is Micro Benchmark? In simple terms, this is benchmark testing at the method level, with an accuracy of microseconds. When you locate a hot method and want to further optimize its performance, you can use JMH to quantify the results of the optimization.

Benchmark testing: refers to the quantitative and comparable testing of a certain performance index of a class of test objects by designing scientific testing methods, testing tools and testing systems. For example, Master Lu, an Tutu, are based on a certain benchmark or under specific conditions to test the performance of an object, such as graphics card, IO, CPU and so on.

Typical application scenarios of JMH are as follows:

  1. You want to know exactly how long a method takes to execute and the correlation between the execution time and the input.
  2. Compare the throughput of different implementations of interfaces under given conditions to find the optimal implementation.
  3. See what percentage of requests are completed in how long.

1 Introduction to JMH case

1.1 the Maven rely on

JMH is shipped with JDK9, but can be used by importing dependencies or JAR packages.

<! --JMH--> <! -- https://mvnrepository.com/artifact/org.openjdk.jmh/jmh-core -->
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-core</artifactId>
    <version>1.21</version>
</dependency>
<dependency>
    <groupId>org.openjdk.jmh</groupId>
    <artifactId>jmh-generator-annprocess</artifactId>
    <version>1.21</version>
</dependency>

Copy the code

1.2 Programming

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
public class HelloJMH {
    /** * String concatenation StringBuilder benchmark */
    @Benchmark
    public void testStringBuilder(a) {
        StringBuilder str = new StringBuilder();
        for (int i = 0; i < 1000; i++) {
            str.append(i);
        }
        String s = str.toString();
    }
    /** * string concatenation directly add benchmark */
    @Benchmark
    public void testStringAdd(a) {
        String str = "";
        for (int i = 0; i < 1000; i++) { str = str + i; }}public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder()
                .include(HelloJMH.class.getSimpleName())  // Contain the method
                .forks(1)   // Separate several processes to test separately
                .build();
        newRunner(options).run(); }}Copy the code

The @benchmark annotation indicates that the method is a Benchmark test.

@benchmarkMode indicates the JMH measurement method and Angle. This is the average measurement time.

@ OutputTimeUnit said the benchmark results by use of units of time, can be used in the class or method, using Java. Util. Concurrent. TimeUnit the standard unit of time.

In the Main method, run the Options instance through the Runner class. An OptionsBuilder object is officially provided to stream builds. Other configuration information for OptionsBuilder is covered below.

1.3 Test Results

1)	# JMH version: 1.21
2)	# VM version: JDK 1.8. 0 _144.Java HotSpot(TM)64-bit Server VM, 25.144-b01 3) # VM invoker: C: Program Files Java jdk1.8.0_144 jre bin java.exe 4) # VM options: - javaagent: D: \ soft \ IntelliJ IDEA 2019.3 \ lib \ idea_rt jar=61956:D:\soft\IntelliJ IDEA 2019.3\bin -Dfile.encoding=UTF-8
5)	# Warmup: 5 iterations, 10 s each
6)	# Measurement: 5 iterations, 10 s each
7)	# Timeout: 10 min per iteration
8)	# Threads: 1 thread, will synchronize iterations
9)	# Benchmark mode: Average time, time/op
10)	# Benchmark: com.thread.test.JMH.HelloJMH.testStringAdd

11)	# Run progress: 0.00% complete, ETA 00:03:20
12)	# Fork: 1 of 1
13)	# Warmup Iteration   1: 506360.123 ns/op
14)	# Warmup Iteration   2: 460295.578 ns/op
15)	# Warmup Iteration   3: 492550.630 ns/op
16)	# Warmup Iteration   4: 482141.558 ns/op
17)	# Warmup Iteration   5: 469897.660 ns/op
18)	Iteration   1: 443427.726 ns/op
19)	Iteration   2: 456970.538 ns/op
20)	Iteration   3: 440686.491 ns/op
21)	Iteration   4: 451894.998 ns/op
22)	Iteration   5: 432889.165 ns/op


23)	Result "com.thread.test.JMH.HelloJMH.testStringAdd":
a)	445173.784Plus or minus (99.9%) 36450.901 ns/op [Average]
b)	(min, avg, max) = (432889.165.445173.784.456970.538), stdev = 9466.183
c)	CI (99.9%) :408722.883.481624.685] (assumes normal distribution)


24)	# JMH version: 1.21
25)	# VM version: JDK 1.8. 0 _144.Java HotSpot(TM)64-bit Server VM, 25.144-b01 26) # VM invoker: C: Program Files Java Java jdk1.8.0_144 jre bin java.exe 27) # VM options: - javaagent: D: \ soft \ IntelliJ IDEA 2019.3 \ lib \ idea_rt jar=61956:D:\soft\IntelliJ IDEA 2019.3\bin -Dfile.encoding=UTF-8
28)	# Warmup: 5 iterations, 10 s each    // Preheat times
29)	# Measurement: 5 iterations, 10 s each   // Measure times
30)	# Timeout: 10 min per iteration
31)	# Threads: 1 thread, will synchronize iterations
32)	# Benchmark mode: Average time, time/op
33)	# Benchmark: com.thread.test.JMH.HelloJMH.testStringBuilder

34)	# Run progress: 50.00% complete, ETA 00:01:40
35)	# Fork: 1 of 1
36)	# Warmup Iteration   1: 10372.126 ns/op
37)	# Warmup Iteration   2: 10301.755 ns/op
38)	# Warmup Iteration   3: 10006.275 ns/op
39)	# Warmup Iteration   4: 9778.343 ns/op
40)	# Warmup Iteration   5: 9868.092 ns/op
41)	Iteration   1: 9641.269 ns/op
42)	Iteration   2: 10259.971 ns/op
43)	Iteration   3: 9844.944 ns/op
44)	Iteration   4: 9704.533 ns/op
45)	Iteration   5: 9711.980 ns/op


46)	Result "com.thread.test.JMH.HelloJMH.testStringBuilder":
a)	9832.539Plus or minus (99.9%) 963.347 ns/op [Average]
b)	(min, avg, max) = (9641.269.9832.539.10259.971), stdev = 250.178
c)	CI (99.9%) :8869.193.10795.886] (assumes normal distribution)


47)	# Run complete. Total time: 00:03:21

48)	REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on
49)	why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial
50)	experiments, perform baseline and negative tests that provide experimental control, make sure
51)	the benchmarking environment is safe on JVM/OS/HW level, ask forreviews from the domain experts. 52) Do not assume the numbers tell you what you want them to tell. 53) Benchmark Mode Cnt Score Error Units 54) JMH. HelloJMH. TestStringAdd avgt 5 to 445173.784 + 36450.901 ns/op 55) JMH. HelloJMH. TestStringBuilder avgt 5 to 9832.539 + 963.347 ns/opCopy the code

Explanation:

Lines 1-10 show basic information about the test, such as the Java path used, the number of iterations to warm up the code, the number of iterations to measure the code, the number of threads used, the statistical unit of the test, and so on.

Starting at line 13, the results of each warm-up iteration are shown. The warm-up iteration will not be counted as the final statistical result. The purpose of warming up is to allow the Java VIRTUAL machine to optimize the code under test enough that, for example, the code under test should be fully JIT compiled and optimized after warming up.

Starting at line 18, the results of each benchmark iteration are displayed, each iteration showing the current execution rate, which is the time taken for an operation.

After 5 iterations, statistics are made and the Result comes after Result. The first paragraph of Result tells us about the maximum value, minimum value, and average value. The second paragraph is the main message. In this case, lines 54 and 55 show the average execution time and error time for the testStringBuilder and testStringAdd functions. As you can see from the results, Using StringBuilder is more efficient when there are a lot of string concatenations.

Benchmark Mode Cnt Score Error Units
The way benchmarks are executed Test mode How many runs score error unit

2 Basic concepts and configurations of the JMH

2.1 Test Mode (Mode)

Mode indicates the measurement Mode and Angle of JMH. There are four methods. Throughput and average time of method execution are the most commonly used statistical methods. This can be configured via the @benchmarkMode annotation.

Throughput: Total Throughput, which indicates the number of calls that can be executed in 1 second. As follows:

1)	Benchmark                        Mode  Cnt   Score    Error   Units
2)	JMH.HelloJMH.testStringAdd      thrpt    5  ≈ 10⁻ ⁵ ops/ns3)	JMH.HelloJMH.testStringBuilder  thrpt    5  ≈ 10⁻ ⁴ ops/nsCopy the code

AverageTime: the AverageTime of a call, the time required for each call, the pattern in the case.

SampleTime: random sampling and output the distribution of sample results, such as “99% calls within XXX ms and 99.99% calls within XXX ms”. As follows:

1)	Benchmark                                                   Mode      Cnt        Score      Error  Units
2)	JMH.HelloJMH.testStringAdd                                sample   110636   451524.056 ± 1674.469  ns/op
3) JMH. HelloJMH. TestStringAdd: testStringAdd p000.            sample            307712.000             ns/op
4) JMH. HelloJMH. TestStringAdd: testStringAdd p0. 50            sample            392192.000             ns/op
5) JMH. HelloJMH. TestStringAdd: testStringAdd p090.            sample            558080.000             ns/op
6) JMH. HelloJMH. TestStringAdd: testStringAdd p095.            sample            649216.000             ns/op
7) JMH. HelloJMH. TestStringAdd: testStringAdd p099.            sample           1337344.000             ns/op
8) JMH. HelloJMH. TestStringAdd: testStringAdd p0999.           sample           2023424.000             ns/op
9) JMH. HelloJMH. TestStringAdd: testStringAdd p09999.          sample           2742493.594             ns/op
10) JMH. HelloJMH. TestStringAdd: testStringAdd p100.            sample           3420160.000             ns/op
11)	JMH.HelloJMH.testStringBuilder                            sample  1228587    10293.875 ±   39.332  ns/op
12) JMH. HelloJMH. TestStringBuilder: testStringBuilder p000.    sample              8688.000             ns/op
13) JMH. HelloJMH. TestStringBuilder: testStringBuilder p0. 50    sample              9600.000             ns/op
14) JMH. HelloJMH. TestStringBuilder: testStringBuilder p090.    sample             10592.000             ns/op
15) JMH. HelloJMH. TestStringBuilder: testStringBuilder p095.    sample             11600.000             ns/op
16) JMH. HelloJMH. TestStringBuilder: testStringBuilder p099.    sample             21280.000             ns/op
17) JMH. HelloJMH. TestStringBuilder: testStringBuilder p0999.   sample             71552.000             ns/op
18) JMH. HelloJMH. TestStringBuilder: testStringBuilder p09999.  sample            695296.000             ns/op
19) JMH. HelloJMH. TestStringBuilder: testStringBuilder p100.    sample           2019328.000             ns/op
Copy the code

SingleShotTime: The default mode is Iteration. Iteration is 1 second. SingleShotTime only runs once. Warmup is often set to 0 at the same time to test performance on a cold start.

All: As the name implies, All modes, which are often used in internal testing.

2.2 Iteration (Iteration)

An iteration is a unit of measurement of JMH. In most measurement modes, an iteration represents one second. During this second, the method under test is called continuously, and the calculation throughput, average time, and so on are sampled.

2.3 preheating (Warmup)

Warmup is the act of warming up before actually benchmarking.

Why warm up? Because of the JVM’s JIT mechanism, if a function is called more than once, the JVM tries to compile it into machine code to speed up execution. To make benchmark’s results more realistic, you need to warm up.

Due to the JIT of the Java virtual machine, the same method will take a different time before and after JIT compilation. Usually only the performance of the method after JIT compilation is considered. JIT optimization can be turned off with the -xint parameter.

2.4 State (State)

The @state annotation applies to the class. State specifies the scope of an object. There are three types of scope:

  1. Scope.Thread: The default State. Each test Thread is assigned an instance, i.e. an object is accessed by only one Thread. In multithreaded pool testing, one object is generated for each thread;
  2. Scope.Benchmark: All test threads share an instance, used to test the performance of stateful instances in multi-threaded sharing
  3. Scope.Group: each thread Group shares an instance;

2.5 Configuration Classes (Options/OptionsBuilder)

Before you start testing, you need to configure your tests. You usually need to specify parameters such as specifying test classes (include), number of processes used (fork), and number of warm-up iterations (warmuplterations). You need to use the configuration class when configuring to start a test.

The common methods of OptionsBuilder and the corresponding annotation form are as follows:

The method name parameter role Corresponding annotations
include Accepts a string expression representing the classes and methods that you want to test. Specify the benchmark classes and methods to run –
exclude Accepts a string expression representing classes and methods that do not need to be tested Specify benchmark class methods not to run –
warmupIterations Number of iterations warmed up Specifies the number of iterations to warm up @Warmup
warmupBatchSize Preheat batch size Specifies the size of the preheat batch @Warmup
warmupForks The preheating modes are INDI, BULK, and BULK_INDI Specify preheat mode @Warmup
warmupMode Warm-up mode Specify the preheating mode @Warmup
warmupTime Preheating time Specify the time for preheating @Warmup
measurementIterations The number of iterations of the test Specifies the number of iterations of the test @Measurement
measurementBatchSize Test batch size Specifies the size of the test batch @Measurement
measurementTime Test time Specify when to test @Measurement
mode Test modes: Throughput, AverageTime, SampleTime, SingleShotTime, All Specify test mode @benchmarkMode – can be used on classes or methods
Fork The child to count
threads Number of open threads per method Multithreaded testing @Threads can be used on methods or classes

2.6 Other Notes

@ OutputTimeUnit: unit of time used by the benchmark results, annotations can be used in the class or method, using Java. Util. Concurrent. TimeUnit the standard unit of time.

@setup: Method annotation, which is executed before benchmark and, as its name suggests, is used for initialization.

@teardown: Method annotation, as opposed to @setup, which is executed after all benchmark executions have finished and is used to recycle resources, etc.

@param: Member annotations that can be used to specify multiple cases of an argument. It is especially useful for testing the performance of a function with different parameters. The @param annotation takes an array of strings that are converted to the corresponding data type before the @setup method executes. For example, if there are two @param annotated fields, the first field has 5 values and the second field has 2 values, then each test method will run 5*2=10 times.

References:

  1. Practical Java High Concurrency Programming

If you don’t understand or need to communicate, you can leave a message. In addition, I hope to like, collect, pay attention to, I will continue to update a variety of Java learning blog!