This post was originally posted by Jenkins Chinese Community

Link to the original article by Abhyudayasharma

Translator: Donghui Wang

Microbenchmark framework for the Jenkins plugin

This article introduces the Microbenchmark framework of the Jenkins plug-in and how to run benchmarks in the plug-in

Microbenchmark framework for the Jenkins plugin

As part of my Google programming summer camp, I’ve been working on improving the performance of the Role Strategy Plugin. Since there was no existing way to measure performance and benchmark on the Jenkins plug-in, my job in the first phase of the project was to create a framework to run the benchmark in the Jenkins plug-in in a Jenkins instance. To make our job easier, we chose the Java microbenchmark tool to run these benchmarks. This allows us to reliably measure the performance of time-critical features and will help keep Jenkins running faster for everyone.

The microbenchmark framework was recently released in Jenkins Unit Test Tool 2.50. The following blog post shows how to run benchmarks in a plug-in.

introduce

The framework runs by starting a temporary Jenkins instance for each fork of the JMH benchmark, like JenkinsRule in the Jenkins test tool. Benchmarks are run directly from JUnit tests, allow failed builds on the fly, and are easy to run from the IDE, just like unit tests. You can easily configure the benchmark and pass the path to the YAML file using the Java method or using the Jenkins Plugin :configuration-as-code:[configuration-as-code plug-in].

To run benchmarks from your plug-in, you need to:

  • Upgrade the minimum Jenkins version required to version 2.60.3 or higher
  • Upgrade plugin-POM to version ≥ 3.46 or manually update Jenkins test tool to version ≥ 2.51

Now, to run the benchmark Test, you need to have a contain [@ Test] (https://my.oschina.net/azibug) benchmark Test to run the program, so that it can run like the JUnit Test. From within the test method, the benchmark can be configured using the OptionsBuilder provided by THE JMH. Such as:

public class BenchmarkRunner {
    @Test
    public void runJmhBenchmarks() throws Exception {
        ChainedOptionsBuilder options = new OptionsBuilder()
                .mode(Mode.AverageTime)
                .forks(2)
                .result("jmh-report.json"); // Automatically detect benchmark classes annotated with @JmhBenchmark new BenchmarkFinder(getClass()).findBenchmarks(options); new Runner(options.build()).run(); }}Copy the code

The sample datum

Now you can write the first benchmark:

No special setup is required

@JmhBenchmark
public class JmhStateBenchmark {
    public static class MyState extends JmhBenchmarkState {
    }

    @Benchmark
    public void benchmark(MyState state) {
        // benchmark code goes here
        state.getJenkins().setSystemMessage("Hello world"); }}Copy the code

Use configuration as code

To use configuration-as-code, in addition to the above dependencies, add the following to your pom.xml:

< the dependency > < groupId > IO. Jenkins < / groupId > < artifactId > configuration - as - code < / artifactId > < version > 1.21 < / version > <optional>true</optional> </dependency> <dependency> <groupId>io.jenkins</groupId> <artifactId>configuration-as-code</artifactId> < version > 1.21 < / version > < classifier > tests < / classifier > < scope >test</scope>
</dependency>

Copy the code

Configuring a benchmark is now as simple as providing the path to the YAML file and specifying the class that contains the state of the benchmark.

@JmhBenchmark
public class SampleBenchmark {
    public static class MyState extends CascJmhBenchmarkState {
        @Nonnull
        @Override
        protected String getResourcePath() {
            return "config.yml"; } @Nonnull @Override protected Class<? >getEnclosingClass() {
            returnSampleBenchmark.class; } } @Benchmark public void benchmark(MyState state) { Jenkins jenkins = state.getJenkins(); // jenkins is configured and ready to be benchmarked. // your benchmark code goes here... }}Copy the code

More examples

As part of this project, benchmarks were created in the Role Strategy Plugin that show examples of configuration for various situations. You can find them here.

Run the benchmark

Run the benchmark from Maven

To facilitate running benchmarks from Maven, a Maven profile is created to run benchmarks and is available starting with plugin-POm version 3.45. You can then run the benchmark from the command line using MVN test-dBenchmark.

Run the benchmark in ci.jenkins.io

If your plug-in is hosted on ci.jenkins.io, you can easily run benchmarks directly from Jenkinsfile. By using the runBenchmarks() method after the buildPlugin() step in the Jenkinsfile, which is now in the Jenkins pipeline library. This function also takes the path of the generated JMH benchmark report as an optional parameter and archives the benchmark results. Running benchmarks in a pull Request build allows you to continuously test to monitor the performance impact of a given change. For example, Jenkinsfile from the Role Strategy Plugin:

buildPlugin()
runBenchmarks('jmh-report.json')

Copy the code

Visualize benchmark results

You can visualize the generated benchmark report (in JSON format) using plugin: JMH – Report [JMH Report plug-in] or by passing the benchmark report to the JMH Visualization Tool Web service. For example, here are some visual reports from benchmarks in the Role Strategy Plugin:

The improvements seen above were obtained through a small pull request to the plug-in and demonstrate that even seemingly small changes can lead to big performance improvements. Microbenchmarks help to find these hot spots and estimate the impact of changes.

Some tips and tricks

  • As in the example aboveBenchmarkRunnerThe class name does not comply with the Maven Surefire plug-in’s test condition naming convention, and the benchmark does not interfere with JUnit testing.
  • Benchmarking methods are needed[@Benchmark](https://my.oschina.net/u/3268003)Annotate them so that JMH can detect them.
  • When the annotated as@JmhBenchmarkWhen, the class containing the benchmark is specified byBenchmarkFinderAutomatic find.
  • References to Jenkins instances can be passedJmhBenchmarkState#getJenkins()Or throughJenkins.getInstance()Get, just as you would in any other case.
  • JmhBenchmarkStateprovidessetup() 和 tearDown()Methods that you can override to configure the Jenkins instance, depending on the needs of your benchmark.
  • Due to thehighmemNodes have limited availability and benchmarks based on CI.jenkins.io are currently limited in flow.
  • The benchmark framework is provided in Jenkins Test Tool 2.50 and version 2.51 is recommended as it contains some bug fixes.

Links and feedback

If you have any feedback, comments, or questions, please feel free to contact me in the Role Strategy Plugin Gitter chat room or through the Jenkins Developer mailing list.

  • Presentation slide
  • Examples in platform SIG meetings
  • Documentation for the microbenchmark framework:
    • Write benchmarks (Jenkins Testing Tools)
    • Using JCasC preconfigured benchmarks
    • Run benchmarks using the Plugin POM profile
    • Run the benchmark build steps on ci.jenkins.io