This post was originally posted by Jenkins Chinese Community
Link to the original article by Abhyudayasharma
Translator: Donghui Wang
Microbenchmark framework for the Jenkins plugin
This article introduces the Microbenchmark framework of the Jenkins plug-in and how to run benchmarks in the plug-in
Microbenchmark framework for the Jenkins plugin
As part of my Google programming summer camp, I’ve been working on improving the performance of the Role Strategy Plugin. Since there was no existing way to measure performance and benchmark on the Jenkins plug-in, my job in the first phase of the project was to create a framework to run the benchmark in the Jenkins plug-in in a Jenkins instance. To make our job easier, we chose the Java microbenchmark tool to run these benchmarks. This allows us to reliably measure the performance of time-critical features and will help keep Jenkins running faster for everyone.
The microbenchmark framework was recently released in Jenkins Unit Test Tool 2.50. The following blog post shows how to run benchmarks in a plug-in.
introduce
The framework runs by starting a temporary Jenkins instance for each fork of the JMH benchmark, like JenkinsRule in the Jenkins test tool. Benchmarks are run directly from JUnit tests, allow failed builds on the fly, and are easy to run from the IDE, just like unit tests. You can easily configure the benchmark and pass the path to the YAML file using the Java method or using the Jenkins Plugin :configuration-as-code:[configuration-as-code plug-in].
To run benchmarks from your plug-in, you need to:
- Upgrade the minimum Jenkins version required to version 2.60.3 or higher
- Upgrade plugin-POM to version ≥ 3.46 or manually update Jenkins test tool to version ≥ 2.51
Now, to run the benchmark Test, you need to have a contain [@ Test] (https://my.oschina.net/azibug) benchmark Test to run the program, so that it can run like the JUnit Test. From within the test method, the benchmark can be configured using the OptionsBuilder provided by THE JMH. Such as:
public class BenchmarkRunner {
@Test
public void runJmhBenchmarks() throws Exception {
ChainedOptionsBuilder options = new OptionsBuilder()
.mode(Mode.AverageTime)
.forks(2)
.result("jmh-report.json"); // Automatically detect benchmark classes annotated with @JmhBenchmark new BenchmarkFinder(getClass()).findBenchmarks(options); new Runner(options.build()).run(); }}Copy the code
The sample datum
Now you can write the first benchmark:
No special setup is required
@JmhBenchmark
public class JmhStateBenchmark {
public static class MyState extends JmhBenchmarkState {
}
@Benchmark
public void benchmark(MyState state) {
// benchmark code goes here
state.getJenkins().setSystemMessage("Hello world"); }}Copy the code
Use configuration as code
To use configuration-as-code, in addition to the above dependencies, add the following to your pom.xml:
< the dependency > < groupId > IO. Jenkins < / groupId > < artifactId > configuration - as - code < / artifactId > < version > 1.21 < / version > <optional>true</optional> </dependency> <dependency> <groupId>io.jenkins</groupId> <artifactId>configuration-as-code</artifactId> < version > 1.21 < / version > < classifier > tests < / classifier > < scope >test</scope>
</dependency>
Copy the code
Configuring a benchmark is now as simple as providing the path to the YAML file and specifying the class that contains the state of the benchmark.
@JmhBenchmark
public class SampleBenchmark {
public static class MyState extends CascJmhBenchmarkState {
@Nonnull
@Override
protected String getResourcePath() {
return "config.yml"; } @Nonnull @Override protected Class<? >getEnclosingClass() {
returnSampleBenchmark.class; } } @Benchmark public void benchmark(MyState state) { Jenkins jenkins = state.getJenkins(); // jenkins is configured and ready to be benchmarked. // your benchmark code goes here... }}Copy the code
More examples
As part of this project, benchmarks were created in the Role Strategy Plugin that show examples of configuration for various situations. You can find them here.
Run the benchmark
Run the benchmark from Maven
To facilitate running benchmarks from Maven, a Maven profile is created to run benchmarks and is available starting with plugin-POm version 3.45. You can then run the benchmark from the command line using MVN test-dBenchmark.
Run the benchmark in ci.jenkins.io
If your plug-in is hosted on ci.jenkins.io, you can easily run benchmarks directly from Jenkinsfile. By using the runBenchmarks() method after the buildPlugin() step in the Jenkinsfile, which is now in the Jenkins pipeline library. This function also takes the path of the generated JMH benchmark report as an optional parameter and archives the benchmark results. Running benchmarks in a pull Request build allows you to continuously test to monitor the performance impact of a given change. For example, Jenkinsfile from the Role Strategy Plugin:
buildPlugin()
runBenchmarks('jmh-report.json')
Copy the code
Visualize benchmark results
You can visualize the generated benchmark report (in JSON format) using plugin: JMH – Report [JMH Report plug-in] or by passing the benchmark report to the JMH Visualization Tool Web service. For example, here are some visual reports from benchmarks in the Role Strategy Plugin:
The improvements seen above were obtained through a small pull request to the plug-in and demonstrate that even seemingly small changes can lead to big performance improvements. Microbenchmarks help to find these hot spots and estimate the impact of changes.
Some tips and tricks
- As in the example above
BenchmarkRunner
The class name does not comply with the Maven Surefire plug-in’s test condition naming convention, and the benchmark does not interfere with JUnit testing. - Benchmarking methods are needed
[@Benchmark](https://my.oschina.net/u/3268003)
Annotate them so that JMH can detect them. - When the annotated as
@JmhBenchmark
When, the class containing the benchmark is specified byBenchmarkFinder
Automatic find. - References to Jenkins instances can be passed
JmhBenchmarkState#getJenkins()
Or throughJenkins.getInstance()
Get, just as you would in any other case. JmhBenchmarkState
providessetup()
和tearDown()
Methods that you can override to configure the Jenkins instance, depending on the needs of your benchmark.- Due to the
highmem
Nodes have limited availability and benchmarks based on CI.jenkins.io are currently limited in flow. - The benchmark framework is provided in Jenkins Test Tool 2.50 and version 2.51 is recommended as it contains some bug fixes.
Links and feedback
If you have any feedback, comments, or questions, please feel free to contact me in the Role Strategy Plugin Gitter chat room or through the Jenkins Developer mailing list.
- Presentation slide
- Examples in platform SIG meetings
- Documentation for the microbenchmark framework:
- Write benchmarks (Jenkins Testing Tools)
- Using JCasC preconfigured benchmarks
- Run benchmarks using the Plugin POM profile
- Run the benchmark build steps on ci.jenkins.io