1. Introduction

With the popularity of cloud native applications and microservices, there is a growing need for embedded Servlet containers. To make it easier to build applications and services, Spring Boot offers developers three mature containers: Tomcat, Undertow, and Jetty.

In this article, we demonstrate a way to quickly compare performance differences between different container implementations by measuring metrics taken at startup and load increase.

2. Rely on

First we specify the spring-boot-starter-web dependency in pum.xml, which we must have before we can measure each of our container implementations.

Normally, we specify to use spring-boot-starter-parent as our parent dependency, and then add the starter we need:

  1. <parent>
  2. <groupId>org.springframework.boot</groupId>
  3. <artifactId>spring-boot-starter-parent</artifactId>
  4. . < version > 2.0.3 RELEASE < / version >
  5. <relativePath/>
  6. </parent>
  7. <dependencies>
  8. <dependency>
  9. <groupId>org.springframework.boot</groupId>
  10. <artifactId>spring-boot-starter</artifactId>
  11. </dependency>
  12. <dependency>
  13. <groupId>org.springframework.boot</groupId>
  14. <artifactId>spring-boot-starter-web</artifactId>
  15. </dependency>
  16. </dependencies>


Tomcat 2.1

Because in our spring-boot-starter-Web dependency, the Tomcat container is used by default, we don’t need to do any more configuration.

2.2 Jetty

To use Jetty, we first need to remove the spring-boot-starter-tomcat dependency from the spring-boot-starter-web.

Then, we simply introduce the spring-boot-starter-jetty dependency:

  1. <dependency>
  2. <groupId>org.springframework.boot</groupId>
  3. <artifactId>spring-boot-starter-web</artifactId>
  4. <exclusions>
  5. <exclusion>
  6. <groupId>org.springframework.boot</groupId>
  7. <artifactId>spring-boot-starter-tomcat</artifactId>
  8. </exclusion>
  9. </exclusions>
  10. </dependency>
  11. <dependency>
  12. <groupId>org.springframework.boot</groupId>
  13. <artifactId>spring-boot-starter-jetty</artifactId>
  14. </dependency>


2.3 Undertow

The Undertow is set in a similar way to Jetty, but after removing the dependency, we will use spring-boot-starter-undertow as our dependency:

  1. <dependency>
  2. <groupId>org.springframework.boot</groupId>
  3. <artifactId>spring-boot-starter-web</artifactId>
  4. <exclusions>
  5. <exclusion>
  6. <groupId>org.springframework.boot</groupId>
  7. <artifactId>spring-boot-starter-tomcat</artifactId>
  8. </exclusion>
  9. </exclusions>
  10. </dependency>
  11. <dependency>
  12. <groupId>org.springframework.boot</groupId>
  13. <artifactId>spring-boot-starter-undertow</artifactId>
  14. </dependency>


2.4 physical

We use Spring Boot’s Actuator component to conduct pressure tests on the system and query application indicators.

You can read this article to learn more about the movement. In this article, we just need to add this dependency to the POM:

  1. <dependency>
  2. <groupId>org.springframework.boot</groupId>
  3. <artifactId>spring-boot-starter-actuator</artifactId>
  4. </dependency>


2.5 Apache Beach

Apache Bench is an open source load testing tool, but it is usually bundled with the Apache Web server.

Windows users can download it here. If you already have the tool on your Windows PC, you should be able to find ab.exe in your Apache /bin directory.

If you are a Linux user, you can install AB with apt-get

  1. $ apt-get install apache2-utils


3. Start indicator

3.1 to collect

To collect our startup metrics, we will register the metrics we care about in Spring Boot’s ApplicationReadyEvent.

We directly use the MeterRegistry tool provided by the Actuator component to obtain the indicators we pay attention to through programming:

  1. @Component
  2. public class StartupEventHandler {
  3. // logger, constructor
  4. private String[] METRICS = {
  5. "jvm.memory.used",
  6. "jvm.classes.loaded",
  7. "jvm.threads.live"};
  8. private String METRIC_MSG_FORMAT = "Startup Metric >> {}={}";
  9. private MeterRegistry meterRegistry;
  10. @EventListener
  11. public void getAndLogStartupMetrics(
  12. ApplicationReadyEvent event) {
  13. Arrays.asList(METRICS)
  14. .forEach(this::getAndLogActuatorMetric);
  15. }
  16. private void processMetric(String metric) {
  17. Meter meter = meterRegistry.find(metric).meter();
  18. Map<Statistic, Double> stats = getSamples(meter);
  19. logger.info(METRIC_MSG_FORMAT, metric, stats.get(Statistic.VALUE).longValue());
  20. }
  21. // other methods
  22. }


In order to avoid using the REST endpoints of the Actuator to manually query the performance metrics, we start a separate JMX process to record the metrics we care about when the application starts.

3.2 choose

The Actuator provides us with a lot of data. After the application is launched, we select three representative metrics that give an overview of key points in the system’s runtime.

  • Jvm.memory. used The total amount of memory used by the JVM after startup
  • Jvm.classes.loaded The total number of class files loaded in the JVM
  • Jvm.threads. Live Specifies the number of threads alive in the JVM. In our tests, this value can be represented as the number of threads in the “rest” state.

4. Runtime metrics

4.1 to collect

In addition to providing startup metrics, when we started Apache Bench, we used the /metrics endpoint provided by the Actuator component as the target URL for requests to keep our application under load.

To test a real application under load, we may need to use the endpoints provided by our application system.

Once our application is started, we use the following command to start and execute ab:

  1. ab -n 10000 -c 10 http://localhost:8080/actuator/metrics


4.2 choose

Apache Bench can quickly give us some useful information: the connection time, the percentage of requests at a given time, and so on.

For our purposes, we usually focus more on the average number of requests per second and processing time per request.

Results of 5.

In the start-up phase, we compared the memory occupied by Tomcat, Jetty and Undertow, and found that Jetty occupied the least memory, Undertow followed, and Tomcat occupied the most.

In our measurement, we can also find the performance comparison of Tomcat, Jetty and Undertow: we can clearly see that Undertow is obviously the fastest, while Jetty is relatively slower.

MetricTomcatJettyUndertowjvm.memory.used (MB)168155164jvm.classes.loaded986997849787jvm.threads.live251719Requests per 6.4836.1486.059 second154216271650Average time per request (ms)

Note that our metrics were measured under a bare project (one that did not add any business code). If it is their own project, then the measurement indicators will most likely be different.

6. Benchmarking discussion

Developing appropriate benchmarks to adequately test the performance of container implementations can be quite complex. In order to extract the most critical information, it is important to have a clear understanding of how to write the right test case for each specific problem.

It is worth noting that the example uses the HTTP GET request from Actutor as the payload to collect measurements of the desired metrics.

Predictably, different workloads lead to different measurements and collections of metrics for container implementations. If more robust and accurate measurements are needed, it is a good idea to establish a test plan that is closer to production use cases.

In addition, more sophisticated benchmark solutions, such as JMeter or Gatling, may yield more valuable test results.

7. Select a container

Choosing an appropriate container implementation should be based on a number of considerations, not just a snap decision based on a few hard metrics profiles. Suitability, features, configurability, and policy are also usually considerable considerations.

Conclusion 8.

In this article, we looked at the implementation of Tomcat, Jetty, and Undertow’s embedded Servlet container. We tested the post-boot runtime metrics of each Servlet container using the default configuration using the Metrics endpoints exposed by the Actuator.

We used Apache Bench to stress test the application and collect performance metrics.

Finally, we talked about the advantages of this strategy and mentioned a few tips to keep in mind when comparing the benchmarks for each implementation. As always, you can get all the source code for your project from Github.