The code snippet in this article is from the corresponding complete sample source project:

  • https://gitee.com/xautlx/package-optimize-demo
  • https://github.com/xautlx/package-optimize-demo

Relevant codes and configurations have been tested in practice. If any problem is found in the verification process, please Issue feedback for timely correction. Thank you for your support!

overview

With the popularity of Spring Boot, you can see how much easier it is to simply build and output a JAR file and then deploy and run your application with a single Java-JAR command. It is common for some single applications that the size of a single JAR file becomes larger and larger as the project scale expands, often reaching 200 or 300 MB. If the micro-service architecture is introduced again, there will be ten or twenty micro-services, and all the module JARs will add up to one or two GB of the entire system deployment file.

Once a system is put into operation, no matter it is new requirement iteration or Bug fixing, it is inevitable to make deployment update. Especially for some delivery-type projects, the deployment files of several hundred MB or several GB need to be transferred for the first deployment or remote update, which is really a headache.

Imagine that the online system finds an urgent and serious Bug and sends it to the supervisor, and tells him to fix it immediately. The research and development colleagues analyze and troubleshoot it in a hurry, submit the code, complete the construction, package and deliver it to the operation and maintenance. After a while the leadership anxious to get angry to ask the question updated to solve it? Operation and maintenance can only very embarrassed answer: not yet, the deployment package file is relatively large, uploading a little slow…

I changed a few lines of code. Why upload hundreds of MB of files for deployment updates? Isn’t there a way to optimize it?

In this case, I suggest you read down, maybe you can find the answer you want.

The content of this article includes:

  • How w to optimize the size of a single Spring Boot JAR file to 100 or 200 KB by separating it into a dependent component lib directory and a business JAR for deployment.
  • How to combine 10 or 20 highly overlapping dependencies of microservices into a single lib directory and multiple 100 or 200 KB business JARs for deployment, optimize the overall project deployment file size from one or two GB to 200 or 300 MB.

This article does not include:

  • Spring Boot configuration file separation is not included. It can be simply configured from an external YAML configuration file overwritten from a JAR file by specifying an active profile or using a configuration service pattern such as NaCos.
  • It does not include Maven best practice usage. It is included in the sample project for demonstration convenience, such as putting some configuration declarations that should be specific to each Boot module directly into the parent at the top level. Please note that the use is optimized for the actual situation.
  • The run-time mode does not include the executable JAR support reference, and the implementation in this paper is mainly oriented towards the Java-JAR run-time mode.

Slimming fight strange upgrade process

Level 0: Regular FAT JAR builds

Refer to the project directory: package-optimize-level0

Main configuration:

<build> <finalName>${project.artifactId}</finalName> <! Note that the project is just for the convenience of demonstrating the configuration. The plugin is configured and run directly in the Build section of Parent. However, in the real project, you need to put these definitions only in the Spring Boot module project (optimally using pluginManagement form), > <plugins> <plugin> < grouppid >org.springframework.boot</ grouppid > <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build>

Configuration output:

cd package-optimize-level0
mvn clean install

ls -lh package-optimize-app1/target/package-optimize-app1.jar
-rw-r--r--  1 lixia  wheel    16M Feb 24 21:06 package-optimize-app1/target/package-optimize-app1.jar

java -jar package-optimize-app1/target/package-optimize-app1.jar

Key notes:

  • (The current demo application only relies on a few components of the spring-boot-starter-web, and the total build output is only about 10 MB.) In practice, the output JAR of a single build can be anywhere from a few dozen MB to a hundred or two MB or more depending on the number of components the project depends on.
  • If you have a dozen or so microservices to deploy, that means you have to transfer one or two gigabytes of files, which can take a lot of time. Even a single update to individual microservices requires a transfer of one or two hundred MB.

Level 1: Common JAR-dependent separate builds

Refer to the project directory: package-optimize-level1

To solve a problem:

  • Reduce the file size of a single microservice JAR so that the deployment process can transfer files in seconds.

Main configuration:

Please refer to the following notes for key configuration instructions:

<build> <finalName>${project.artifactId}</finalName> <! Note that the project is just for the convenience of demonstrating the configuration. The plugin is configured and run directly in the Build section of Parent. However, in the real project, you need to put these definitions only in the Spring Boot module project (optimized to use pluginManagement form) to avoid interfering with other module projects such as Util, Common, etc. --> <plugins> <! > <plugin> < grouppid >org.apache.maven.plugins</ grouppid > <artifactId>maven-dependency-plugin</artifactId> <executions> <execution> <id>copy-dependencies</id> <phase>package</phase> <goals> <goal>copy-dependencies</goal> </goals> <configuration> <outputDirectory>${project.build.directory}/lib</outputDirectory> <excludeTransitive>false</excludeTransitive> <stripVersion>false</stripVersion> <silent>true</silent> </configuration> </execution> </executions> </plugin> <! > <plugin> <groupId>org.springframework. Boot </groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <includes> <! -- Include references that do not exist; exclude all Maven dependent JARs; > <include> <groupId>null</groupId> <artifactId>null</artifactId> </include> </includes> <layout>ZIP</layout> </configuration> <executions> <execution> <goals> <goal>repackage</goal> </goals> </execution> </executions> </plugin> </plugins> </build>

Configuration output:

cd package-optimize-level1
mvn clean install

ls -lh package-optimize-app1/target/package-optimize-app1.jar
-rw-r--r--  1 lixia  wheel   149K Feb 24 20:56 package-optimize-app1/target/package-optimize-app1.jar

java -jar -Djava.ext.dirs=lib package-optimize-app1/target/package-optimize-app1.jar

Realized effect:

  • A single build output JAR according to the number of dependent components of the project is usually only one or two hundred KB, which can basically be transmitted in seconds.
  • This is one of the most common optimizations you can see on the Web, and it’s worth digging deeper: if you have a dozen or so microservices, each with a JAR and a lib directory file, you’ll probably need to transfer one or two GB files for your first deployment.

Level 2: Merge all module dependent JARs into one lib directory

Refer to the project directory: package-optimize-level2

Solve the problem:

  • Merge all module dependent JAR to the same lib directory. Generally, due to the high overlap degree of dependent JAR of each module project, the total size of all service deployment files is basically two or three hundred MB
  • However, if -djava.ext.dirs =lib is used to load all JARs into each JVM, on the one hand, each JVM has completely loaded all the JARs consumed by resources, on the other hand, different versions of microservice components may cause version conflict

Main configuration:

Please refer to the following notes for key configuration instructions:

<build> <finalName>${project.artifactId}</finalName> <! Note that the project is just for the convenience of demonstrating the configuration. The plugin is configured and run directly in the Build section of Parent. However, in the real project, you need to put these definitions only in the Spring Boot module project (optimized to use pluginManagement form) to avoid interfering with other module projects such as Util, Common, etc. --> <plugins> <! Maven-jar-plugins: </ grouppid >org.apache.maven.plugins</ grouppid > </ grouppid > </ grouppid > <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifest> <addClasspath>true</addClasspath> <classpathPrefix>lib/</classpathPrefix> <useUniqueVersions>false</useUniqueVersions> </manifest> </archive> </configuration> </plugin> <! > <plugin> < grouppid >org.apache.maven.plugins</ grouppid > <artifactId>maven-dependency-plugin</artifactId> <executions> <execution> <id>copy-dependencies</id> <phase>package</phase> <goals> <goal>copy-dependencies</goal> </goals> <configuration> <! -- Each sub-module defines the corresponding attribute value of each module according to the actual level, To check that all microservice module dependent JAR files are combined and copied to the same directory, see the Boot-jar-output property definition in each submodule --> <outputDirectory>${boot-jar-output}/lib</outputDirectory> <excludeTransitive>false</excludeTransitive> <stripVersion>false</stripVersion> <silent>false</silent> </configuration> </execution> </executions> </plugin> <! > <plugin> <groupId>org.springframework. Boot </groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <includes> <! -- Include references that do not exist; exclude all Maven dependent JARs; > <include> <groupId>null</groupId> <artifactId>null</artifactId> </include> </includes> <layout>ZIP</layout> <! Maven-jar-plugin outputs the microservice JAR file to the same directory as the lib file. To refer to the same lib directory together, see the Boot-jar-output property definition in each submodule --> <! -- --> <outputDirectory>${boot-jar-output}</outputDirectory> </configuration> <executions> <execution> <goals> <goal>repackage</goal> </goals> </execution> </executions> </plugin> </plugins> </build>

All lib directory files and various micro-service build JARs are aggregated into the DevOps public directory.

The Meta-Info/Manifest file in the microservice JAR file generates a class-path attribute based on the list of module dependent components, thus avoiding different versions of the JAR:

Class-Path: Lib/spring - the boot - starter - web - 2.4.3. Jar lib/spring - the boot - starte r - 2.4.3. Jar lib/spring - the boot - 2.4.3. Jar Lib /spring-boot-autoconfigure -2.4.3.jar /spring-boot-starter-logging-2.4.3.jar lib/spring-boot-autoconfigure -2.4.3.jar /spring-boot-starter-logging-2.4.3.jar/logback-class-1.2.3.jar Jar lib/ log4j-api-1.7.30.jar lib/ log4j-t-o-slf4j-2.13.3.jar Jar lib/jul-to-slf4j-1.7.30.jar lib/jakarta.annotation-api-1.3.5.jar lib/ spring-core-5.3.3.jar lib/ SPR ing-jcl-5.3.4.jar Lib/snakeyaml - 1.27. Jar lib/spring - the boot - starter - json - 2. 4.3 the jar lib/Jackson - databind - 2.11.4. Jar Jar lib/ Jackson-annotations -2.11.4.jar lib/ Jackson-core-2.11.4.jar lib/ Jackson-datatype-jdk8-2.11.4.jar L The ib/Jackson - datatype - jsr310-2.11.4. Jar lib/Jackson - module - parameter - the name s - 2.11.4. Jar Lib/spring - the boot - starter - tomcat - 2.4.3. Jar lib/tomcat embed - core - 9.0.43. Jar lib/Jakarta. El - 3.0.3. Jar Lib/tomcat-embed-websocket-9.0.43. Jar lib/spring-web-5.3.4.jar lib/ spring-bean-5.3.4. jar lib/ tomcat-embed-websocket-9.0.43. Jar Lib/spring aop -- 5.3.4. Jar lib/spring - the context - 5.3.4. Jar lib/spring - expression - 5.3.4. Jar

Configuration output:

CD package-optimize level2 MVN clean install ls-LH devops/ total 912 drwxr-xr-x 34 lixia wheel 1.1k Feb 24 22:27 lib -rw-r--r-- 1 lixia wheel 150K Feb 24 22:31 package-optimize-app1.jar -rw-r--r-- 1 lixia wheel 149K Feb 24 22:31 package-optimize-app2.jar -rw-r--r-- 1 lixia wheel 149K Feb 24 22:31 package-optimize-app3.jar java -jar devops/package-optimize-app1.jar

Realized effect:

  • The startup process no longer requires the -djava.ext.dirs =lib parameter definition.
  • All microservice JARs refer to the common directory of all project merge dependent components, and the total size of the deployment files is typically 200 or 300 MB.
  • By customizing the Class-Path in the META-INFO/MANIFEST file of each micro-service JAR file to clearly indicate the dependent version component Class, we can solve the problem of version conflict of different components of each micro-service.

Level 3: Support for unofficial tripartite dependency components introduced by System

Refer to the project directory: package-optimize-level3

Solve the problem:

  • Some unofficial third party JARs, such as SDK JARs, are submitted to the Maven local private server for reference, which is the same as the normal dependency JARs.

However, in the absence of a Maven private server, a common simplification is to simply place the dependency JAR in the project and define it as a System Scope in the POM.

  • The maven-jar-plugin component does not have a direct parameter declaration that contains the specified scope for the component that was introduced in the POM in the systemPath manner.

These Scope-defined components will not appear in the META-INFO/MANIFEST without special processing, resulting in the run-time classes not being found.

Main configuration:

Please refer to the following notes for key configuration instructions:

<build> <finalName>${project.artifactId}</finalName> <! Note that the project is just for the convenience of demonstrating the configuration. The plugin is configured and run directly in the Build section of Parent. However, in the real project, you need to put these definitions only in the Spring Boot module project (optimized to use pluginManagement form) to avoid interfering with other module projects such as Util, Common, etc. --> <plugins> <! Maven-jar-plugins: </ grouppid >org.apache.maven.plugins</ grouppid > </ grouppid > </ grouppid > <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifest> <addClasspath>true</addClasspath> <classpathPrefix>lib/</classpathPrefix> <useUniqueVersions>false</useUniqueVersions> </manifest> <manifestEntries> <! The maven-jar-plugin component has no direct parameter declaration. The component that contains the specified scope is appended to the specified dependent component list by using an additional class-path value defined. In a submodule, you can specify the jar-manifestEnts-classpath value as applicable, for example (note the dot characters and space delimiters) : Lib /xxx-1.0.0.jar lib/yyy-2.0.0.jar See "->" for an example of boot-jar-output property definition in each submodule <Class-Path>${jar-manifestEntries-classpath}</Class-Path> </manifestEntries> </archive> </configuration> </plugin> <! > <plugin> < grouppid >org.apache.maven.plugins</ grouppid > <artifactId>maven-dependency-plugin</artifactId> <executions> <execution> <id>copy-dependencies</id> <phase>package</phase> <goals> <goal>copy-dependencies</goal> </goals> <configuration> <! -- Each sub-module defines the corresponding attribute value of each module according to the actual level, To check that all microservice module dependent JAR files are combined and copied to the same directory, see the Boot-jar-output property definition in each submodule --> <outputDirectory>${boot-jar-output}/lib</outputDirectory> <excludeTransitive>false</excludeTransitive> <stripVersion>false</stripVersion> <silent>false</silent> </configuration> </execution> </executions> </plugin> <! > <plugin> <groupId>org.springframework. Boot </groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <includes> <! -- Include references that do not exist; exclude all Maven dependent JARs; > <include> <groupId>null</groupId> <artifactId>null</artifactId> </include> </includes> <layout>ZIP</layout> <! Maven-jar-plugin outputs the microservice JAR file to the same directory as the lib file. To refer to the same lib directory together, see the Boot-jar-output property definition in each submodule --> <! -- --> <outputDirectory>${boot-jar-output}</outputDirectory> </configuration> <executions> <execution> <goals> <goal>repackage</goal> </goals> </execution> </executions> </plugin> </plugins> </build>

Main configuration of sub-module:

<properties> <! -> <boot-jar-output>... -> <boot-jar-output>... /devops</boot-jar-output> <! The maven-jar-plugin component has no direct parameter declaration. Components containing the specified scope are appended to the specified dependent component list by using an additional class-path value defined. You can specify the jar-manifestEnts-classpath value as appropriate, for example (note the dot character and space delimiter, and the artifactid-version. Jar format after the lib rather than the actual file name) : .lib /xxx-1.0.0.jar lib/yyy-2.0.0.jar --> < jar-manifestEnt-classpath >. Lib/hik - SDK - 1.0.0. Jar < / jar - manifestEntries - classpath > < / properties > < dependencies > <! > <dependency> < grouppid >com.hik</ grouppid > <artifactId>hik-sdk</artifactId> The < version > 1.0.0 < / version > < scope > system < / scope > < systemPath > ${project. The basedir} / lib/hik - SDK - 1.0.0. Jar < systemPath > </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> </dependencies>

In the microservice output JAR file, the Meta-Info/Manifest file generates a class-path attribute based on the list of module dependent components, and appends the Jar-ManifestEnd-classpath attribute with the specified values:

Class-Path: .lib /hik-sdk-1.0.0.jar lib/spring-boot-starter-web-2.4.3.ja r lib/spring-boot-starter-2.4.3.jar Log log log log log log log log log log log log log log log log log log log log log log log log log log log log log log log log log log log log log log Jar lib/ logback-class-1.2.3.jar lib/logback-core-1.2.3.jar lib/slf4j-ap i-1.7.30.jar lib/log4j-to-slf4j-2.13.3.jar Jar lib/log4j-api-2.13.3.jar lib/ jul-to-slf4j-1.7.30.jar lib/jakarta.annotation -api-1.3.3.jar lib/sprin g-core-5.3.4.jar Jar lib/spring-jcl-5.3.4.jar lib/snakeyaml-1.27.jar lib/ spring-boot-starter-json-2.4.3. jar lib/jackson-databind-2.11.4.jar Jar lib/jackson -annotations-2.11.4.jar lib/jackson-core-2.11.4.jar lib/jackson-da tatype-jdk8-2.11.4.jar Lib/Jackson - datatype - jsr310-2.11.4. Jar lib/jacks on the module - parameter - names - 2.11.4. Jar lib/spring - the boot - starter - tomcat 2.4 Jar lib/tomcat-embed-core-9.0.43. Jar lib/jakarta.el-3.0.3.jar lib/to McAt-embed-websocket-9.0.43. Jar lib/tomcat-embed-core-9.0.43. Jar lib/jakarta.el-3.0.3.jar lib/to McAt-embed-websocket-9.0.43 Jar lib/spring-web-5.3.4.jar lib/spring-web-5.3.4.jar lib/spring-web-5.3.4.jar lib/s Pring - context - 5.3.4. Jar lib/spring - expression - 5.3.4. Jar

Configuration output:

CD package-optimize-level3 MVN clean install ls-LH devops/ total 912 drwxr-xr-x 36 lixia wheel 1.1k Feb 24 23:14 lib -rw-r--r--@ 1 lixia wheel 150K Feb 24 23:14 package-optimize-app1.jar -rw-r--r-- 1 lixia wheel 150K Feb 24 23:14 package-optimize-app2.jar -rw-r--r-- 1 lixia wheel 150K Feb 24 23:14 package-optimize-app3.jar java -jar devops/package-optimize-app1.jar

The final effect

  • All the dependent components of the service are combined into a single directory, with a total size of 200 or 300 MB, which significantly speeds up the first deployment transport efficiency.
  • Each micro service JAR size of one or two hundred KB, daily emergency Bug repair update individual JAR is basically instant second transmission.
  • Each microservice JAR defines a list of components that depend on the specified version, so there will be no loading conflict between different versions of components.
  • Unofficial tripartite dependent components can also be referenced properly.

Special note

Once the above is handled separately by deploying components, daily updates only need to transfer a hundred or two hundred KB of business JAR files. However, if a project’s Maven dependent component makes a change in configuration, you need to be careful to synchronize the changed JAR files to the common lib directory.

Tips for minimizing changes to JAR files: The commit view can clearly identify the change files in the lib directory and the business JAR for this release, including the micro-service JAR and dependent JAR change files, so as to minimize the transfer of files.