Brief introduction: Due to the particularity of ali mother league team responsible for the business, the system has a large external dependence, dependence on group LiuQiShiGe team service and utility components from N, we accumulated through this article and share with you some of the complex rely on effective management experience, in addition to simple technical skills summary, will also discuss some thinking about the architecture, I hope this article can completely solve the problem of Java dependency conflict.
The author source | | chengjiang ali technology to the public
An overview of
Due to the particularity of ali mother league team responsible for the business, the system has a large external dependence, dependence on group LiuQiShiGe team service and utility components from N, we accumulated through this article and share with you some of the complex rely on effective management experience, in addition to simple technical skills summary, will also discuss some thinking about the architecture, I hope this article can completely solve the problem of Java dependency conflict.
The essential causes of dependency conflict
To resolve dependency conflicts, we must first understand the nature of Java dependency conflicts.
The above picture shows an example. At present, most of Ali’s Java projects are Maven projects. These projects have to go through the following two important steps from development to launch:
1 Compilation and Packaging
Maven only relies on the first-level JAR package (A.jar, B.jar, *.jar) to complete the compilation of the application code. As for transferring the dependent JAR package (Y.jar, Maven will first arbitrate dependent jar packages of different versions with the same name, and then download the corresponding JARS according to the arbitration results and put them in the specified directory. (For example, y.jar in the figure above will arbitrate only one version 1.0 or 2.0. Z.jar does not belong to maven arbitration because the name of z.jar is not the same as y.Jar.
One thing to note is that there may be differences between Maven versions, which can lead to inconsistent application logic in some cases where the local environment is inconsistent with the daily and pre-packaged versions (note that there are other reasons for this, not necessarily the inconsistent arbitration results).
2 Release online
To clarify, in the JVM, an instance of a type is uniquely identified by its full class name and the ClassLoader instance that loaded it. Therefore, the so-called “class isolation” is actually achieved by loading the classes that need to be isolated through different instances of class loaders. In this way, even two classes with the same full name but different contents can coexist in a container process as long as they have different instances of class loaders, and run separately without interference.
When launching containers, whether Tomcat, Taobao Tomcat, PandoraBoot or other containers, the specific class loader instance is used to first load jar packages that the container itself depends on. Containers generally have multiple class loader instances. Jar packages relied on by containers themselves are generally loaded by special class loaders to achieve absolute isolation from application packages. For example, Pandroa also has special class loaders to load middleware to avoid conflicts between middleware and application classes, as shown in the following figure:
After the jar dependencies are loaded inside the container, the necessary step is to load the jar packages and the.class program generated during the compilation and packaging phase by an application ClassLoader instance (usually not the same as the container class loader instance), so that the container can run services and ensure that the application does not interfere with the container operation.
For example, as shown in Figure 1, both Y.jar-2.0 and Z.jar have com.taobao.Cc.class class in the final application package, but an application ClassLoader instance can load only one version of the com.Taobao.
Which version of the com.taobao.Cc. Class class will be loaded? The answer is not necessarily, it depends on the container application class loading implementation strategy, and from what we’ve seen before, Tomcat, Taobao – Tomcat and Pandora directly load the list of all. Jar package files under the application lib package (for example, A.jar,B.jar,*. Jar, y.jar, z.jar). In addition to tomcat have not read the source code verified, there is a mistake welcome to correct). But when Java loads all the JARS in a directory, the order in which it loads depends entirely on the operating system! The order of Linux depends entirely on the order of inodes. The order of inodes can not be completely consistent. Therefore, I have encountered similar problems before. In this case, the only way to solve the problem is to follow the methods in the following chapters. In theory, the most correct approach would be to load application JAR packages in the specified order when the container loads them.
Based on the above analysis, we can conclude that almost all class conflicts are essentially caused by either maven’s dependency on quorum JARS that do not meet runtime requirements, or by classes loaded during container class loading that do not meet runtime requirements.
There are a lot of information about container class loading isolation policies available on THE ATA website. This article focuses on the various ways to resolve conflicts. You only need to know the above principles.
Understanding the essential causes of dependency conflicts, how to effectively locate the specific JAR packages that cause conflicts? Please go on to the next chapter.
Efficient localization techniques for dependency conflict problems
The occurrence of dependency conflict is mainly manifested as an exception during system startup or operation, 99% of which are manifested as three NoClassDefFoundError, ClassNotFoundException, and NoSuchMethodError. The following explains the positioning techniques one by one.
1 Locate NoClassDefFoundError and ClassNotFoundException
“Could not initialize”, “Caused by:…”, “Caused by:…” Static code block exception causes class load failure:
A NoClassDefFoundError is caused by an exception in the static code block. Modify the static code block to avoid raising an exception. If the problem is not caused by an exception in a static code block, proceed to the next step.
STEP2. If loading fails due to an exception not occurring in the static code block, the missing class name will be clearly displayed in the exception message keyword, for example:
Step 3, in the IDEA (shortcut Ctrl + N) to find the abnormal stack tip missing classes, in which version of the jars in the example above org.apache.com mons. Lang. CharUtils
STEP4 check the application lib package directory (usually /home/admin/union-uc/target-/) on the application deployment machine ProjectName/lib or union – pub/target / {projectName} / lib or union – pub/target/projectName/lib or union – pub/target / {projectName}. The war/W Eb-inf /lib) check whether the version of the jar detected in the previous step exists. The above situation is generally because the application depends on the earlier VERSION of the JAR package, and the JAR package does not conflict with the class. The majority of NoClassDefFoundError and ClassNotFoundException localization acknowledgements are due to the fact that the version of the JAR package that maven’s dependency mediation ultimately adopted is not the version required by the runtime.
2 NoSuchMethodError Procedure
STEP1, NoSuchMethodError occurs, exception stack log core fragment (exception stack at the bottom of the fragment, have seen a lot of students abnormal mess over, that is meaningless, to have a purpose to turn the key place, do not mess over) will clearly show the specific class, which method is missing, exception stack core fragment example is as follows:
Caused by: java.lang.NoSuchMethodError: org.springframework.beans.factory.support.DefaultListableBeanFactory.getDependencyComparator()Ljava/util/Comparator; at org.springframework.context.annotation.AnnotationConfigUtils.registerAnnotationConfigProcessors(AnnotationConfigUtils.ja va:190) at org.springframework.context.annotation.ComponentScanBeanDefinitionParser.registerComponents(ComponentScanBeanDefinitionP arser.java:150) at org.springframework.context.annotation.ComponentScanBeanDefinitionParser.parse(ComponentScanBeanDefinitionParser.java:86 ) at org.springframework.beans.factory.xml.NamespaceHandlerSupport.parse(NamespaceHandlerSupport.java:73)Copy the codeFirst of all need to confirm the lack of the current load in the JVM method class, such as “org. Springframework. Beans. Factory. Support. DefaultListableBeanFactory” class from which the jar package, by far the most efficient way to:
In the case of an external environment container, or a container version that is too old to support Arthas online diagnostics, you can restart the system by adding “-xx :+TraceClassLoading” to the JVM startup parameter, and then see the JVM loaded classes in the system engineering log. You can find out from which JAR the JVM was loaded.
STEP2 in IDEA (shortcut key Ctrl+N) find out which versions of jar packages the missing classes in the abnormal stack are in, as shown in the picture below:
Then check the source code of the conflicting classes in each version of jar package. Some jars in the project are packaged with the source code package, and you can see the source code directly. If you do not have the source code, you need to decompile it with IDEA plug-in (recommended JAD). Then search for the conflicting classes in each JAR package in turn. The first step is to click a version class in the figure above and search for the class hierarchy relationship in IDEA (shortcut key Ctrl+H), as shown below:
Then find the “getDependencyComparator()Ljava/util/Comparator” exception in the NoSuchMethodError exception.
In the example above, spring-beans-3.2.1. Release.jar can be found by searching. Spring – 2.5.6. SEC03. Two versions of the jar DefaultListableBeanFactory class and none of the parent class “getDependencyComparator () Ljava/util/Comparator” method, Spring beans – 4.2.4. RELEASE. The jar, Spring beans – 4.3.5. RELEASE. Two versions of the jar DefaultListableBeanFactory class there missing “getDependencyComparator () Ljava/util/Comparator” method.
STEP3 check the application lib package directory (usually /home/admin/union-uc/target-/) on the application deployment machine ProjectName/lib or union – pub/target / {projectName} / lib or union – pub/target/projectName/lib or union – pub/target / {projectName}. The war/W Eb-inf /lib), find the version of the relevant JAR package, as in the above example:
To the location problem is driven by application startup loading “org. Springframework. Beans. Factory. Support. DefaultListableBeanFactory” class not loaded into the runtime expected the spring beans — 4.3.5. The release.jar version was instead loaded as a result of spring-2.5.6.sec03.jar.
Basically 99% of dependency conflicts can be located to the root cause by following the above process steps. How can conflicts be resolved once the cause is identified? In fact, sometimes resolving conflicts is not as simple as “MVN Dependency :tree” as many posts on the Intranet describe. Continue to the next section for details.
4. Resolve dependency conflicts by adjusting dependency JARS through Maven
1 Upgrade or degrade jar packages to resolve dependency conflicts
In the first example in the previous section, the simplest case is to simply upgrade the JAR package version in the POM if the higher version of the conflicting JAR package is fully compatible with the functionality of the lower version.
However, if the higher version of the conflicting JAR package is incompatible with the earlier version, and the application dependency is not very complex, you can analyze the impact of the upgrade of the conflicting JAR package on services. It is recommended to use the IDEA Maven Helper plugin to find the business dependencies of conflicting JARS (” MVN Dependency :tree” is not recommended; most Maven projects I have seen have multiple modules, such as
-dal,
MVN Dependency :tree () : MVN Dependency :tree () : MVN Dependency :tree () : MVN Dependency :tree As shown below:
The conflicting package is then upgraded and the business points corresponding to the affected two-party library are tested by regression.
If applications rely on very complex (for example, conflict with dozens of second party libraries depend on or rely on conflict package two party libraries is a basic package, clear business systems can’t enumerate two party libraries using the affected business point), in this case, if you want to by upgrading the jar package dependency resolution conflict, the application must be complete regression function. I have a few times because the regression is not fully caused by the tragic experience of failure, I hope you do not repeat the same mistake. Through this case, the author deep understanding to our age’s greatest computer scientists Dijkstra great god “is simple and reliable prerequisites” this words of wisdom, deeply realized if a system is a complex to you can’t completely clear his complex dependencies, that shows you the refactoring your system, Otherwise, system maintenance will gradually become a nightmare.
Of course, not all cases can be resolved by upgrading or degrading a JAR, for example:
As shown in the figure above, it is assumed that the application system relies on both A.jar and B.jar, while both a. jar and B.jar rely on protobuf-Java, and the system runs using the functions of the protobuf part of a. jar and B.jar respectively. B. Jar dependent Protobuf versions cannot be adjusted to be consistent by increasing or decreasing the version. All aspects of class content are incompatible with the Protobuf-Java version 2.0 due to the Protobuf-Java version 3.0 serialization protocol. In this case, adjusting dependencies in any way does not solve the problem of conflict. To solve the problem, read on to Chapter 5 and chapter 6.
Resolve dependency conflicts by excluding JAR packages
In the second example from the previous chapter, the main reason is that the container is not loaded with the expected spring-beans-4.3.5.release.jar class, but with the spring-2.6.3.sec03.jar package. If the spring-2.4.sec03.jar exclusion has no impact on services, you can resolve the conflict by excluding spring-2.4.sec03.jar. Similar to the example in the previous section, you can use the IDEA Maven Helper plugin to determine which JAR spring-2.5.6.sec03.jar is indirectly dependent on. As in the previous section, not all similar cases can be solved with exclusion jars.
Resolve dependency conflicts through Pandora custom plug-in
As discussed in Chapter 4, if an application has two incompatible VERSIONS of jar packages running at the same time, there is no way Maven can adjust dependencies. In chapter 2, when explaining the principle of dependency conflict, it is mentioned that Pandora realizes the isolation between the middleware of the group through the class isolation mechanism. Pandroa also supports the business side to create a plug-in that can run in the Pandora container according to the specification, and the container helps the business side to realize the loading isolation.
The alliance Etao team cut and packaged nuclear weapons packages such as IC and card coupons according to their own business needs and made Pandora plug-ins to avoid dependency conflicts, which achieved good results.
Using the Pandora plugin really does solve the dependency conflict problem perfectly without making too many changes to the application or affecting performance.
However, some problems are not suitable for local solutions, such as:
When maintaining application dependencies is too complex and each application relies on thirty or forty external binary libraries. This kind of heavyweight application can seriously affect productivity.
As you can see, in my early days in charge of affiliate user platforms, I came across two giant apps, ADV (6W + code) and PUB (12W + code).
On the one hand, due to the high dependence, I will encounter various upgrades, security problems and minor repairs of the group every week. On the one hand, there are more business publishing requirements.
This resulted in frequent posts, such as 566 in one year. At this point, the huge dependency will lead to deployment efficiency and impact assessment regression will be difficult. At this point, we should not view from the perspective of local conflict resolution, but should consider optimizing the application architecture and implementing dependency governance to avoid conflicts as far as possible.
Resolve dependency conflicts through dependency architecture governance
Complexity depends on standardization and simplified governance
First, dependency is itself a complex business. Most dependencies have deep business domain knowledge or technical domain knowledge behind them.
Let’s say we query search.
In terms of business domain knowledge, sales volume alone includes number of transactions, number of transactions, search sales volume (some orders are not included in search sales volume), etc.
Technical field knowledge, the main search, alliance advertising search engine is sometimes used together, such as the merchant before entering the advertisement to show the goods information to the merchant need to check the main search, and after entering the advertising down need to use the advertising engine. Different engines call different methods.
As shown in the figure below, if each business application is implemented separately, each application developer will have to digest a lot of business and technical knowledge related to the search client. The cost is high.
Faced with such a situation, if such complex dependencies are unified and standardized by the owner, it will greatly improve the collaborative efficiency of the organization. As shown in the figure below.
We did this through a unified encapsulation of the main search engine, the federation engine. Standardized encapsulation of returned results for search criteria. It has greatly reduced the cost of access for students. In the past, it took about 2 days to get familiar with the access of an engine. It only took 0.5 days to use the wrapper with standardized packaging under the guidance of special personnel and standard documents, which greatly improved the efficiency.
Heavyweight dependency proxy servitization
As mentioned in Section 5, too many jar packages will cause your application to start slowly. Therefore, be careful if more than 30 JAR packages are imported into a dependency. For example, IC, TP, the two-party package of the discount center is a typical example.
Currently we address this dependency by directly encapsulating a standard proxy service to avoid the application being slowed down by such a giant binary package.
Through the above comprehensive treatment means, good results have been achieved. The alliance rarely needs everyone to resolve conflicts any more.
The original link
This article is the original content of Aliyun and shall not be reproduced without permission.