Resolving server process exit problems (Metaspace overflow

The phenomenon of

Hprof file java_pid16298.hprof file appeared in the root directory of the process. It was a surprise to see that the memory overflow occurred. Is unlikely to be overflow, heap memory because of the small, preliminary suspect is permanent area overflow (Java8 # Metaspace), the following was verified, because the launch parameters added – XX: + HeapDumpOnOutOfMemoryError, also appeared hs_err_pid. The log, The JVM fatal error log.

Log query (vim/grep/less/more)

1. vim std.log esc /OutOfMemoryError ? OutOfMemoryError n/ n Next

2. less std.log | grep OutOfMemoryError
   
   Caused by: java.lang.OutOfMemoryError: Metaspace

3. The grep OutOfMemory STD. The log – A 50 to 50 B | less/OutOfMemoryError n q next exit

4. Less std.log appears colon /OutOfMemoryError search q exit also ok? OutOfMemoryError

5. More std.log /OutOfMemoryError Search q exit only /

   Caused by: java.lang.OutOfMemoryError: Metaspace at Java. Lang. This defineClass1 (Native Method) ~ [na: 1.8.0 comes with _40] the at Java. Lang. This. DefineClass (760). This Java: ~ [na: 1.8.0 comes with _40] the at Java. Security. SecureClassLoader. DefineClass (SecureClassLoader. Java: 142) ~ [na: 1.8.0 comes with _40] the at Java.net.URLClassLoader.defineClass URLClassLoader. Java: (467) ~ [na: 1.8.0 comes with _40] at java.net.URLClassLoader.access 100 (73) URLClassLoader. Java: [na: 1.8.040] atjava.net.URLClassLoader100 (URLClassLoader. Java: 73) ~ [na: 1.8.0 comes with _40] the at Java.net.URLClassLoader100 URLClassLoader. Java: (73)/na: 1.8.040 atjava.net.URLClassLoader1.run (URLClassLoader. Java: 368) ~[na:1.8.0_40] at JAVa.net.urlClassLoader () ~[na:1.8.0_40] at Java. Security. The AccessController. DoPrivileged (Native Method) ~ [na: 1.8.0 comes with _40] the at Java.net.URLClassLoader.findClass URLClassLoader. Java: (361) ~ [na: 1.8.0 comes with _40] the at Java. Lang. This. LoadClass (424). This Java: ~ [na: 1.8.0 comes with _40] the at Java. Lang. This. LoadClass (357). This Java: ~ [na: 1.8.0 comes with _40]

The log output shows that Metaspace is out of memory. I set the boot parameter size to 48M -xx :MaxMetaspaceSize= 48M

Metaspace used 47519K, Capacity 48950K, committed 49152K, reserved 1093632K… You can also see similar logs that indeed Metaspace is almost full.

why?

Metaspace Some of the data stored in Permgen in Java7 and Java8 has been moved to the Heap. Starting with the permanent generation removal, some of the data stored in the permanent generation has been moved to the Java Heap or Native Heap. However, the permanent generation still exists in JDK7 and is not completely removed: Symbols are transferred to the native heap; Literals (interned strings) are transferred to the Java Heap; Class statics are moved to the Java Heap.

In JDK 8, classes metadata is now stored in the native heap and this space is called Metaspace.

What takes up space (personal analysis is mostly generated classes)

Fastjson# asm(debug)

• deserializer

ASMDeserializerFactory# createJavaBeanDeserializer, When called like JSON#parseObject(String text, Class clazz), a clazz corresponding FastjsonASMDeserializer_53_xx Class is generated for deserialization, Current uses include profiles, spreadsheets, player-related data, etc.

• serializer

ASMSerializerFactory# createJavaBeanSerializer, When called such as json.tojsonString (Object Object), a class such as ASMSerializer_1_xx corresponding to Object #clazz is also generated for write/ serialization.

They add up to about 200.

All places where lambda expressions are used generate a class such as xx LambdaLambdaLambda1 (150 or so), others such as protobuf (200 or so), and other inner classes like $. Also special such as generated class has not been found, the search contains the class Numbers, because usually have dynamically generated similar Numbers, etc., found a lot of sun. Reflect. GeneratedMethodAccessor344… Probably has more than 350, discovered at the same time have the same number of sun, reflect the DelegatingClassLoader (is only one class, but there is a corresponding number of instances)… The same sun. Reflect. GeneratedConstructorAccessor… Java bytecode accessor the JVM initially uses JNI by default. Java Bytecode calls when the same class is called a certain number of times (a new classloader and a Clazz)

There is a lot of content on the web about the memory overflow problem because of this, you can search to see where the current business logic calls reflection frequently:

• Reflection execution of handler logic methods
• Deserialization of protobuf
• Other three-party library reflection, etc

Tool use

Jvisualvm# load hprof_installable plugin

Generated date: Mon Sep 25 14:30:30 CST 2017 file: D:\xx\landon\ Task \2017.9\server_err\java_pid16298.hprof File size: 56.1 MB Total bytes: 47,508,830 Total classes: 7,743 Total instances: 568,577 Class loaders: 380 Garbage collection root node: 2,703 Number of pending objects waiting to end: 0 Heap dump occurred when OutOfMemoryError occurred queue-executor-handler-8Copy the code

< span style = “box-sizing: block; line-height: 22px; display: block; font-size: 14px! Important;” Lambda expressions are used to generate inner classes. From the output, the inner class numbers generated by Lambda expressions start with 1, then ++. As you can see, there are 151 Lambda inner classes, and you can search for matches directly under the class information.

The use of mat

Open heap dump Size: 22.6MB Classes: 7.5K Objects: 578K Class Loader: 357

JavaBasics#class loader explorer

Class Name | Defined Classes | No. of Instances ------------------------------------------------------------------------------------------ Sun. Misc. The Launcher $ExtClassLoader @ 0 x800230b0 | 4312 | 93289 < system class loader > | 2617 | 484122 com.alibaba.fastjson.util.ASMClassLoader @ 0x805fd848| 129 | 129 com.alibaba.fastjson.util.ASMClassLoader @ 0x805e2858| 73 | 73 ------------------------------------------------------------------------------------------ Class Name | Shallow Heap | Retained Heap ---------------------------------------------------------------------------------------- class sun.reflect.GeneratedMethodAccessor344 @ 0x80593e18| 0 | 568 class sun.reflect.GeneratedMethodAccessor343 @ 0x80593ee0| 0 | 568 class sun.reflect.GeneratedMethodAccessor342 @ 0x80593fa8| 0 | 568 class sun.reflect.GeneratedMethodAccessor341 @ 0x80594070| 0 | 568 class sun.reflect.GeneratedMethodAccessor340 @ 0x80594138| 0 | 568 class sun.reflect.GeneratedMethodAccessor339 @ 0x80594200| 0 | 568 class sun.reflect.GeneratedMethodAccessor338 @ 0x805942c8| 0 | 568 class sun.reflect.GeneratedMethodAccessor337 @ 0x80594390| 0 | 568 class sun.reflect.GeneratedMethodAccessor336 @ 0x80594458| 0 | 568 ... ---------------------------------------------------------------------------------------- Class Name | Defined Classes | No. of Instances ---------------------------------------------------------------------------------------------------- Sun. Misc. The Launcher $ExtClassLoader @ 0 x800230b0 | 4312 | 93289 < system class loader > | 2617 | 484122 com.alibaba.fastjson.util.ASMClassLoader @ 0x805fd848 | 129 | 129 com.alibaba.fastjson.util.ASMClassLoader @ 0x805e2858 | 73 | 73 javax.management.remote.rmi.NoCallStackClassLoader @ 0x806fc4f8| 1 | 0 javax.management.remote.rmi.NoCallStackClassLoader @ 0x806fc5d0| 1 | 0 sun.reflect.DelegatingClassLoader @ 0x80593db8 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80593e80 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80593f48 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594010 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x805940d8 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x805941a0 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594268 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594330 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x805943f8 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x805944c0 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594588 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594650 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x805947c8 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594890 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594958 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594a20 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594ae8 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594bb0 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594c78 | 1 | 1 sun.reflect.DelegatingClassLoader @ 0x80594d40 | 1 | 1 ----------------------------------------------------------------------------------------------------Copy the code

The main several classloaders #sun.misc.Launch$ExtClassLoader#defined class 4312

com.alibaba.fastjson.util.ASMClassLoader(Deserializer_)#129 Com. Alibaba. Fastjson. Parser. The deserializer. FastjsonASMDeserializer_53_xxConfig, screening is needed here why all Config objects are generated deserialization inner class? There are other such as xx_R EdisConfig, etc. (understand fastjson#asm principle can be just)

com.alibaba.fastjson.serializer.ASMSerializer_70_xxConfig Discovered that a xxMonsterConfig here to see the code, this is in, didn’t add SerializerFeature serialization to redis. IgnoreNonFieldGetter, serialization mongo player has been unified with the feature.

Dirs =lib (ExtClassLoader), not AppClassLoader. OQL can be used for both. OQL Syntax, SELECT DISTINCT OBJECTS classof(s) FROM “com.xx.*” s There are about 600 classes that the query object belongs to in the com.xx package.

Summary and solutions

From the above analysis, it should be true that metaspace allocated too little space 48M, ready to adjust to 128M and then the actual running test

• Lambda expressions generate inner classes
• The JVM also generates classes when reflection calls frequently

conclusion

Because the cost of process crash is very high, although you can set a large metaspace, but if the leak, the process crash, the impact will be very large, so it is recommended not to set this parameter, JVM adjust itself. If there is a leak, the memory will continue to grow rapidly and our operation and maintenance monitoring system can listen to it, can immediately alarm, and then run the normal shutdown(before shutdown can be jMAP hprof), Ps: Shutdownhook will be executed when OutOfMemory crashes, but the process suddenly crashes, which will affect the player experience and may cause loss.

Author: landon30

Reference: club.perfma.com/article/192…