In-depth Understanding of the JVM – Live JVM Tools (Part 2)

preface

Following up on the previous article, which simulated two somewhat familiar scenarios and analyzed how old-time optimizations were handled and how JVM problems were solved using the Jstat analysis tool, this section expands to include more examples of JVM problems online.

These reviews

The previous section explained how some of the more specific parameters affect the JVM through an APP’s JVM memory analysis, as well as analyzing how jSTAT was analyzed and optimized in a previous article on old-time optimization.

Summary:

  1. Introduces three JVM tuning cases, with a step-by-step analysis of the problems and solutions.
  2. Summarize and analyze the thinking and solving process, self-reflection and reflection.
  3. Summary and personal feelings.

Case of actual combat

Case 1: Method area crashes continuously how to check?

Business scenarios:

Previous cases involved in basic and heap is, this case is special is caused by the JVM parameter setting error frequently card problem, is simply set up parameters appear caton and online access system for FULL GC alarm, here don’t show which changed parameters, but to look at the first:

  • It was found that the system performed three FULL GC sessions in ten minutes, which was a very high frequency
  • A JSTAT check on the line found an errorMethoddata GC ThreasholdThe words etc.

The FULL GC is triggered by a JVM method overflow. The FULL GC is triggered by a JVM method overflow. ** This is actually true, because FULL GC usually drives the method area collection as well. This piece of information online can be searched to a lot of, here is no longer specific introduction.

Problem analysis:

Add verification parameter analysis:

Since this is a method area problem, in order to further check this method area overflow problem, we need to add the following two parameters:

  • -XX:TraceClassLoading
  • -XX:TraceClassUnloading

The log file contains the following contents:

Can clearly see the JVM constant loading a class called GeneratedSerializationCOnstructorAccessor, is this class load increasing metaspace area fill, this also caused the metaspace objects too much and trigger a FULL GC, So the culprit is the strange GeneratedSerializationCOnstructorAccessor.

Why are there strange classes?

Here we do a Google search to see what this class is. The query turns out to be a built-in JDK class, and by looking up the data we can see that this class is generated by reflection.

Reflection is no longer added, and can be understood as a way to create objects through the JVM’s classloader combined with the JVM’s toolkit, which is the soul of many frameworks.

In fact, reflection requires the JVM to dynamically produce some of the above strange classes to the MetaSpace region, such as to generate the dynamic class ProxyClass@Proxy123123 and other objects (reflection produces a special class identifier). The JDK requires the above helper objects to operate.

Here we also need to understand the concept of ** reflection generated classes are using soft references! ** As to the impact of this soft reference, here is also sold in the future, to the following combined with the system crashed parameters together analysis.

** What is a soft reference? ** is forcibly reclaimed when there is insufficient memory, regardless of whether local variable references exist.

Following the analysis of the lifetime of the soft reference, the JVM uses the following formula to calculate the lifetime of the soft reference:

Freespace indicates how long a SoftReference has not been accessed, freespace indicates the free memory in the current JVM, and softref indicates how long a SoftReference is allowed to live for each MB of memory space.

Estimated value: Assuming that there are 3000M objects in the current space, softrefLRUpolicyMSPerMB has a value of 1000ms, which means that these objects will live for 3000 seconds or about 50 minutes.

That’s where the weird classes come in, because of reflection.

Investigation Results:

What parameters are set?

We set the following parameters to allow reflection to generate objects to fill the method area:

The parameter – XX: SoftRefLRUPolicyMSPerMB = 0. As a result, the JVM overturned.

​

Why are there more and more strange objects?

Let’s look at the formula above again:

What if we accidentally set this value to zero?

The result is that the JVM finds that the allocated object is immediately recycled each time it is reflected, and then generates the proxy object through the proxy. In short, this parameter causes the soft reference object to be recycled as soon as it is allocated.

Again, reflection causes dynamic proxy classes to be added, but these objects are immediately recycled, resulting in more and more garbage objects in the method area. This is why there are more and more strange objects.

Why do you want to set this parameter?

The reason for setting this parameter is also naive: to make the proxy objects generated by reflection garbage collected as quickly as possible, if set to 0, the memory footprint of the method area will be smaller and can be collected in time, and the result is good intentions.

Solutions:

The solution here is simply to set a value greater than 0, preferably 1000, 2000, 5000, etc., but do not set it too small or set it to 0, otherwise it will cause the method area to constantly occupy the result method overflow and eventually lead to FULL GC.

Conclusion:

This case solved may look at the instructions is very simple, actually encountered similar problems, however, is sure to appear various scratching their heads, I hope this case when the JVM parameter can be set for readers to investigate the influence of the parameters and make sure his parameters and actual effect is the same!

This problem is also entirely a human problem, adding no curiosity to set a strange parameter for granted, will not cause all kinds of strange problems.

Finally, only to learn more practical cases, usually see how others troubleshoot problems, which is also very helpful to their own improvement.

Case 2: How to deal with the online system with tens of GC per day?

Business scenarios:

This case, like the previous one, is a practical one. Without further ado, it can be concluded that the JVM performance of the then unoptimized system is roughly as follows:

  • Machine configuration: 2 core 4G
  • JVM heap memory size: 2G
  • System running time: 6 days
  • The number and time of Full GC within 6 days of system operation: 250 times, over 70 seconds
  • The number and time of Young GC within 6 days of system operation: 26,000 times and 1400 seconds

Problems will be found as follows when using:

  • Full GC occurs over 40 times per day, an average of 2 times per hour, each timeFull GCAround 300 milliseconds
  • YGC occurs more than 4,000 times per day, 3 times per minute, with each YGC lasting around 50 seconds.

As you can see from this introduction, the performance of this system is quite poor, with Full GC every 2 hours. This has to be optimized.

JVM parameters before optimization:

The following are the parameters set before system optimization:

-Xms1536M -Xmx1536M -Xmn512M -Xss256K -XX:SurvivorRatio=5 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=68 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
Copy the code

Without looking at anything else, let’s take a look at the points in the parameters themselves that might make GC frequent:

  1. As you can see, small machines don’t have much heap memory for the JVM, threads and methods have to be split up a bit, and JVM resources are stretched.
  2. The Cenozoic generation is obviously too small, and the ratio is set to 5, resulting in a pitiful 300M or so of EDEN in the end
  3. The collection threshold of 68% of CMS in the old age seems a little small, and can be changed to 92% to execute the collection, because the space in the old age is relatively large
  4. -XX:+CMSParallelRemarkEnabledand-XX:+UseCMSInitiatingOccupancyOnlyPlease check the function of these two parameters.

Problem analysis

  1. In the subsequent trying to find every ten minutes will appear a lot of large objects directly into old age, the reason of large object is due to the developers to use the “full table query” led to hundreds of thousands of data was found out, here you can use the jmap tool testing found that generates a large ArrayList, and inside are the same object.

  2. Although few objects entered the old age after Cenozoic recovery (tens of meters), survior still entered the old age space after dynamic rule judgment.

  3. The new generation of space is easy to full, the old reserved space is larger.

  4. If the threshold value of CMS is set to 68, the collection will start when it reaches 68 in the old age, which is a little too conservative

Solutions:

  1. If possible, it is necessary to add machines, because the performance of machines is really limited. (2G I can’t even open IDEA)
  2. The Cenozoic era was clearly too small, so it was necessary to scale up to 1G of space while still giving survior space and size in a 5:1:1 allocation scheme.
  3. The CMS reclamation threshold is set to 92%. You don’t have to be too conservative.
  4. The method area specifies a size of 256M. If no parameters are set, the default method area size is only about 64M.

Optimized parameters:

The following are the parameters after system optimization:

-Xms1536M -Xmx1536M -Xmn1024M -Xss256K -XX:SurvivorRatio=5 -XX:PermSize=256M -XX:MaxPermSize=256M  -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=92 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC
Copy the code

Here again to emphasize the above mentioned need baidu parameters:

+ UseCMSInitiatingOccupancyOnly – – XX: XX: + CMSParallelRemarkEnabled these two parameters, these parameters have what use?

The first parameter: indicates that garbage collection is performed every time the CMS meets its set goal, without the JVM’s dynamic judgment leading to a FULL GC ahead of time

(This parameter needs to be explained, because the previous article mentioned that the default ratio of CMS after JDK6 is 92%, but the actual implementation of this ratio can be advanced or delayed according to the CMS old situation, but the JVM details are a little too much, just remember that when this parameter is enabled, – CMS will fix the Full GC garbage collection until it reaches 92%.

The second parameter indicates that YGC should be performed before the step of concurrent marking. In theory, it should be performed once, but in practice, if the JVM is not configured, it will decide whether to perform YGC based on the actual situation.

Case 3: Severe FULL GC causing jam?

Finally, a simple case, really simple, can be read in 3 minutes:

Business scenarios:

  1. FULL GC takes place once a second, each time taking hundreds of milliseconds
  2. At ordinary times, when the flow is not large, the growth of the new generation object is not fast, the old generation occupies less than 10%, and the method area is only 20% used. But once the peak period is frequent FULL GC

Analysis:

GC log analysis was performed at frequent FULL GC points, and JMAP analysis was used to find that a large number of objects occurred during peak periods. This object was based on a report batch processing operation, which produced a large number of objects and immediately set out to collect. It was found that there was no room in the JVM to cause frequent FULL GC.

This is strange because we all know that even under normal circumstances, even if the bulk processing of large data is not so bad as FULL GC once a second in most cases, this is definitely a problem with the code.

It turns out that a developer has manually called garbage collection, namely system.gc (). This is a notorious method, and can be explained in Article 8 of Effective Java: Avoid finalizing and cleaning methods.

Solutions:

To prevent system.gc () from taking effect, this is disabled with the following arguments:

​ -XX:+DisableExplictGC

Conclusion:

Do not write system.gc (), and it is best not to know about this method at all. To explore the cause is actually a waste of time.

Write in the last

This is the end of the first two articles on the JVM tools, and the following articles will still be the actual part. From these cases, we can see that in more cases, it is not the JVM that is the problem, but the human problem.

So writing good and easy to understand code is the duty, writing good performance code is the embodiment of proficiency. Only by writing a good code can we avoid all kinds of puzzling problems that are difficult to troubleshoot online, and mastering online troubleshooting and thinking means can effectively exercise one’s ability. Therefore, more experiments and attempts are the significance of this article

References:

GeneratedSerializationConstructorAccessorThe information

1: How the sun, reflect GeneratedSerializationConstructorAccessor class generated have special access to the Internet pose suggested reading

The answer from stackoverflow.com/questions/1…

Here's an excerpt from the answer:Copy the code

The first answer:

This is because (probably because you use reflection in your application) there is not enough heap space, and the GC tries to free up some memory by Unloading unused objects, which is why you see an semantics class

​ sun.reflect.GeneratedSerializationConstructorAccessor

The second answer

Method accessors and constructor accessors are either native or generated. This means that we use NativeMethodAccessorImpl or GeneratedMethodAccessor for methods, The constructor to use NativeConstructorAccessorImpl and GeneratedConstructorAccessor. Accessors can be native or generated and are controlled and determined by two system properties:

sun.reflect.noInflation = false(The default isfalse) sun. Reflect. InflationThreshold =15(The default is15)Copy the code

When sun. Reflect. NoInflation when set to true, the visitor will always generate the use of system properties sun. Reflect. InflationThreshold doesn’t make any sense. When sun. Reflect. NoInflation to false and sun, reflect. When the inflationThreshold to 15 (if not specified, this is the default behavior), then this means that for the constructor (or methods) 15 times before the visit, The native generator will be used, after which a generated accessor (from the ReflectionFactory) will be provided for use.

Native accessors use local calls to access information, and the generated accessors are bytecode, so they are very fast. The generated accessor, on the other hand, takes time to instantiate and load (basically bloat, so the name of the system property that controls it includes the word “bloat”).

More details can be found on the original blog at…..