The problem

Some time ago, I found that the Full GC of a dubbo service on line was quite frequent, and it would be executed once every two days or so.

The cause of Full GC

We know that the trigger conditions of Full GC are as follows:

  1. The program executed system.gc () // suggesting that the JVM execute fullGC, but not necessarily
  2. Jmap-histo :live PID command // This will trigger fullGC immediately
  3. A series of checks that are performed during a MINOR GC
When the Minor GC is performed, the JVM checks to see if the maximum contiguous free space in the old generation is greater than the total size of all objects in the current generation. If it is, the Minor GC is performed directly (at this point it is not risky to perform). If it is, the JVM checks to see if the space allocation guarantee mechanism is enabled, and if it is not, it simply performs Full GC instead. If enabled, the JVM checks to see if the maximum contiguous available space in the old age is greater than the average size for successive promotions to the old age, and if so, performs Full GC instead. If it is greater, a Minor GC is performed, and if the Minor GC fails, a Full GC is performedCopy the code
  1. If you use large objects // large objects go straight to the old age
  2. A reference to an object that has been held in a program for a long time will age when the object reaches a specified threshold

For our situation, we can initially exclude 1,2, and 4 and 5 are most likely. To further investigate the cause, we turned on -xx :+HeapDumpBeforeFullGC online.

Note that a STOP the Word event occurs during a JVM dump operation, which means that all user threads are suspended. To provide services properly during this period, you are advised to use distributed deployment and appropriate load balancing algorithmsCopy the code

JVM parameter Settings:

The online dubbo service is distributed and -xx :HeapDumpBeforeFullGC is enabled on one of the machines. The overall JVM parameters are as follows:

-Xmx2g 
-XX:+HeapDumpBeforeFullGC 
-XX:HeapDumpPath=. 
-Xloggc:gc.log 
-XX:+PrintGC 
-XX:+PrintGCDetails 
-XX:+PrintGCDateStamps 
-XX:+UseGCLogFileRotation 
-XX:NumberOfGCLogFiles=10 
-XX:GCLogFileSize=100m 
-XX:HeapDumpOnOutOfMemoryError 
Copy the code

Dump File Analysis

The dump file is about 1.8 g, use jvisualvm view, found with char [] type of data takes up the 41% of memory, at the same time another com. Alibaba. Druid. Stat. JdbcSqlStat type of data takes up 35% of the memory, That means the heap is almost entirely composed of these two types of data. The diagram below:

When you look at the char[] data, you see that it’s almost all SQL statements.

Next look at the reference to char[] :

Public JdbcSqlStat(String SQL){this. SQL = SQL; this.id = DruidDriver.createSqlStatId(); } to look at the function call, find com. Alibaba. The druid. Stat. JdbcDataSourceStat# createSqlStat method:

public JdbcSqlStat createSqlStat(String sql) {
    lock.writeLock().lock();
    try {
        JdbcSqlStat sqlStat = sqlStatMap.get(sql);
        if (sqlStat == null) {
            sqlStat = new JdbcSqlStat(sql);
            sqlStat.setDbType(this.dbType);
            sqlStat.setName(this.name);
            sqlStatMap.put(sql, sqlStat);
        }

        returnsqlStat; } finally { lock.writeLock().unlock(); }} a map is used to store all SQL statements.Copy the code

Druid allows you to monitor SQL statements, and you have enabled the SQL statement monitoring function. Simply turning this off in the configuration file should have eliminated the problem, and it did, since turning this off no FullGC has been triggered online so far

other

If using MAT, it is recommended to check “Keep unreachable objects”, otherwise MAT will remove unreachable objects from the heap, and our analysis may become meaningless. Window–>References. In addition, JVisualVM does not support Ool very well, so mat is recommended if OQL is required.

Welcome to subscribe: