1 Service Scenario

Service: Export annotation data in the production environment. Because there is no limit on the number of batches, the batch data size may be large. Storage: MongoDB Batch size: 1.6W data JDK version: 1.8

2 Error occurrence and location

The following error message is displayed:

STDERR: Exception in thread "pool-4-thread-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
STDERR: at org.bson.Document.<init>(Document.java:57)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:146)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:45)
STDERR: at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47)
STDERR: at org.bson.codecs.DocumentCodec.readValue(DocumentCodec.java:222)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:151)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:45)
STDERR: at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47)
STDERR: at org.bson.codecs.DocumentCodec.readValue(DocumentCodec.java:222)
STDERR: at org.bson.codecs.DocumentCodec.readList(DocumentCodec.java:229)
STDERR: at org.bson.codecs.DocumentCodec.readValue(DocumentCodec.java:218)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:151)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:45)
STDERR: at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47)
STDERR: at org.bson.codecs.DocumentCodec.readValue(DocumentCodec.java:222)
STDERR: at org.bson.codecs.DocumentCodec.readList(DocumentCodec.java:229)
STDERR: at org.bson.codecs.DocumentCodec.readValue(DocumentCodec.java:218)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:151)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:45)
STDERR: at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47)
STDERR: at org.bson.codecs.DocumentCodec.readValue(DocumentCodec.java:222)
STDERR: at org.bson.codecs.DocumentCodec.readList(DocumentCodec.java:229)
STDERR: at org.bson.codecs.DocumentCodec.readValue(DocumentCodec.java:218)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:151)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:45)
STDERR: at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47)
STDERR: at org.bson.codecs.DocumentCodec.readValue(DocumentCodec.java:222)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:151)
STDERR: at org.bson.codecs.DocumentCodec.decode(DocumentCodec.java:45)
at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52)
STDERR: at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60)
STDERR: at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84)
Copy the code

Keywords: GC overhead limit exceeded; From the above it is not difficult to see the error information is at the end of the error message on the com.mongodb.operation.Com mandResultArrayCodec. The decode () the execution of the error

2.1 GC overhead limit exceeded

Oracle database

Exception in thread thread_name: java.lang.OutOfMemoryError: GC Overhead limit exceeded Cause: The detail message “GC overhead limit exceeded” indicates that the garbage collector is running all the time, and the Java program is making very slow progress. After a garbage collection, if the Java process is spending more than approximately 98% of its time doing garbage collection and if it is recovering less than 2% of the heap and has been doing so for the last 5 (compile time constant) consecutive garbage collections, then a java.lang.OutOfMemoryError is thrown. This exception is typically thrown because the amount of live data barely fits into the Java heap having little free space for new allocations. Action: Increase the heap size. The java.lang.OutOfMemoryError exception for GC Overhead limit exceeded can be turned off with the command-line flag -XX:-UseGCOverheadLimit.

This means that by default, the JVM throws this error if the Java process spends more than 98% of its time performing GC and less than 2% of the heap is recovered each time. That is, our application ran out of nearly all available memory, and the garbage collector spent too long trying to clean it up and failed multiple times, which is a heap overflow

2.2 Analysis from JVM memory structure

Original address:www.processon.com/view/link/5…

Before JDK1.8, strings were stored in Perm. In JDK1.8, strings were stored in local memory and VM parameters were executed by default in JDK1.8-XmxThe default value is 1/4 of the maximum memory on your current machine-XmsThe default value of the current machine for you 1/64 of the maximum amount of memory, and the memory size of the machine production is 128 g, so the program can be up to 32 gb available heap memory theory (reach) actually, according to the understanding of normal memory is enough big, as long as the physical machine. It is hard to discern the Java should lang. An OutOfMemoryError, When the actual process started running, I observed that the memory swished up, and the program basically stopped moving at about 24G of memory

2.3 Code Location

Previously derived logic (simplified) :

Step 1: MongoDB query pull all data Step 2: Handler for data deduplication and good and bad data grouping and filtering operations third supplement: generate XML files one by one for the processed results and export them to the specified folderCopy the code

It’s easy to tell if our program is going wrong in step 1. Without further ado, let’s just look at the code

public List<Result> queryResultListByCurStep(ExportParam param) throws Exception {
    /* all data is stored in memory and is not released, and the GC cannot collect */
    List<Result> results = new ArrayList<>();
    CloseableIterator<Result> iterator = null;
    try {
        /* We used the streaming API of MongoDB to obtain data, which can effectively prevent the MongoDB query problem caused by too many results
        iterator = generateResultStream(param);
        while (iterator.hasNext()) {
            // The problem is this sentence
            results.add(iterator.next());
        }
        iterator.close();
    } catch (Exception e) {
        log.error("Query error E:" + e.getMessage());
        iterator.close();
    }
    return results;
}
Copy the code

In fact, the problem has been relatively clear in the comments, we can start to change. PS: Some let the configuration of -xx: -usegcoverheadlimit, close the abnormal times when GC occupies a long time, or increase the heap memory space, this is a method of drinking poison to solve the problem, hiding the ear and cheating the alarm

3 transformation

3.1 Logical optimization scheme

Step 1: start command additional start resultType parameters, enumerated values (Default | null: original execution logic unchanged; Stream: incoming handler directly introduced into the Stream iterators 】 step 2: Change the handle method to receive the CloseableIterator<Result> method, compatible with streamExport in the code (in addition to pagination query is also a way, here directly to bypass this solution) Because the data is taken one by one, it is necessary to redundancy the good and bad data filtering and multi-link re-related business logic when the file is dynamically generated. Because it is "counting one by one", GC can recover normally and naturally solve the problemCopy the code

PS: When we transform our business, we should not easily overturn and completely rewrite it. We need to expand on the original basis and make multiple plans compatible at the same time

Lu code 3.2

Parameter receiving class

@parameter (names = {"-resultType"},description = "resultType")
protected String resultType;
Copy the code

ResultType enumeration

public enum ResultType {
    Default,
    Stream;
    
    public static ResultType vOf(String type) {
        try {
            return ResultType.valueOf(type);
        } catch (Exception e) {
            returnDefault; }}}Copy the code

Handler code excerpts

@Override
public void handle(ExportParam param) throws Exception {
    String zipPath = getZipPath(param, param.getResultList());
    switch (param.getResultType()) {
        case Stream:
            streamExport(param, zipPath);
            break;
        default:
            defaultExport(param, zipPath);
            break; }}private void streamExport(ExportParam param, String zipPath) {
    Map<String, SimpleResult> resultMap = new HashMap<>();
    // File count
    int i = 1;
    try (CloseableIterator<Result> iterator = param.getIterator()) {
        while (iterator.hasNext()) {
            if (i++ % 10= =0) {
                log.info("============ stream processing progress: {} result ===============", i - 1);
            }
            Result result = iterator.next();

            /* Duplicate data filtering */
            if (resultMap.containsKey(result.getFileId())) {
                i--;
                SimpleResult oldResult = resultMap.get(result.getFileId());
                // Replace old and new data. If there is old data, delete the file and regenerate it
                dealEqualStep(oldResult, result, resultMap);
                continue;
            }
            resultMap.put(result.getFileId(), SimpleResult.builder().build());
            
            /* Good and bad data filtering */
            if(! filter(param, result)) {continue;
            }

            // Single file exportexportSingleResult(result, zipPath); }}catch (Exception e) {
        log.error("Result read failed", e); }}Copy the code

After the transformation was completed, the test went online. A total of 1.6W XML files were generated. After the zip decompression, the size of the text files was about 700M, with an average of 60K each. Has now been solve this problem, the program run memory footprint is back to normal, but because of the single thread was running, ran nearly 20 minutes, 1.6 W a file is a little slow, if later have optimization requirements, the resulting data can be segmentation, adopt the way of multithreading generated file, theoretically speed can reach (single thread takes/threads).

Reference link: blog.csdn.net/qq_37598011…