Takeaway: when a serious shortage of the JVM memory, will be thrown. Java lang. An OutOfMemoryError. This article summarizes common OOM causes and solutions, as shown in the figure below. If there are omissions or mistakes, welcome to add corrections.

Java Heap Space

When the Heap memory (Heap Space) does not have enough Space to store the newly created object, will be thrown. Java lang. OutOfMemoryError: Javaheap Space error (according to the actual production experience, You can configure a keyword alarm for OutOfMemoryError in the program log and handle it immediately if it is found.

– Cause analysis

Common causes of Javaheap space errors fall into the following categories:

  • Request to create a very large object, usually a large array.
  • Unexpected traffic/data volume, usually a spike in upstream system requests, is common in various promotional/kill activities, and can be combined with business traffic indicators to check for spikes.
  • Overusing finalizers, this object is not immediately GC.
  • A Memory Leak, in which a large number of object references are not released and cannot be automatically reclaimed by the JVM, is common when resources such as files are used and not reclaimed.

– Solution

For most cases, it is usually just a matter of scaling up the JVM heap memory space with the -xmx parameter. If the problem persists, refer to the following conditions for further handling:

  • If the object is very large, you can check its reasonableness, such as whether all the results of the database are queried at once without limiting the number of results.

  • In case of peak service pressure, add machine resources or perform traffic limiting degradation.

  • If there is a memory leak, you need to find the holding object and modify the code design, such as closing the unfreed connection.

2, GC overhead limit exceeded

When Java process takes more than 98% of the time doing GC, but back to less than 2% of the memory, and repeated five times in a row, the activity will be thrown. Java lang. OutOfMemoryError: GC overhead limit exceeded the error. Simply put, the application has almost exhausted all available memory and the GC cannot reclaim it.

The causes and solutions for this kind of problem are very similar to Javaheap space, as mentioned above.

3, Permgen space

This error indicates that the Permanent Generation is full, usually because too many classes are loaded or too large.

– Cause analysis

Permanent generation storage objects include the following types:

  1. Load/cache class definitions into memory, including class names, fields, methods, and bytecodes;
  2. The constant pool;
  3. The class associated with an array of objects/types;
  4. Class information optimized by the JIT compiler.

The amount of PermGen used is positively correlated with the number/size of classes loaded into memory.

– Solution

Depending on the timing of the Permgen Space error, different solutions are available, as follows:

  • Program startup error, modify -xx :MaxPermSize startup parameter, increase the permanent generation space.
  • Application redeployment error, most likely no application has been restarted, resulting in the loading of multiple class information, just need to restart the JVM to resolve the problem.
  • The application may create a large number of classes on the fly with a short lifetime, but the JVM does not unload classes by default. The -xx :+CMSClassUnloadingEnabled and -xx :+UseConcMarkSweepGC parameters can be set to allow the JVM to unload the class.

Jmap -dump:format=b,file=dump.hprof, Then use the Eclipse MAT www.eclipse.org/mat function to analyze the most expensive classloaders and duplicate classes one by one.

4, Metaspace

JDK 1.8 used Metaspace instead of Permanent Generation. This error indicates that Metaspace is full, usually because of too many classes loaded or too large.

The causes and solutions to these problems are very similar to Permgenspace, as mentioned above. Note that the startup parameter to adjust the Metaspace space size is -xx :MaxMetaspaceSize.

5, Unable to create new native thread

Each Java thread consumes a certain amount of memory, and this error is reported when the JVM requests the underlying operating system to create a new native thread if there are not enough resources to allocate.

– Cause analysis

When a JVM fails to request the OS to create a nativethread, it will throw an Unableto CreatenewnativeThread. Common reasons include:

  1. The number of threads exceeds the ulimit limit.
  2. The number of threads exceeds kernel.pid_max (restart only);
  3. Native memory is insufficient.

The common process of this problem mainly includes the following steps:

  1. An application within the JVM requests the creation of a new Java thread;
  2. The JVM Native method proxies this request and asks the operating system to create a native thread.
  3. The operating system tries to create a new native thread and allocate memory for it;
  4. If the operating system runs out of virtual memory or is limited by the address space of 32-bit processes, the operating system will reject this native memory allocation.
  5. The JVM throws Java. Lang. OutOfMemoryError: Unableto createnewnativethread errors.

– Solution

  • Upgrade configuration to provide more memory for the machine;
  • Reduce the Java Heap Space size;
  • Fix thread leaks in applications
  • Limit thread pool size;
  • Reduce the size of the thread stack with the -xss argument;
  • To increase the maximum number of threads on the OS layer, run the ulimia-a command to view the maximum number of threads and run the ulimit-u XXX command to adjust the maximum number of threads.

ulimit -a …. ….. max user processes (-u) 16384

6. Out of swap space?

This error indicates that all available virtual memory has been used up. Virtual Memory consists of Physical Memory and Swap Space. Outof swap Space is reported when the virtual memory requested by the runtime runs out. Error.

– Cause analysis

Common reasons for this error include the following categories:

  • Insufficient address space;
  • Physical memory is used up.
  • A native leak in an application, such as an application that continuously requests local memory but does not release it.
  • Run the jmap-histo:live command to force Full GC. If the memory decreases significantly after several times of execution, the Direct ByteBuffer problem is basically confirmed.

– Solution

Based on the error causes, you can take the following solutions:

  • Upgrade the address space to 64 bits.

  • Use Arthas to check for Inflater/Deflater decompression problems and, if so, explicitly call the end method.

  • For Direct ByteBuffer problems, the threshold can be lowered with the startup parameter -xx :MaxDirectMemorySize.

  • Upgrade server configuration/quarantine deployment to avoid contention.

Kill process or sacrifice child

There is a Kernel Job called Out of Memory Killer that “kills” certain processes when the available Memory is low. Surviving the Linux OOM Killer scores all the processes and then “kills” the ones with lower ratings.

Unlike other OOM errors, the Killprocessorsacrifice Child error is not triggered at the JVM level, but at the OS level.

– Cause analysis

By default, the Linux kernel allows processes to apply for more memory than the available memory of the system. In this way, system resources can be used more efficiently.

However, this approach will inevitably bring a certain “oversold” risk. For example, some processes continue to occupy system memory, which then causes other processes to have no memory available. At this point, OOM Killer is automatically activated to look for low-scoring processes and “kill” them to release memory resources.

– Solution

  • Upgrade server configuration/quarantine deployment to avoid contention.

  • OOM Killer Was tuned.

8, Requested array size exceeds VM limit

The JVM limits the maximum size of an array. This error indicates that the program is requesting an array that exceeds the maximum size limit.

Before allocating memory to an array, the JVM checks whether the data structure to be allocated is addressable in the system, typically integer.max_value-2.

This type of problem is rare and typically requires a code review to determine whether the business needs to create such a large array and whether it can be broken up into chunks and executed in batches.

9. Direct buffer memory

Java allows applications to access out-of-heap Memory directly using Direct ByteBuffer. Many high performance programs use Direct ByteBuffer in conjunction with Memory Mapped files to implement high speed IO.

– Cause analysis

The default size of Direct ByteBuffer is 64 MB, and if the limit is exceeded, a Directbuffer Memory error is thrown.

– Solution

  1. Java only through a ByteBuffer. AllocateDirect method using Direct ByteBuffer, therefore, can be diagnosed by Arthas and other online tools to intercept the method for screening.
  2. Check whether NIO is used directly or indirectly, such as Netty, Jetty, etc.
  3. Adjust the upper limit of Direct ByteBuffer using the startup parameter -xx :MaxDirectMemorySize.
  4. Check that the JVM argument has the -xx :+DisableExplicitGC option and remove it if it does, as it invalidates System.gc().
  5. Check the out-of-heap memory usage code to see if there is a memory leak; Or using reflection to call Sun.misc.cleaner’s Clean () method to actively free up memory held by Direct ByteBuffer.
  6. The memory capacity is insufficient. Upgrade the configuration.

Recommended tools & products

JVM memory analysis tool MAT

  • Eclipse Memory Analyzer

www.eclipse.org/mat

Ali Cloud APM product, support OOM abnormal keyword alarm

  • ARMS

Help.aliyun.com/document_de…

Arthas Java Diagnostics Tool

  • alibaba Arthas

Github.com/alibaba/art…

The author | wisdom elder brother

The original link

More technical dry goods please pay attention to code agriculture architecture technology number: code agriculture architecture – nuggets

This article is the original content of the code farm, and shall not be reproduced without permission.