1. Find the problem:
Labor Day returned to find the production environment down.
2. Troubleshooting:
1. Log in Linux and use the command top to find that PID: 19693 has a high CPU and memory usage (map has been solved, here is used to make up the number) and has not decreased.
2. Run the JPS command to find that PID: 19693 serves the order
3. Using jmap-heap 19693, we found that the arrow positions all occupied 99.9% of memory.
4. The problem has been found, the order service memory is full, JVM frequent GC caused high CPU usage, resulting in server downtime
3. Problem solving:
1. Use jmap-histo :live 19693 to pull to the top to see which classes are occupying the most memory.
2. After finding the most occupied classes, it is found that there is a problem with the code of MQ processing messages. Each time a message is processed, an mqClient instance will be created. The question of why the mqClient instance was not collected by gc remains to be investigated.
A rough calculation of 400,000 messages over five days created 400,000 MQClients
