Have you ever been woken up by a frantic alarm in the early morning due to the high CPU usage of online applications……


Have you ever experienced: just arrived at the company, the bench has not covered the heat, but a pile of customer service complaints in the inbox, you need to check log positioning problems……


Have you ever experienced: off work bell is about to ring, but you have to work overtime, statistical application per second, per minute peak and other indicators, because the BOSS to take these indicators, in tomorrow’s technology conference external bragging……


Have you ever experienced: late at night when you are sleeping, the operation and maintenance on duty frantically CALL you, due to the online application memory problems……


I’m sure you’ve probably experienced all of the above. In fact, whether you have experienced it or not, you are here today. Because I will be combined with the previous experience summary, in the ape door open altar try to take out sunflower treasure dian display one or two.


Water dripping through a stone is not a day, Rome was not built in a day, experience is not a pit and contributed to, anti-wolf art, we first from the overall situation, take a look at this technology treasure book (can not see it doesn’t matter, feel very cow break off on the line).



This technology treasure book is really very hardcore, mainly divided into four moves, see moves dismantle moves, let us one by one to dismantle.


Tip 1: Online applications occupy too much CPU.


Open to recruit:

Run the top command to find the process PID with the highest CPU usage. Through the ps - ef | grep PID view the corresponding application, and see who is at work; The jstack-lPID >> pid. log Obtains the stack information of the process. Ps-mp pid-o THREAD,tid,time to obtain the highest CPU usage of the THREAD tid; usingprintf "%x\n"Tid Tid of the hexadecimal thread. Use grep tiD-a20 pid. log to determine what is wrong with the thread.Copy the code

After the most ~ :
Leg pain cure leg, syndrome differentiation treatment, appropriate medicine. Locate the code and adjust the code.



Tip 2: Online application memory overflow.


Open to recruit:

Use the top command to find the PID corresponding to the application. Use jmap-heap PID to verify that less memory is allocated. The jmap - histo: live PID | more analysis to find the memory consumption object; pay attention to take up how many G object 】 【 The ps - efL | | grep PID wc-lView the number of threads created by the process. By ll/proc/PID/task | wc-lYou can also view the number of threads created by the process. Using netstat apn | | grep PID wc-lView the number of process network connections.Copy the code

Most ~ after: leg pain leg, syndrome differentiation treatment, appropriate medicine.

A. If the memory allocation is really small, adjust the memory appropriately; B. The object is frequently created and not released to optimize the code; C. Constantly create threads or network connections to optimize code.Copy the code



Step 3: Troubleshoot business problems.



Open to recruit:

Run the tail -fn 200 log_file command to query online logs in real time. Find the keyWord of the log search, such as orderId, mobileId, reqId, etc. Run the grep keyWord log_file command to query the logs of the line where the keyWord resides. Run grep -c n keyWord log_file to match the n lines above and below the keyWord. Using grep keyWord log_file | wc-lThe number of rows that match the keyword.Copy the code

Last to last: Tail and grep are most used in log search based on actual log search scenarios.


Move 4: BOSS statistics problem.


Open to recruit:

Cat log_file is used to read the log file. The cut command is used to intercept the timestamp of the log. If according to second statistics intercept to second; If the statistics are intercepted in minutes, Uniq-c was used for statistical de-repetition. Sort-nr is used to carry out reverse order according to the value of the first column; Use head-1 to display only the first line.Copy the code

After: statistics problem solved, so statistics peak per second command how to write?

For example, logs:

1118 115856 066 - REQID0000000000188 ... .Copy the code

Command combination:

SecondPeak=`cat log_file|cut -d.-f1| cut -c 1-11|sort|uniq -c |sort -nr|head -1`Copy the code


Well, today’s sharing is nearing the end, I don’t know how much you get, understand and don’t understand, suggest you collect, in case of need; If you want to make more people progress, you might as well carry forward the spirit of sharing, move your fingers and resend it.