Minicoredump god also!

After fixing the memory leak in our last article on atypical programmers, we had a good sleep. In the dream, suddenly the golden light, light into the tunnel of time, trance came to the Tang Shenlong years. Green sac at this time incarnation of Di Renjie senior assistant, accompanied by the eldest brother and Yuan Fang and Qianniu Wei came to the scene, a survey quickly locked the real murderer. Although the whole day fu write code at the desk, but early life tamed so cautious, good at observing the green sac also already understood the mystery of this case.

But, standing in the side of the Yuan Fang frown locked, face hard……

Di Gong stepped forward and asked: Yuan Fang, have a worry?

Yuan fang got up and said, “adults do as god, just like changan such cases frequently, each time the crime, will be deployed hundreds of thousand NiuWei to blockade the scene large area, visiting tens of thousands of people, human cost, also blocking traffic, affected the normal production order, and lead to damage to other business units, has been complaining.”

Di Gong smiled, finger to green sac, “sac ah, give you yuan Fang elder brother show.”

After a while of sucking up to the green sac, he took out a compass from the bag and said complacently: “I call sysAk as the ruler’s bag. This compass is called minicoredump. It will be clear how to blockade the scene after seeing it.” Yuan Fang took the compass, handy to play with a few, the above show to block the scene narrowed down a lot, the official way is also smooth many. Yuan Fang face worry to xi, can not help but ask: have this sharp weapon, lock the scene carefree! Just how it is done, but also ask adults to teach.

Di Gong laughed: “no hurry, and listen to me slowly.”

What is coredump?

Coredump is, as the name suggests, a coredump. In the process of running our program, if abnormal exit occurs, it is often difficult to locate the root cause of the problem by relying on the log of the program itself. The OPERATING system provides a CoredUMP mechanism to store the vMA information in core files when exceptions occur. Using this file, we can recover the information of the abnormal scene, from which the locator can obtain the variable value, stack information, memory data, the running position of the abnormal program (even record the code line number) and so on, and improve the efficiency of problem location. For example, the most critical step in solving a case is to obtain first-hand information, restore the scene of the crime, and deduce the case on this basis.

So what does coredump’s process look like?

As shown in the following figure, when a process is faulty, the kernel starts the Coredump mechanism to dump the vMA information at the fault site into a core file. After the failure, GDB is used to load the Coredump file to restore the fault site. You can quickly determine the fault type, locate the fault code, and locate the root cause by obtaining the variable values and call stack information on the fault site.

Just like Duke Di every time to solve a case, must first lock the discovery site, and then personally investigate, extract all the physical evidence and testimony, combined with the scene information deduction, restore the case process, and finally lock the criminal.

Coredump resource consumption problem

The operating system generates core files in kernel mode to dump all valid VMA information of the process. This means that coredump files for large processes can be very large, even in terabytes. The generation and saving of such large files severely consumes I/O, CPU, and memory bandwidth, which affects system stability. Just like before, to solve a major crime, it is necessary to block off traffic within a radius of several kilometers, and use hundreds of police officers to do a lot of canvassing work. But as technology has improved, information has become more accurate and tracking methods more sophisticated. Similarly, can CoredUMP information collection be refined as well?

Minicoredump appearance

“Yuanfang, locating problems is just like solving cases every day. The key to visiting is to collect high-value clues, such as stack space, data and BSS. Other anonymous page information, such as heap space, although the data in it is very important, it is not used in most cases for solving crimes, and belongs to low-clue information. However, in the large process OF VMA information, low cue information accounted for a very high proportion. Therefore, we should collect selectively and purposefully. At this point, it’s time for minicoredump to sift through the information. While the workload is optimized, the impact on normal production can be minimized.”

Cysac stood by, looking admiringly at Big Dick, silently adding Minicoredump to the sysAK. Minicoredump not only has memory leak detection tricks, network diagnostics, and now minicoredump, but also can effectively filter out anonymous page information, slimming core files.

Take a chestnut

As shown in the table below, this is a typical distribution of crime scene information. Minicoredump will be collected in a targeted manner: the red-marked areas need to be collected. With the help of GDB, we can replay the crime process (call stack) and extract key evidence (obtain the information of variables on the stack and global variables), while the green-marked areas can be collected when necessary without affecting our case solving.

103249: ./main

0000000000400000 4K r-x– main

0000000000600000 4K r—- main

0000000000601000 4K Rw — Main # Data section of executable file

0000000001FA5000 132K RW — [anon] # Heap space

00007f8188000000 10372K rw— [ anon ]

00007f8188a21000 55164K —– [ anon ]

00007f8190000000 10372K rw— [ anon ]

…

00007f819ca21000 55164K —– [ anon ]

00007f81a0000000 10372K rw— [ anon ]

00007f81a0a21000 55164K —– [ anon ]

00007f81a495d000 4K —– [ anon ]

00007F81A495E000 8192K Rw — [anon] # stack space

00007f81a6dfe000 10244K rw— [ anon ]

00007f81a77ff000 4K —– [ anon ]

00007f81a7800000 8192K rw— [ anon ]

00007f81a8000000 10372K rw— [ anon ]

00007f81a8a21000 55164K —– [ anon ]

00007f81ac15c000 4K —– [ anon ]

00007f81ac15d000 8192K rw— [ anon ]

00007f81ac95d000 4K —– [ anon ]

00007f81ac95e000 8192K rw— [ anon ]

…

00007f81ae160000 4K —– [ anon ]

00007f81ae161000 8192K rw— [ anon ]

F81ae961000 1808 k r – 00007 – x libc – 2.17. So

2044 k – 00007 f81aeb25000 libc – 2.17. So

16 k r – 00007 f81aed24000 libc – 2.17. So

00007F81aED28000 8K Rw — libC-2.17. so #so data segment

00007F81aED2a000 20K Rw — [anon] #so BSS segment

F81aed2f000 92 k r – 00007 – x libpthreads – 2.17. So

2044 k – 00007 f81aed46000 libpthreads – 2.17. So

R – 00007 f81aef45000 4 k libpthreads – 2.17. So

00007 f81aef46000 4 k rw libpthreads – 2.17. So

00007f81aef47000 16K rw— [ anon ]

F81aef4b000 136 k r – 00007 – x ld – 2.17. So

00007f81af159000 12K rw— [ anon ] #so link map

00007f81af169000 12K rw— [ anon ]

00007 f81af16c000 4 k r – ld – 2.17. So

00007 f81af16d000 4 k rw ld – 2.17. So

00007f81af16e000 4K rw— [ anon ]

00007fff7eafc000 132K rw— [ stack ]

00007fff7eb58000 8K r—- [ anon ]

00007fff7eb5a000 8K r-x– [ anon ]

ffffffffff600000 4K r-x– [ anon ] #syscall

total 532892K

Actual performance

Talk about real data: we picked a real environmental comparison. The same process, Coredump, reduced the file size from 3.5GB to 1.9GB.

After the coredump space is reduced, functions such as pushing and pushing are not affected:

In more complex production environments, the actual optimization can reach 80% or more, and business jitter is reduced by 30%. In other words, with the introduction of new technology, there are fewer blockades and traffic flows more smoothly.

Duke Di asked: Yuan Fang, this method is good?

Yuan Fang repeatedly nodded, TSK TSK praise: Minicoredump god also god also, Duke Di not, not ah!

Qingsac, a senior assistant, stood by, wiping the compass with her sleeve over and over again, and her heart was filled with joy. Half a year later, Di Gong and Yuan Fang line in the clean Dali Temple archives warehouse, the present is not stacked into a mountain of files. Benefited from the thinning of the new file, but also moved out of a world, tea said case, relaxed and happy.

Di Gong xi asked —

In the side of the green sac was happy, suddenly scouts sent a keep out of the newspaper, Di Gong opened to see, said: the official road just out of the pile big case, the impact is very big, we have to start immediately. Green sac a surprise, wake up from a dream, thinking: official way? Maybe there is a new network problem. To find out what happens next, listen next time. (after)

The original link

This article is the original content of Aliyun and shall not be reproduced without permission.