A: background
1. Tell a story
Last week, a friend of mine came to me, saying that his program CPU and handle are increasing, there is no turning back trend. After several days of checking, there is no progress, Tejia WX asked for help, the screenshots are as follows:
I plan to use a separate article to investigate and read the CPU explosion problem. This article will first talk about handle leakage. After all, I have written more than 20 articles, and this is the first time to talk about handle leakage.
2. What is a handle
A managed layer holds a reference to an unmanaged resource. With this reference, we can forcibly reclaim an unmanaged resource. What is an unmanaged resource? My personal understanding is that the places that gc can’t handle are unmanaged resources.
Usually include this type of handle class: FileStream, Socket, etc., if you have this front base, then you can use WinDBG to analyze!
Two: WinDBG analysis
1. Look at symptoms
Handle =8770 =8770 =8770 =8770 =8770 =8770 Before saying this, we have not encountered this phenomenon, is no matter how the program leaks, as long as we exit the EXE, then all the resources will be magically released, whether it is managed resources or unmanaged resources, so say I believe there are very curious friends this is how to achieve?? You can think about it for 10 seconds.
Reveal the answer! In short, the CLR maintains a handle table internally. When the program is closed, the CLR forces the release of all handles in the handle table. Then the problem is simple. Gchandles command.
2. View the handle table
Here is a reminder! Gchandles is scoped to AppDomain, not Process, so let’s look at the command output:
0:000> !gchandles -stat
Statistics:
MT Count TotalSize Class Name
...
00007ffccc1d2360 3 262280 System.Byte[]
00007ffccc116610 72 313224 System.Object[]
00007ffccc3814a0 8246 593712 System.Threading.OverlappedData
Total 10738 objects
Handles:
Strong Handles: 312
Pinned Handles: 18
Async Pinned Handles: 8246
Ref Count Handles: 1
Weak Long Handles: 2080
Weak Short Handles: 59
Dependent Handles: 22
Copy the codeLook from the output, there are special dazzling, a set of data that is: Async Pinned Handles = 8246 [System. The Threading. OverlappedData], what is the meaning of this? This is an OverlappedData handle associated with asynchronous IO. If an ASYNCHRONOUS IO is pinned, one byte[] is pinned, and the OverlappedData context object of the asynchronous IO is overlapped.
The next question is: since it is asynchronous IO, what type of handle is it, FileStream or Socket? To find out, we need to dig deep into the OverlappedData object with the associated command:! dumpheap -mt xxx & ! do … , please refer to the following:
0:000> !DumpHeap /d -mt 00007ffccc3814a0
Address MT Size
000001aa2acb39c8 00007ffccc3814a0 72
000001aa2acb3fd8 00007ffccc3814a0 72
000001aa2ad323d0 00007ffccc3814a0 72.0:000>!do 000001aa2acb39c8
Name: System.Threading.OverlappedData
MethodTable: 00007ffccc3814a0
EEClass: 00007ffccc37ca18
Size: 72(0x48) bytes
File: C:\xxx\xxx\vms_210819\System.Private.CoreLib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffccc21f508 40006b2 8 System.IAsyncResult 0 instance 0000000000000000 _asyncResult
00007ffccc110ae8 40006b3 10 System.Object 0 instance 000001aa2acb4020 _callback
00007ffccc381150 40006b4 18. eading.Overlapped0 instance 000001aa2acb3980 _overlapped
00007ffccc110ae8 40006b5 20 System.Object 0 instance 000001aa2acb9fe8 _userObject
00007ffccc11f130 40006b6 28 PTR 0 instance 000001aa2a9bd830 _pNativeOverlapped
00007ffccc11ecc0 40006b7 30 System.IntPtr 1 instance 0000000000000000 _eventHandle
0:000> !DumpObj /d 000001aa2acb3980
Name: System.Threading.ThreadPoolBoundHandleOverlapped
MethodTable: 00007ffccc3812a0
EEClass: 00007ffccc37c9a0
Size: 72(0x48) bytes
File: C:\xxx\xxx\vms_210819\System.Private.CoreLib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffccc3814a0 40006ba 8. ng.OverlappedData0 instance 000001aa2acb39c8 _overlappedData
00007ffccc34fcd0 40006a4 10. ompletionCallback0 instance 000001aa2acb3920 _userCallback
00007ffccc110ae8 40006a5 18 System.Object 0 instance 000001aa2acb38c8 _userState
00007ffccc380120 40006a6 20. locatedOverlapped0 instance 000001aa2acb3960 _preAllocated
00007ffccc11f130 40006a7 30 PTR 0 instance 000001aa2a9bd830 _nativeOverlapped
00007ffccc380eb8 40006a8 28. adPoolBoundHandle0 instance 000001aa2acb3900 _boundHandle
00007ffccc1171c8 40006a9 38 System.Boolean 1 instance 0 _completed
00007ffccc34fcd0 40006a3 458. ompletionCallback0 static 000001aa2acb4020 s_completionCallback
0:000> !DumpObj /d 000001aa2acb3900
Name: System.Threading.ThreadPoolBoundHandle
MethodTable: 00007ffccc380eb8
EEClass: 00007ffccc37c870
Size: 32(0x20) bytes
File: C:\xxx\xxx\vms_210819\System.Private.CoreLib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffccc1d76b0 40006a1 8. rvices.SafeHandle0 instance 000001aa2acb1d30 _handle
00007ffccc1171c8 40006a2 10 System.Boolean 1 instance 0 _isDisposed
0:000> !DumpObj /d 000001aa2acb1d30
Name: Microsoft.Win32.SafeHandles.SafeFileHandle
MethodTable: 00007ffccc3807c8
EEClass: 00007ffccc37c548
Size: 48(0x30) bytes
File: C:\xxx\xxx\xxx\System.Private.CoreLib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffccc11ecc0 4000bb4 8 System.IntPtr 1 instance 0000000000000428 handle
00007ffccc11b1e8 4000bb5 10 System.Int32 1 instance 4 _state
00007ffccc1171c8 4000bb6 14 System.Boolean 1 instance 1 _ownsHandle
00007ffccc1171c8 4000bb7 15 System.Boolean 1 instance 1 _fullyInitialized
00007ffccc2f1ae0 4001c3d 20. Private.CoreLib]]1 instance 000001aa2acb1d50 _isAsync
00007ffccc380eb8 4001c3e 18. adPoolBoundHandle0 instance 0000000000000000 <ThreadPoolBinding>k__BackingField
Copy the codeThe handle value 0000000000000428 on the fifth row from the bottom is the specified handle value, which can be used next. The handle command displays detailed information about its value.
0:000> !handle 0000000000000428 7
Handle 428
Type File
Attributes 0
GrantedAccess 0x100081:
Synch
Read/List,ReadAttr
HandleCount 2
PointerCount 65489
Copy the codeType: File Type: File Type: File
😪😪😪, although I dug up some information, this information is not enough for me to find the root cause of the problem. In terms of reference chain, these objects in Gchandles are at the top of the reference chain. In other words, I need to find some data objects downstream of the reference chain. A good entry point is to dig into the heap.
3. Find the descendants of managed heap
First we use! Dumpheap-stat take a look at the managed heap.
0:000> !dumpheap -stat
Statistics:
MT Count TotalSize Class Name
...
00007ffccc3c5e18 939360 52604160 System.Collections.Generic.SortedSet`1+Node[[System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]], System.Private.CoreLib]]
00007ffccc1d2360 16492 69081162 System.Byte[]
000001aa2a99af00 10365 76689384 Free
00007ffccc1d1e18 1904987 116290870 System.String
Copy the codeByte[] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] : [] From the output of the script I pulled several addresses to look at! Gcroot, it’s probably something like this.
0:000> !gcroot 000001aa47a0c030
HandleTable:
000001AA4469C090 (async pinned handle)
-> 000001AA491EB908 System.Threading.OverlappedData
-> 000001AA491EB8C0 System.Threading.ThreadPoolBoundHandleOverlapped
-> 000001AA491EB860 System.Threading.IOCompletionCallback
-> 000001AA491EAF30 System.IO.FileSystemWatcher
-> 000001AA491EB458 System.IO.FileSystemEventHandler
...
-> 000001AA47A0C030 System.String
0:000> !gcroot 000001aa2d3ea480
HandleTable:
000001AA28FE9930 (async pinned handle)
-> 000001AA2DD68220 System.Threading.OverlappedData
-> 000001AA2DD681D8 System.Threading.ThreadPoolBoundHandleOverlapped
-> 000001AA2DD68178 System.Threading.IOCompletionCallback
-> 000001AA2DD67848 System.IO.FileSystemWatcher
...
-> 000001AA2D3EA480 System.String
Copy the codeFrom the point of the entire chain of references, there is a System. IO. FileSystemWatcher, this and the previous analysis handle = File is consistent, and then export the string, found that most of them are associated with appSettings, as shown below:
string: appSettings:RabbitMQLogQueue
string: appSettings:MedicalMediaServerIP
string: appSettings:UseHttps
...
Copy the codeAnd then use! The strings command performs a fuzzy match and finds that such strings are up to 61W…
Appsettings is being watched, but there is a problem with the way it is done…
4. Look for the final answer
After giving the survey results to my friend, let my friend focus on whether there is a problem with the way of watching appsetting? A few hours later, my friend was finally found.
Appsetings are already monitored by setting reloadOnChange=true, but the writer is not familiar with this area and polling the AppSetings every 10 seconds.
Three:
In fact, the main reason for this accident is that I am not familiar with how to sense the latest data in appSettings in real time. I used netcore’s own reloadOnChange monitoring, and also used polling method for data perception. Therefore, the foundation is very important, do not take it for granted to write! 😁 😁 😁
For more high-quality dry goods: See my GitHub:dotnetfly