We are inHDFS – What is MetadataMentioned, the metadata will exist with memory and disk, memory in order to improve the response speed, disk is in order to ensure the security of data persistence, but write disk speed relative to the memory is slow for several orders of magnitude, if the NameNode every time the data on disk that is can’t deal with so many of the client’s request, So NameNode uses double caching and segmented locking.

Double buffering defines two memory blocks, BufCurrent for current metadata writing and BufReady for disk writing. Both blocks are 512K.

The first lock

Thread 1 picks up the lock, then checks to see if the cache is currently being swapped (as determined here by isautosyncScheduled), and if it isn’t, then picks up a globally unique transaction ID, which is incremental. Once you have the transaction ID, start writing to BufCurrent. Once you have written to it, determine if BufCurrent exceeds 512K. If not, thread 1 will finish the process (the background is white at the end).

Since thread 2,3,4 will wait until thread 1 releases the lock, if thread 2,3,4 acquires the lock, it will continue the operation of thread 1.

Since the operations in this lock are memory based, the speed can be very lumpy.



Let’s assume that after thread 4 has finished writing and finds that the BufCurrent memory exceeds 512K, isAutoSyncScheduled is set to true, indicating that memory is being switched and that no other thread can do so. BufCurrent here is the data written by thread 1,2,3,4.



At this point, thread 5 comes in and takes the lock. We color the process that holds the lock darker and notice that it is about to start swapping memory.

The second lock

Let’s assume that thread 4 acquires the lock next. He found that no other threads were swapping memory at this point (according to ISSyncrunning), so he began swapping BufCurrent and BufReady memory, and after that, BufCurrent’s data was cleared. In this case, isSyncrunning is set to true, isAutoSyncScheduled is set to false, and the wait thread is finally awakened.



After Thread 4 wakes up the other threads, it starts to write the data of bufReady to disk. This operation is time-consuming, so it is not locked, but it is running, so the image below is marked in green.

Here, bufReady is the data that threads 1,2,3,4 write.

At this point, thread 5 acquires the lock, realizes the memory swapping is complete, and starts writing data to BufCurrent.



Then thread 6 gets the lock, and when the BufCurrent passes 512K again, isAutoSyncScheduled is set to true. This means that memory is being swapped, and no other thread can write to the BufCurrent. But he found that isSyncrunning was true, indicating that other threads were writing disks, so he waited.

The third lock

At this point, thread 4 finishes writing the disk, then it acquires the lock, changes the SYNCTXID to its own transaction ID, and assigns isSyncrunning to false, indicating that the disk write is complete. Finally, wake up the other wait threads.



At this point, Thread 6 regained the lock, and he realized isSyncrunning was false, which means that other threads had already scrubbed their contents to disk, so Thread 6 started doing what Thread 4 had done, swapping memory and writing to disk.

conclusion

The first lock, which determines the value of isautosyncScheduled and the assignment of isautosyncScheduled, indicates that BufCurrent and BufReady are exchanging memory. The second lock is mainly for judging ISSyncrunning and assigning values of ISSyncrunning and ISautosyncScheduled. ISSyncrunning is used to determine whether a disk is being written, ISautosyncScheduled is used to determine whether memory is being swapped, and if it is swapping, it cannot write to BufCurrent, and if it is writing to disk, it cannot write to disk. The third lock, Assign ISSyncrunning, indicates that the disk write is complete. During this period, the most time-consuming operation is not locked, other memory operations are locked, but the speed is relatively fast, using the way of locking in this segment and double buffering mechanism, greatly improving the performance.