“This is the second day of my participation in the August More Text Challenge. For details, see: August More Text Challenge.”
Preparation knowledge (Suggestions)
- Familiar with Java programming language
- Familiar with network communication protocol
- Familiar with C language
- Familiar with Linux operating system
- Familiar with Unix environment programming
- Familiar with network packet capture and interception analysis
Introduction to NFSV4 File locking
File lock is one of the most basic features of a file system. With the help of file lock, an application can control the concurrent access to files by other applications. NFS, the standard network file system for UniX-like systems, has natively supported file locking over the course of its development (starting with NFSv4).
Through the commandman nfs
You can viewOfficial description of manual contents:
The important one is the third sentence: NLM only supports recommended file locking. To lock an NFS file, use the FCNTL (2) and F_GETLK and F_SETLK commands. The NFS client converts the file lock obtained through Flock (2) to an advisory lock. This means that an application can manage NFS file locks through the FCNTL () or flock() system calls.
The following is the call process diagram of Ali Cloud file storage NAS using NFSv4 to obtain file locks: Photo credit:# NFS file lock consistency
It is easy to see from the call stack above that the implementation logic of NFS file lock basically overuses the design and data structure of the VFS layer. After successfully obtaining the file lock from the Server through RPC, call locks_lock_inode_wait() function to obtain the file lock and hand it over to the MANAGEMENT of the VFS layer. There is plenty of information about the design of file locks in the VFS layer, so you can search for it yourself
Looking at the file lock invocation above, you can see that the primary interaction is with the nfs4_proc_lock() method, By looking at the Linux kernel source https://git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/tree/fs/nfs/nfs4proc.c can see related implementation calls
In addition to the Linux kernel source code above, you can also check out the Network File System (NFS) Version 4 Protocol specification, and read chapter 9 to see how NFSv4 supports File locking
NFSV4 implements the server file lock function by adding the stateID to the communication protocol. NFSv3 does not have the concept of stateID. Therefore, it is not possible to determine whether the client application process that sends the READ or WRITE operation also obtains the proper byte range lock on the file. Therefore, there is no way to enforce locking. This barrier has been removed using the StateID construct.
Introduction to Java file locking
Java provides the FileLock class, which can control the concurrent access of different programs (JVMS) to the same file, and realize the inter-process file synchronization operation.
FileLock is a Java 1.4 class that locks a writable file (W) so that only one process can access the file at the same time. Other processes that can’t get the lock choose to either wait or do something else, which ensures that the file is accessed sequentially by all processes. It is also possible to take advantage of this nature of file locking. In some scenarios, although we do not need to operate on a file, we can also use FileLock to control the concurrency, ensure that the process is executed sequentially, and avoid data errors.
The concept of the lock
- Shared lock: Read operations are shared, but only one write is allowed (read operations can be simultaneous, but write operations cannot be simultaneous). Shared locks prevent other running programs from acquiring duplicate exclusive locks, but allow them to acquire duplicate shared locks.
- Exclusive lock: Only one read or one write (neither read nor write at the same time). Exclusive locks prevent other programs from acquiring locks of any type.
FileLock low-level implementation
We need to get the FileLock lock by calling lock() or tryLock() on the FileChannel class,
FileChannel.java
Here is chosentrylock()
, actually chooselock()
Again, since the method is abstract, you need to look at the concrete logic in the implementation class Impl.
FileChannelImpl.java
Notice the part of the try statement. Normally a try does something important. You can see that the lock operation is taking place.
FileDispatcher.java
lock()
It’s also an abstract method here, so keep looking for its implementation class Impl.
FileDispatcherImpl.java
Finally, we find the implementation of Lock (), discover that the final call is lock0(), and drill down to find a native method.
FileDispatcherImpl.c
seenative method
Must be implementation logic written in C, so findFileDispatcherImpl.c
Found a call in its implementationfcntl()
The use of nfSv4 file locks also requires this function to be called.
Actual demonstration of NFS file lock
You’ve already learned how to install and configure NFS. You’ve also learned about NFSv4 file locks and Java file locks.
-
First create a new Java class that will read and write the file and lock it as follows:
package com.jpsite.utils; import cn.hutool.core.io.FileUtil; import cn.hutool.core.io.file.FileAppender; import cn.hutool.core.io.file.FileReader; import java.io.File; import java.io.IOException; import java.io.RandomAccessFile; import java.net.InetAddress; import java.nio.channels.FileChannel; import java.nio.channels.FileLock; import java.util.Objects; public class FileUtilTest { public static void test1(a) throws Exception { InetAddress ia = InetAddress.getLocalHost(); String host = ia.getHostName();// Get the computer host name String IP= ia.getHostAddress();// Get the computer IP final File file = FileUtil.touch(new File("/data/share/b.txt")); FileAppender appender = new FileAppender(file, 1.true); appender.append("123"+IP); appender.append("abc"+IP); appender.append("xyz"+IP); appender.flush(); FileReader fileReader = new FileReader(file); String result = fileReader.readString(); System.out.println(result); final FileLock fileLock = getFileLock(file); if (Objects.isNull(fileLock)) { System.out.println("not get lock"); } else { final String lockType = fileLock.isShared() ? "share lock" : "exclusive lock"; System.out.println(String.format("getted lock %s", lockType)); int i = 0; while (true) { Thread.sleep(1000); i++; System.out.println(lockType + "no release"); if (i == 30) { fileLock.release(); if(! fileLock.isValid()) { System.out.println("lock is valid ,return while"); break; } } } } } public static FileLock getFileLock(File file) throws IOException { final RandomAccessFile rw = new RandomAccessFile(file, "rw"); final FileChannel channel = rw.getChannel(); return channel.tryLock(); } public static void main(String[] args) throws InterruptedException { new Thread(() -> { try { test1(); } catch(Exception e) { e.printStackTrace(); } }).start(); }}Copy the code
-
Upload the class to the Linux machine of the NFS Client, such as centos 1, centos2, CD to the SRC path of the Java project, and execute the class separately
# for an explanation of command can check my boiling point javac classpath/root /. M2 / repository/cn/hutool/hutool - all / 4.6.17 / hutool - all - 4.6.17. Jar ./com/jpsite/utils/FileUtilTest.java java -classpath . : / root/m2 / repository/cn/hutool/hutool - all / 4.6.17 / hutool - all - 4.6.17. Jar com. Jpsite. Utils. FileUtilTestCopy the code
You can see that the Java code in CentoS1 is executing normally
Centos2 exited because the file lock could not be obtained. Procedure
M1 macbook test problem
The previous test was on two cloud host centos machines, now change to one centos and one local computer, first mount the local computer by executing the command
Sudo mount -t NFS -o vers = 4.0, proto = TCP, timeo = 60 129. XXX. X. 139: / data/share/Users/hahas/Documents/data/shareCopy the code
The test produced two results:
- ✅ Result 1: The Java code in centos is executed first, and then the Java code in M1 macbook is executed. The result is consistent with the test results of the two cloud host centos machines.
- ❌ Result 2: First execute the Java code of THE M1 macbook, then execute the Java code of the centos, and find that the file lock can be obtained.
Cause 2: The NFS client version of THE M1 macbook is too early to provide effective support. Or because of the use of arm architecture core, resulting in the failure of related functions.
Network protocol data analysis
There are many methods and tools for network packet capture. In this article, the centos uses the built-in tcpdump, while the macbook uses wireshark
The Wireshark allows you to view data related to network protocols
// View the network data sent to the 129.XXX.X.139 NFS server
(ip.dst eq 129.xxx.x139.) and nfs
Copy the code
The lock and unlock operations are used to capture packets. The content of network packets in the lock and unlock operations is consistent with that in the chapter on NFSV4 file Locking. Therefore, data fields such as the STATEID in network protocols are required for NFS file locks to be used.
Ps: If there is a powerful partner or hacker, you can use the method of packet interception and tampering directly to lock the FILES of NFS server 🔒 👍 niubius🐂 plus without using the system low-level method call
NFS Client mount configuration
local_lock
Linux NFS clients provide a way to set the lock local. This means that an application can lock a file, but this lock provides exclusions only for other applications running on the same client. Remote applications are not affected by these locks.
The mount -t NFS -o vers = 4.1, proto = TCP, local_lock = flock 129. XXX. X. 139: / data/share/data/shareCopy the code
cto
NFS implements a weak data consistency called “near open consistency” or CTO. This means that when a file is closed on the client, all modified data associated with the file is flushed from the server. If your NFS client allows this behavior, make sure the CTO option is set.
The mount -t NFS -o vers = 4.1, proto = TCP, the cto 129. XXX. X. 139: / data/share/data/shareCopy the code
ac / noac
To improve performance, NFS clients cache file attributes. Every few seconds, the NFS client checks for updates to the properties of each file in the server version. Changes that occur on those small servers remain undetected at intervals until the client checks the server again. The NOAC option prevents clients from caching file properties so that applications can detect file changes on the server more quickly. In addition to preventing clients from caching file properties, the NOAC option also forces application writes to stay synchronized so that local changes to files are immediately visible on the server. In this way, other clients can quickly detect recent write operations when checking file properties. Using the NOAC option provides greater cache consistency between NFS clients accessing the same files, but it extracts a significant performance penalty. Therefore, we encourage the judicious use of file locking.
The DATA AND METADATA COHERENCE section Attribute Caching discusses these tradeoffs in detail.
Use the NOAC mount option to achieve property cache consistency across multiple clients. Almost every file system operation checks file property information. Clients cache this information for a period of time to reduce network and server load. When NOAC takes effect, file property caching on the client is disabled, so the properties of each file that needs to be checked are forcibly returned to the server. This allows the client to very quickly see the cost of many additional network operations for changes to the file. Be careful not to confuse the NOAC option with “no data cache”. The NOAC mount option prevents clients from caching file metadata, but there are still competitions that can cause data caches to be inconsistent between the client and the server. The NFS protocol is not designed to support true clustered file system cache consistency unless some type of application implements it. If absolute cache consistency is required between clients, the application should use file locking. Alternatively, applications can use the O_DIRECT flag to open files and disable data caching entirely.
Relevant reference
Analysis of NFS file lock consistency design principles
FCNTL (2) and Flock (2)
Using NFSv4 as a shared file system
Network File System (NFS) Version 4 Protocol
Analysis of NFS read/write block size
The Linux kernel source
Follow + like 👍 collect ❤️ don’t get lost
Article weekly continuous update, wechat search “ten minutes to learn programming” the first time to read and urge, if this article is well written, feel something if you support and recognition, is the biggest motivation for my creation, we will see the next article!