Linux learning: File IO (without buffers), atomic manipulation concepts

In this section, we will focus on IO without buffers. Atomic operations are very important for file sharing, so we will introduce some concepts related to atomic operations.

1: File descriptors For the kernel, all open files are referred to through file descriptors. When we open or create a file, we return a non-negative integer, which we use to open and create files. Also, there are three file descriptors associated with standard input (0: STDIN_FILENO), output (1: STDOUT_FILENO), and error (2: STDERR_FILENO) open by default when executing a process. Their macros are defined in the unistd.h file. The file descriptor upper limit is OPEN_MAX.

2: Introduction to the open function

Int open(const char* pathname, int flag); int open(const char* pathname, int flag, mode_t mode); Pathname: indicates the pathname of the open file. Flag: are options for opening files. Mode: Specifies the mode of mode only when we need to create a file. O_RDONLY: opens files only in read mode. O_WRONLY: opens a file in write mode only. O_EXCL: tests whether the file exists, and creates the file if it does not. (Note: An error occurs if O_CREAT is specified and the file already exists.) O_APPEND: Each write is appended to the end of the file. O_CREAT: Creates a file if it does not exist. When using this option, a third parameter is required. Mode: the permission bit, which is the bit of the various read and write permissions that we display using ls -L. S_IRWXU: The user has read/write execution permission. S_IRUSR: the user has read permission. S_IWUSR: the user has the write permission. S_IXUSR: the user has execution permission. S_IRWXG: The user in the same group has read/write execution permission. S_IRGRP: The user in the same group has read permission. S_IWGRP: The same user has the write permission. S_IXGRP: The user in the same group has the execution permission. S_IRWXO: Other users have read/write execution permission. S_IROTH: Other users have read permission. S_IWOTH: Other users have write permission. S_IXOTH: Other users have execution permission. O_TRUNC: If the file exists and we open it only for reads and writes, the file length will be truncated to 0. That is, the file length will be 0, which is equivalent to deleting the entire file and writing it again. O_NOBLOCK: specifies the file as non-blocking IO. Normally, when we write to a file, we wait for the write to complete and then return. If this flag is specified, it returns directly regardless of whether the write is complete. O_SYNC: The file properties are updated before each write operation. O_RSYNC: Waits for any pending write to the same part of the file to complete. (This logo and the following logo are not very understandable, if you do understand, please discuss in the comments section. It would be nice to have an actual example). O_DSYNC: Each write waits for the physical operation to complete. This flag affects file attributes only if they are grounded to reflect changes in file data. The file descriptor returned by the open function must be the smallest file descriptor currently available. Truncation of filename and pathname: This concept is briefly introduced. When we type the pathname of the open file, we cannot recognize the pathname that long, so we keep part of it. This will not identify the specific open file (we save this part of the pathname may be the same as the pathname of other files). We can get some pathname length here using pathconf, and I found that my machine has a length limit of 4096. Creat: create file function prototype: create (const char *pahtname, mode_t mode); This function and the open function are similar parts, do not understand. Close: close the open file descriptor (we open a file must use some resources to manage the open file, so close is to release the resources used by the stack) function prototype: int close(int filedes); The lseek function: changes the offset of the current file, that is, where we start reading and writing files. When we open a file or create a new file, the file offset defaults to 0. That is, read and write from the beginning of the file. We can also use the lseek function to locate the current file offset to the specified set of the file. Lseek (int filedes, off_t offset, int whence); lseek_64(int filedes, off64_t offset, int whence); Filedes: is the file descriptor. Offset: indicates the offset size. (can be positive < increase file offset > or negative < decrease file offset >) I will briefly explain whether off_t is sufficient. But when we look at the function prototype, there are both 32-bit and 64-bit file offsets. If it's 32 bits then the file offset is at its maximum 2 to the 32 minus one, so is 64 bits. 32-bit files have a maximum offset of 2TB (2^31-1 bytes) and 64-bit files have a maximum offset of 2^33TB, which should be sufficient. Different platforms use different sizes, and only 32 and 64 bits are listed here. The specific implementation depends on the hardware, here will not enumerate one by one. whence: is the designated location to start cheap. SEEK_SET: i.e. from the beginning of the file for only cheap. SEEK_CUR: Cheap from the current file location. SEEK_END: Cheap starting at the end of the file. If the file descriptor refers to a pipe or socket, then lseek returns -1. Read: reads a specified number of bytes from the file. Ssize_t read(int filedes, void *buf, size_t nbytes); Filedes: file descriptor. Buf: reads the buffer where the bytes are stored. Nbytes: Indicates the number of bytes read. Return value: the number of bytes actually read. The actual number of bytes read may be less than the number of bytes we requested to read. (eg, the current file location still has thirty bytes readable to the end of the file, but we want to read 200 bytes, which returns 30 bytes. But on the next read, 30 bytes are returned. And reading data from the network, sometimes less than the number of bytes we read due to network latency). Write function: writes a specified number of bytes of data to a file. Ssize_t write(int filedes, const void* buf, size_t nbytes); Again, buf is the write byte storage location. Nbytes is the number of bytes written. The return value is the number of bytes actually written. Also, if we specify O_APPEND, we set the file offset to the end of the file every time we write. File sharing: Before we get to file sharing, let's look at the structure the kernel uses to represent open files. The kernel uses three data structures to represent open files. (1) Each process has a record entry, which contains an open file descriptor table, and each file descriptor occupies one entry. We use an integer in the process to record the location of the file item. Each file descriptor is associated with two parts: the file flag and a pointer to the file table. (2) The kernel maintains a file table for open files. The file table contains the file status flag, the offset of the current file, and the V-node pointer. (3) Every open file has a V node. The V-dot contains the file type and Pointers to functions that operate on the file. I node information, the length of the current file. Just know about it. As shown in the figure below, the relationship among the three can be well understood.Copy the code

Also, when our two processes open a file at the same time, the structure looks like the following.Copy the code

It is understandable that when multiple processes simultaneously write data to a file, unexpected results may occur. If you want to avoid this problem, you need to understand the concepts associated with atomic operations. Atomic operation: Concept: the complete process of performing an operation independently of other influences. In the following program, we add content to the end of a file. lseek(fd, SEEK_END, 0); write(fd, buf, size); If it's a program, then we can write it this way. However, if there are two independent processes, this may be troublesome (for example, process A and process B, if process A needs to write 100 bytes and process B needs to write 1000 bytes). Then maybe process B starts writing when process A writes 50 bytes, and process B writes 1000 bytes. If process A writes, the first 50 bytes will be overwritten by process B, and the recorded information may be corrupted. As shown in the figure below, two files write to one file.Copy the code

Dup and dup2 functions: Copy file descriptors. Function prototype: int dup(int filedes); int dup2(int filedes, int filedes2); The previous function returns a new file descriptor that points to the same file entry that the Filedes file descriptor points to (the returned file descriptor must be the smallest file descriptor in the current process table entry). The second function uses the specified file descriptor to copy the Filedes file descriptor, if filedes2 is already in use, it closes filedes2 first and then copies Filedes. Refer to the following figure to understand the meaning of copying a file descriptor.Copy the code

As mentioned earlier, when we use the IO buffer to write data to disk, we may not be able to write data to disk in time. When our machines have an accident, we may lose data. Also, some databases need to be saved in time, so we need to make the write operation happen in time. Here are three functions that write data to disk in time. Function prototype: int fsync(int filedes); Writes specified file descriptor data to disk, in this case waiting for the return after the write operation. int sync(); Queue all modified buffers to write operations. int fdatasync(int filedes); Data is written to disk while file properties are updated. Some of the properties of the file we've seen before, but here's a function that can change some of the properties of the open file through the file descriptor. FCNTL (int filedes, int CMD...) ; (1) : Copy the existing descriptor. (2) : Get/set the file descriptor flag. (3) : Get/set the file status flag. (4) : Acquire/set asynchronous I/O ownership (asynchronous I/O is implemented via signals, as explained in the section on sockets). (5) : Refer to the MAN manual for obtaining/setting records, we will find many CMD as follows. F_DUPFD: Copies a new file descriptor. F_GETFD/F_SETFD: Gets and sets file descriptor tags. There is currently only one type of tag, FD_CLOEXEC (the file descriptor is closed when the exec function executes). F_GETFL/F_SETFL: Gets and sets the file status flag. The file status flag above can be changed except for O_DSYNC and O_SYNC. (eg: Read, write, append, etc.) WHAT I refer to here is my version of MAN manual. If you want to know more, please refer to your version of MAN manual. F_GETOWN/F_SETOWN: Obtains and sets the ownership of asynchronous I/OS. You'll learn about this in the Learning Sockets section. For record locks, use the structure shown below. F_SETLK/F_SETLKW/F_GETLK: Request or release a lock/set a file lock/return a lock (if there is no lock then set to F_UNLCK in the L_TYPE field). When learning about multiple processes, I will write an experiment about file locking, that is, multiple processes read and write files, using the file record lock to achieve file sharing. ! [file](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/b29f827e806c483bbf24b805ef9f7d73~tplv-k3u1fbpfcp-zoom-1.image)Copy the code