The Master said, “To recite three hundred poems and teach them to govern is not enough; Make in the quartet, not specifically on; I don’t think so, though.” The Analects of Confucius: Sub-Chapter

A hundred blog series. This is:

V69. Xx HongMeng kernel source code analysis (file handle) | why do you call a handle?

File system related sections are as follows:

  • V62. Xx HongMeng kernel source code analysis concept (file) | why everything is the file
  • V63. Xx HongMeng kernel source code analysis (file system) | said in books management file system
  • V64. Xx HongMeng kernel source code analysis (inode) | who is the most important concept of file system
  • V65. Xx HongMeng kernel source code analysis (mount directory) | why need to mount the file system
  • V66. Xx HongMeng kernel source code analysis (the root file system) | on first/File system on
  • V67. Xx HongMeng kernel source code analysis (character device) | bytes read/write device for the unit
  • V68. Xx HongMeng kernel source code analysis file system (VFS) | the foundation of the harmonious coexistence
  • V69. Xx HongMeng kernel source code analysis (file handle) | why do you call a handle?
  • V70. Xx HongMeng kernel source code analysis (pipe file) | how to reduce the data flow cost

Handle | handle

int open(const char* pathname,int flags);
ssize_t read(int fd, void *buf, size_t count);
ssize_t write(int fd, const void *buf, size_t count);
int close(int fd);
Copy the code

These functions are familiar to anyone who has written application code and manipulated files. The key steps of file manipulation are the same as filling an elephant in a refrigerator. You have to open the fridge, then put the elephant in, then close the fridge. One of the most important parameters is fd, on which all application operations on a file are based. Fd can be called a file descriptor, or a file handle, as I prefer to call it. Just like opening a door, you can open the door only by holding the handle. The handle is the handle for entering and closing the door. Mapped to the file system, the FD is the application layer’s handle to the kernel layer. Handle is a number, the open | creat to apply for the number, the kernel will create files related to a series of objects, return number, later via code can manipulate these objects. The principle is so simple, this article will start from FD, tracking the entire process of file operation.

Keep in mind that there are two types of file handles at different levels in the Kernel:

  • System file handle (sysfd), managed by the kernel, and the process file handle to form a mapping relationship, asysfdCan be multipleprofdMapping, which means opening a file takes up only onesysfd, but can occupy more than oneprofdThat is, a file is opened by multiple processes.
  • Process file handle (profd), which are managed by the process called process file handlesfdThat is, a process can access only those of the same processfdTo illustrate the relationship between: 
    File sysfd profd eat A peach. Mp4 10 13(A process) eat A peach. Mp4 10 3(B process) Allow the mother to be wrong.txt 12 3(A process) allow the mother to be wrong.txt 12 3(C process)Copy the code

Process file handle

By default, a process can have a maximum of 256 FDS, i.e., a maximum of 256 files can be opened. Files_struct. Files_struct can be understood as the file manager of the process, which only puts files related to the process, and the thread shares these files. In addition, the child process also copies a copy of the parent process files_struct into its own files_struct. In the parent-child process chapter, the essence of fork is to copy resources, including file contents.

Typedef struct ProcessCB {//... #ifdef LOSCFG_FS_VFS struct files_struct *files; /**< Files held by the process */ / every process has its own file manager, which records operations on Files. Note: a file can be manipulated by multiple processes} LosProcessCB; Struct files_struct {// files_struct; Struct fd_table_s * FDT; // Unsigned int file_lock; // File mutex unsigned int next_fd; // next fd #ifdef VFS_USING_WORKDIR spinlock_t workdir_lock; // Workspace directory spinlock char workdir[PATH_MAX]; // Workspace path, maximum 256 characters #endif};Copy the code

Fd_table_s is a member of files_struct, responsible for recording all process file handle information, PERSONALLY feel hongmong this implementation is a bit messy, not packaged well.

Struct fd_table_s {// process fd unsigned int max_fds; Struct file_table_s *ft_fds; /* process fd array associate with system fd *// Fd_set *proc_fds; 0,1,2 (stdin,stdout,stderr) fd_set *cloexec_fds; sem_t ft_sem; /* Manage access to file table */ / manage access to file table semaphores};Copy the code

File_table_s records the binding or mapping between process fd and system FD

Struct file_table_s {// struct file_table_s {// struct file_table_s; /* system fd associate with the tg_filelist index */ };Copy the code

Fd_set implements bitmap management of process fd. The series operations are fd_set,FD_ISSET,FD_CLR,FD_ZERO divided by 8 because char has 8 bits. Try to understand the implementation of bitwise operations.

typedef struct fd_set
{
  unsigned char fd_bits [(FD_SETSIZE+7)/8];
} fd_set;
#define FD_SET(n, p)  FDSETSAFESET(n, (p)->fd_bits[((n)-LWIP_SOCKET_OFFSET)/8] = (u8_t)((p)->fd_bits[((n)-LWIP_SOCKET_OFFSET)/8] |  (1 << (((n)-LWIP_SOCKET_OFFSET) & 7))))
#define FD_CLR(n, p)  FDSETSAFESET(n, (p)->fd_bits[((n)-LWIP_SOCKET_OFFSET)/8] = (u8_t)((p)->fd_bits[((n)-LWIP_SOCKET_OFFSET)/8] & ~(1 << (((n)-LWIP_SOCKET_OFFSET) & 7))))
#define FD_ISSET(n,p) FDSETSAFEGET(n, (p)->fd_bits[((n)-LWIP_SOCKET_OFFSET)/8] &   (1 << (((n)-LWIP_SOCKET_OFFSET) & 7)))
#define FD_ZERO(p)    memset((void*)(p), 0, sizeof(*(p)))
Copy the code

Vfs_procfd. c implements the file for the process file handle. The 0,1, and 2 fd of each process are occupied by the system and not allocated.

  • STDIN_FILENO(fd = 0) Standard input receives input from the keyboard
  • STDOUT_FILENO(fd = 1) Standard output output to the screen
  • STDERR_FILENO(fd = 2) Standard error output to the screen
/* minFd should be a positive number,and 0,1,2 had been distributed to stdin,stdout,stderr */ if (minFd < MIN_START_FD) { minFd = MIN_START_FD; Static int AssignProcessFd(const struct fd_table_s * FDT, int minFd) {if (FDT == NULL) {return VFS_ERROR;} static int AssignProcessFd(const struct fd_table_s * FDT, int minFd) {return VFS_ERROR; } if (minFd >= fdt->max_fds) { set_errno(EINVAL); return VFS_ERROR; } /* search for unused fd from table */ for (int I = minFd; i < fdt->max_fds; i++) { if (! FD_ISSET(i, fdt->proc_fds)) { return i; } } set_errno(EMFILE); return VFS_ERROR; Void FreeProcessFd(int procFd) {struct fd_table_s * FDT = GetFdTable(); if (! IsValidProcessFd(fdt, procFd)) { return; } FileTableLock(fdt); FD_CLR(procFd, fdt->proc_fds); FD_CLR(procFd, FDT ->cloexec_fds); fdt->ft_fds[procFd].sysFd = -1; // Unbind system file descriptor FileTableUnLock(FDT); }Copy the code
  • The allocation and release algorithms are simple and are performed by bitmap operations.
  • fdt->ft_fds[i].sysFdIn theiProcess representativefd.- 1Is not bound to a system file handle.
  • The meaning and relationship between process file handles and system file handles are described in VFS.

System file handle

The implementation of system file handle is similar, but it is not in the cloud kernel project, but in the NuttX project fs_files.c, because the cloud kernel project uses other third-party projects, so we need to add together to see the full implementation of the cloud kernel. The subsystem warehouses involved are as follows:

  • Subsystem annotation repository

    In the process of annotating the source code of the masked kernel, we found that only annotating the kernel repository is not enough, because it is associated with other subsystems, if you do not understand these subsystems is difficult to annotate the masked kernel completely, so we also annotated some of these associated repositories, including:

    • Compile build subsystem | build_lite
    • Protocol stack | lwip
    • The file system | NuttX
    • The standard library | musl

  • System file handles are also managed by bitmaps. The detailed operations are as follows

SYS FD static unsigned int bitmap[CONFIG_NFILE_DESCRIPTORS / 32 + 1] = {0}; Static void set_bit(int I, void *addr) {unsigned int tem = (unsigned int) I >> 5; /* Get the bitmap subscript */ unsigned int *addri = (unsigned int *)addr + tem; unsigned int old = *addri; old = old | (1UL << ((unsigned int)i & 0x1f)); /* set the new map bit */ *addri = old; Bool get_bit(int I) {unsigned int *p = NULL;} // get the specified bit to see if it is allocated bool get_bit(int I) {unsigned int *p = NULL; unsigned int mask; p = ((unsigned int *)bitmap) + (i >> 5); /* Gets the location in the bitmap */ mask = 1 << (i & 0x1f); /* Gets the mask for the current bit int bitmap */ if (! (~(*p) & mask)){ return true; } return false; }Copy the code
  • tg_filelistIs the global system file list, unified management systemfdAnd the key structure isfileThis is the entity that the kernel describes as a file object, and is the most important part of this article. 
    #if CONFIG_NFILE_DESCRIPTORS > 0 struct filelist tg_filelist; #endif struct filelist {sem_t fl_sem; /* Manage access to the file list */ struct file fl_files[CONFIG_NFILE_DESCRIPTORS]; }; struct file { unsigned int f_magicnum; /* file magic number */ int f_oflags; /* Open mode flags */ struct Vnode *f_vnode; /* Driver interface */ loff_t f_pos; /* File position */ unsigned long f_refcount; /* reference count */ char *f_path; /* File fullpath */ void *f_priv; /* Per file driver private data */ const char *f_relpath; /* realpath */ struct page_mapping *f_mapping; /* mapping file to memory */ void *f_dir; /* DIR struct for iterate the directory if open a directory */ const struct file_operations_vfs *ops; int fd; };Copy the code
    • f_magicnumMagic numbers, magic numbers are different for each file format,gifis47 to 49 46 38.pngis89 50 4e 47
    • f_oflagsPermission mode for manipulating files, read/write/execute
    • f_vnodeThe correspondingvnode
    • f_posRecords the current location of the action file
    • f_refcountThe number of times the file was referenced, that is, the number of times the file was opened by all processes.
    • f_privPrivate data for a file
    • f_relpathRecord the actual path of the file
    • f_mappingThe mapping between record files and memory is described in detail in the file Mapping section.
    • opsFunction that operates on the contents of a file
    • fdFile handle number. System file handles are unique until the application is completef_refcountWhen 0, the kernel is reclaimedfd.

The open | creat | application file handle

Obtain the file handle through the file pathname pathname. The hongmeng implementation process is as follows

Fp_open //vnode layer files_allocate filep->ops->open(filep) AssociateSystemFd // Bind the system file handleCopy the code

Create a file object where I is the allocated system file handle.

Int files_allocate(struct Vnode *vnode_ptr, int oflags, off_t pos, void *priv, int minfd) allocate(struct Vnode *vnode_ptr, int oflags, off_t pos, void *priv, int minfd) While (I < CONFIG_NFILE_DESCRIPTORS)// System descriptor {p = (((unsigned int *)bitmap) + (I >> 5); /* Gets the location in the bitmap */ mask = 1 << (i & 0x1f); /* Gets the mask for the current bit int bitmap */ if ((~(*p) & mask))// this bit can be allocated {set_bit(I, bitmap); List ->fl_files[I]. F_oflags = oflags; list->fl_files[i].f_pos = pos; List ->fl_files[I]. F_vnode = vnode_ptr; //vnode list->fl_files[i].f_priv = priv; List ->fl_files[I]. F_refcount = 1; List ->fl_files[I]. F_mapping = NULL; List ->fl_files[I]. F_dir = NULL; List ->fl_files[I]. F_magicnum = files_magic_generate(); Process_files = OsCurrProcessGet()->files; Return (int) I; } i++; } / /... }Copy the code

read | write

SysRead / / system call | read files: read from the file content in buf nbytes length (user space) fd = GetAssociatedSystemFd (fd); // Obtain system fd read(fd, buf, nbytes) from process fd; Fs_getfilep (fd, &filep); Ret = (int)filep->ops->read(filep, (char *)buf, (size_t)nbytes); // Invoke read operations for a specific file systemCopy the code
SysWrite / / write file system call | : (user space) in buf nbytes length fd = GetAssociatedSystemFd writes the content of the file (fd); // Obtain system fd write(sysfd, buf, nbytes) from process fd; Fs_getfilep (fd, &filep); File_seek64 file_write(filep, buf, nbytes); Ret = filep->ops->write(filep, (const char *)buf, nbytes); // Invoke a write operation for a specific file systemCopy the code

Only the file_write implementation is shown here

ssize_t file_write(struct file *filep, const void *buf, size_t nbytes) { int ret; int err; if (buf == NULL) { err = EFAULT; goto errout; } /* Was this file opened for write access? */ if ((((unsigned int)(filep->f_oflags)) & O_ACCMODE) == O_RDONLY) { err = EACCES; goto errout; } /* Is a driver registered? Does it support the write method? */ if (! filep->ops || ! filep->ops->write) { err = EBADF; goto errout; } /* Yes, then let the driver perform the write */ ret = filep->ops->write(filep, (const char *)buf, nbytes); if (ret < 0) { err = -ret; goto errout; } return ret; errout: set_errno(err); return VFS_ERROR; }Copy the code

close

Int SysClose(int fd) {int ret; /* Process fd convert to system global fd */ int sysfd = DisassociateProcessFd(fd); Ret = close(sysfd); DisassociateProcessFd if (ret < 0) {if (ret < 0) {if (ret < 0) { // continue the correlation with return-get_errno (); } FreeProcessFd(fd); // Release process fd return ret; }Copy the code
  • Terminate the processfdAnd the systemfdBinding relation of
  • closeI’m going to have to decide if the number of references to this file is zero0, only for0It will actually be executed_files_close 
    int files_close_internal(int fd, LosProcessCB *processCB) { //... list->fl_files[fd].f_refcount--; if (list->fl_files[fd].f_refcount == 0) { #ifdef LOSCFG_KERNEL_VM dec_mapping_nolock(filep->f_mapping); #endif ret = _files_close(&list->fl_files[fd]); if (ret == OK) { clear_bit(fd, bitmap); }} / /... } static int _files_close(struct file *filep) { struct Vnode *vnode = filep->f_vnode; int ret = OK; /* Check if the struct file is open (i.e., assigned an vnode) */ if (filep->f_oflags & O_DIRECTORY) { ret = closedir(filep->f_dir); if (ret ! = OK) { return ret; } } else { /* Close the file, driver, or mountpoint. */ if (filep->ops && filep->ops->close) { /* Perform the close operation */ ret = filep->ops->close(filep); if (ret ! = OK) { return ret; } } VnodeHold(); vnode->useCount--; /* Block char device is removed when close */ if (vnode->type == VNODE_TYPE_BCHR) { ret = VnodeFree(vnode); if (ret < 0) { PRINTK("Removing bchar device %s failed\n", filep->f_path); } } VnodeDrop(); } /* Release the path of file */ free(filep->f_path); /* Release the file descriptor */ filep->f_magicnum = 0; filep->f_oflags = 0; filep->f_pos = 0; filep->f_path = NULL; filep->f_priv = NULL; filep->f_vnode = NULL; filep->f_refcount = 0; filep->f_mapping = NULL; filep->f_dir = NULL; return ret; }Copy the code
  • The lastFreeProcessFdResponsible for releasing resources that this file occupies at the process level

Intensive reading of the kernel source code

Four code stores synchronous annotation kernel source code, >> view the Gitee repository

Analysis of 100 blogs. Dig deep into the core

Add comments to hongmeng kernel source code process, sort out the following article. Content based on the source code, often in life scene analogy as much as possible into the kernel knowledge of a scene, with a pictorial sense, easy to understand memory. It’s important to speak in a way that others can understand! The 100 blogs are by no means a bunch of ridiculously difficult concepts being put forward by Baidu. That’s not interesting. More hope to make the kernel become lifelike, feel more intimate. It’s hard, it’s hard, but there’s no turning back. 😛 and code bugs need to be constantly debug, there will be many mistakes and omissions in the article and annotation content, please forgive, but will be repeatedly amended, continuous update. Xx represents the number of modifications, refined, concise and comprehensive, and strive to create high-quality content.

Compile build The fundamental tools Loading operation Process management
Compile environment

The build process

Environment script

Build tools

Designed.the gn application

Ninja ninja

 Two-way linked list

Bitmap management

In the stack way

The timer

Atomic operation

Time management

 The ELF format

The ELF parsing

Static link

relocation

Process image

 Process management

Process concept

Fork

Special process

Process recycling

Signal production

Signal consumption

Shell editor

Shell parsing
Process of communication Memory management Ins and outs Task management
spinlocks

The mutex

Process of communication

A semaphore

Incident control

The message queue

 Memory allocation

Memory management

Memory assembly

The memory mapping

Rules of memory

Physical memory

 Total directory

Scheduling the story

Main memory slave

The source code comments

Source structure

Static site

 The clock task

Task scheduling

Task management

The scheduling queue

Scheduling mechanism

Thread concept

Concurrent parallel

The system calls

Task switching
The file system Hardware architecture
File concept

The file system

The index node

Mount the directory

Root file system

Character device

VFS

File handle

Pipeline file

 Compilation basis

Assembly and the cords

Working mode

register

Anomaly over

Assembly summary

Interrupt switch

Interrupt concept

Interrupt management

HongMeng station | into a little bit every day, the original is not easy, welcome to reprint, please indicate the source.