preface

The company is currently working on an enterprise-level intelligent customer service system, which has high requirements for system stability, but it is difficult to ensure that users will not have problems in use. Android SDK is integrated into customers’ apps, and due to the fragmentation of Android, it is particularly difficult to identify problems with SDK. Therefore, it is very important to record user operation logs.

The initial plan

At first, the SDK logs directly by writing to a file. When a log is written, the file is first opened, the log is written, and the file is closed. The problem with this is the frequent I/O, which affects the performance of the application, and the fact that the SDK maintains a background process to ensure the timeliness of messages, which makes it worse when one of the processes writes to the log while the other is locked out. Using this approach may not seem to have a significant impact on the program at the moment, but as the volume of logging increases and more IO operations are performed, performance bottlenecks are bound to occur.

Let’s analyze the process of writing directly to a file:

  1. The user initiated the write operation
  2. The operating system searches for the page cache. A. If the page cache is not matched, a page missing exception is generated, a page cache is created, and the incoming content is written to page cache B. On a hit, the user’s incoming content is written directly to the page cache
  3. The user write call completes
  4. A page becomes dirty. The operating system has two mechanisms to write the dirty page back to disk: a. The user manually calls fsync() b. Dirty pages are periodically written back to disk by the PDFlush process

As you can see, when data is written from a program to disk, there are actually two copies of data: one from user-space memory to the kernel space cache, and one from the kernel space cache to the hard disk when written back. This also involves frequent switching between kernel space and user space when write back occurs.

There is also a “write magnification” problem with SSD storage compared to mechanical hard drives. The problem has to do with the physical structure of SSD storage. After all data is written on an SSD, the data written to the SSD cannot be directly updated. The data can only be overwritten. Before overwriting, the data must be erased. But the smallest unit of writing is Page, and the smallest unit of erasing is Block, and blocks are much larger than pages, so to write new data, you need to first read the data on the Block and combine it with the data to be written, then erase the Block, and finally write the read data back to the storage. As a result, the actual data written may be much larger than the data originally written.

I did not expect that simple writing files involved so many operations, but for the application layer transparent.

Since there are so many times to write to a file, can we cache the log and then write it to disk once after a certain number of times?

This can significantly reduce the number of I/OS, but it can lead to another, more serious problem — log loss

If the logs are cached in memory, the integrity of the logs cannot be guaranteed when the program crashes or the process is killed. Moreover, because the SDK has multiple processes, the order of logs in multiple processes cannot be guaranteed.

A complete log scheme needs to be met

  • The system performance is not affected. The application cannot be slowed down due to the introduction of the log module
  • Ensure the integrity of the log, if not, then log collection is meaningless
  • For multi-process applications, ensure that the final log order is accurate

High performance solution

Since we can’t reduce the number of writes, can we optimize the writing process?

The answer is yes, using MMAP

Mmap is a method of memory mapping files, that is, a file or other objects are mapped to the address space of the process, to achieve a mapping relationship between the file disk address and a virtual address in the process virtual address space. The function prototype is as follows

void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);
Copy the code

The MMAP operation provides a mechanism for user programs to access device memory directly, which is more efficient than copying data between user space and kernel space. It is commonly used in applications requiring high performance.

In addition, MMAP ensures log integrity. The write back time of MMAP is as follows:

  • Out of memory
  • Process exits
  • Call msync or munmap
  • MAP_NOSYNC 30s-60s(FreeBSD only)

When a file is mapped, the program will apply for a space of the same size in the native memory. Therefore, it is recommended to map a small section of content, such as 64K, each time, and then map the content behind the file again when it is full.

With the log write performance and integrity issues resolved, how do you ensure log order across multiple processes?

Since MMAP uses shared memory to write data, if two processes map the same file at the same time, there will be a log override problem.

Since the order cannot be guaranteed directly, the next best thing is that the two processes map different files, merge them once a day, and sort the logs as they merge.

Continue to optimize

According to the above scheme, there seems to be no problem in designing JNI interface, packaging SO and introducing SDK. However, as an SDK, I always feel that including SO is not very friendly, which will increase the difficulty of access to some extent.

Can we not use so?

Java already provides an implementation of memory mapping — MappedByteBuffer

The MappedByteBuffer is located under the Java NIO package and is used to map the contents of the file to the buffer using the MMAP technology. The buffer can be created using FileChannel’s map method

RandomAccessFileraf = new RandomAccessFile(file, "rw");
MappedByteBuffer buffer = raf.getChannel().map(FileChannel.MapMode.READ_WRITE, position, size);
Copy the code

In order to test the efficiency of MappedByteBuffer, we wrote 64byte data to memory, MappedByteBuffer and disk file for 500,000 times respectively, and calculated the time

methods Time consuming
memory 384ms
MappedByteBuffer 700ms
Disk file 16805ms

As you can see, MappedByteBuffer performance is not as good as memory writing performance, but it is a quality improvement compared to writing disk files.

conclusion

This paper mainly analyzes the problems existing in the direct file writing log mode, and proposes a high-performance file writing scheme, MMAP, which takes into account the writing performance and integrity, and ensures the log order in multi-process through compensation scheme. Finally, the implementation of memory mapping in Java layer is found, avoiding the introduction of SO.

A Brief Book on Transferring Self 2018.01.28