I. Preface: Why does the mobile terminal need a tripartite log system

Logging systems are used to record user behavior and data as well as thread call stacks in the event of a crash to help programmers solve problems and optimize the user experience.

IOS system has its own Crash collection App “ReportCrash” to collect App Crash information. I have also had an in-depth understanding of the process of iOS Crash information collection and recorded the process from CPU anomalies to the generation of Crash Log. However, Crash is not the only problem encountered by users. In some cases, Crash cannot be located only by Crash Log, and Crash information collected by ReportCrash can be shared with developers only after users’ consent. To illustrate the importance of user logs, here is a faceu team – light face camera – brother Tom’s joke

“For example, the user feedback, the photo is too yellow, through a dozen render pipeline, no log, you can not ask the user to take a photo again.”

Therefore, most apps have Crash collection framework and log collection framework, and Crash information is also a kind of log information. Why should it be divided into two frameworks for collection? Because information collection methods are different, Crash collection framework collects information by capturing Mach anomalies and Unix signals sent by the system, while log collection framework is information collection triggered by programmer’s active code. Information collection part shares few codes, so it is easier to maintain the two frameworks.

The log system described here is a log system that collects non-crash information. The system is divided into three parts:

  • Collection part: Use wechat open source log collection component XLog, which has the advantages of security, fluency, integrity and fault tolerance.
  • Transfer part: Spring Boot is used to develop the Web server. The Web server provides simple functions of uploading and downloading files, obtaining and setting the whitelist of uploading files.
  • Management part: Use Ant Design Pro framework to develop background management system, provide pages in the management system to set the log upload whitelist, and log download.

Let’s talk about the selection process of my technology

Technology selection: the technology stack of the log system

It is said that technical service business, considering the selection of technology is inseparable from the business needs of course.

Instead of using a tripartite framework, they wrote some simple policies themselves, such as writing files by priority or reporting to the server. But it still can not meet the needs of client development

  • There is a fatal disadvantage: there is no visual platform for looking for logs, and the server students corresponding to the interface of reporting logs have long left their jobs, and it is time-consuming and laborious to find other server students to help retrieve logs.
  • Secondly, the Log file written in plaintext cannot guarantee the security, the Log integrity cannot be guaranteed in the blink back, and the local Log management policy does not limit the number of files and the size of a single file
  • In addition, logs are only output to console and files. If you want to output data to web pages for display through web sockets in real time, the code level is not easy to expand. This function can help debugging push and testing students to view buried information in real time.

Development has a time limit, and requirements have a priority

2.1 Highest priority requirements: stable log transfer service and visual log management

Server and front-end human nervous, product demand and this is not to earn money, not necessarily for the scheduling server and front-end students support the stable log transmission service and visualization of the log management “this requirement, and we Crash collection and check tool is done with tencent, is not the same company the same department, Don’t expect them to support you, just look for solutions online.

2.1.1 Seafile Personal Web disk + CocoaLumberjack log collection

I lack of experience in server-side development, only engaged in VPS to build VPN and blog, and did not play web server, find a solution that does not need to build a Web server. The server builds Seafile personal web disk service

Seafile is a set of open source, professional and reliable cloud storage project management software made in China, which solves the problems of centralized file storage, sharing and cross-platform access. It was officially released in October 2012. In addition to cloud storage and sharing functions provided by general web disk, Seafile also provides auxiliary functions such as message communication and group discussion to help employees to work better together around files, which has been used by more than 100,000 users.

Seafile is a mature solution, but also provides interface to upload and download files, quickly buy a 10 yuan/month Tencent cloud student machine to take a Seafile personal web disk verification.

I solved the problem of “stable log transfer service and visual log management” in half my spare time. Then I investigated the client log collection framework and found that CocoaLumberjack Start has a large number of logs and recently submitted records. Moreover, the framework is designed to be easy to expand. In addition, CocoaLumberjack also supports setting the number of files and the size of a single file. CocoaLumberjack this program looks very good, and immediately masturbated a Demo to verify.

I took the Demo to discuss with my colleagues and summarized several problems:

  • In Seafile, high customization sacrifices scalability

    • In addition to file upload and download, there is no way to customize the interface for more C/S interactions, such as the client asking the server about the log upload type, whether to upload user logs or database files, whether to send to wechat friends or upload to the server.
    • The data store is lonely, cannot be displayed in association with other data, and has poor scalability. For example, it is reasonable to put the log files of user feedback problems in the same place as Crash collection files. Although Crash collection uses a third-party framework, it cannot modify the code to expand by itself. In case we do not cooperate with that third-party framework in the future, or the data of user log needs to be linked with other data.

  • CocoaLumberjack still cannot guarantee the integrity of the Log and cannot guarantee that the Log has been written to the file when the App crashes. Because CocoaLumberjack uses -[NSFileHandle writeData:] to write files, this method cannot ensure Log integrity and is slower than mMAP method that uses memory-mapped files to write files

2.2 Scheme Optimization

Since you can’t find the server and the front end of the students to help, then brave it on their own. Break them down and take them down one by one! The log system is divided into three parts: collecting part, transmitting part and managing part

  • Collection part: Use wechat open source log collection component XLog, which has the advantages of security, fluency, integrity and fault tolerance.
  • Transfer part: Spring Boot is used to develop the Web server. The Web server provides simple functions of uploading and downloading files, obtaining and setting the whitelist of uploading files.
  • Management part: Use Ant Design Pro framework to develop background management system, provide pages in the management system to set the log upload whitelist, and log download.

2.2.1 Collection: High-performance log collection component XLog

Fluency is the first consideration

In the collection section, THE importance of fluency has not been mentioned before. In fact, this is the highest priority requirement of the log collection component, and the application should not lag or consume more power because of frequent log writes. Because logging was not used at a high frequency and high density in the project, the importance of fluency was not realized early enough. Why does frequent document writing stall?

When a file is written, data is first written to the system cache (dirty Page) rather than directly to the disk. The system usually writes dirty page to the disk in the following cases:

  • The operating system periodically writes the dirty pages back to disk when the page flag is marked as changed.
  • Call the user-mode write interface -> trigger the kernel-mode sys_write-> file system to write data back to disk. The time when the file system writes back to disks is also uncontrollable. If the memory usage of dirty pages exceeds a certain percentage of the system memory, the file system writes back.

In addition, when data is written from program to disk, two copies of data are involved: one copy of user space memory to kernel space cache, and one copy of kernel space cache to disk during write back. Frequent switching between kernel space and user space also leads to performance losses.

Ensuring fluency is not equal to completeness

To avoid frequent file writing, create buffers in memory first and write files as appropriate. While this approach ensures smoothness, it does not guarantee integrity, and centralizing the compression of logs can cause CPU spikes for short periods of time. If the program crashes, the data in memory has not been persisted, and the real-time writing of files cannot guarantee the fluency, what should be done?

Mmap ensures fluency and integrity

Mmap uses logical memory to map disk files without any copy operation, avoiding data copy of written files. Manipulating memory is like manipulating files, avoiding frequent switching between kernel space and user space.

In order to verify whether mmap really has the efficiency of directly writing to memory, the wechat team wrote a simple test case: write 512 bytes of data to 150 KB of memory and mmap, as well as disk files 100w times and counted the time consuming

In addition to ensuring smooth running, MMAP can also take into account log integrity. In the following cases, write back to disk automatically

  • Out of memory
  • Process exits
  • Call both functions
    • msync(mmap_ptr, mmap_size, MS_ASYNC)Synchronous or asynchronous write back to disk
    • munmap(mmap_ptr, mmap_size)When a map is removed, the content is written back to disk
  • 30s-60s without MAP_NOSYNC (FreeBSD only)
Xlog also guarantees thatsecurity 和 Fault tolerance

By compression and encryption, log information can be written into the disk in plaintext and the mMAP size can be reduced. Strategy is

Logs are compressed, encrypted, and written to logical memory before being written to logical memory

WeChat choose specific compression scheme can refer to the article cloud.tencent.com/developer/a… In simple terms, streaming compression ensures fault tolerance of logs,

Even if some data in a compression unit is corrupted, streaming compression does not affect the decompression of the logs in this unit before the data is corrupted, but only the logs in this unit after the data is corrupted.

So in one sentence, why is Xlog safe, smooth, complete, and fault tolerant

A single line of logs is compressed by streaming compression. After being compressed and encrypted, the logs are written to the Mmap as the intermediate buffer of logs. When the data in the MMAP reaches a certain size, the data is written to the disk file

You can also customize mMAP-based log collection components for your team by referring to the open source XLog API


FILE *fp = fopen(file_path, "wb+"); file_num = fileno(fp); ftruncate(file_num, size); / / adjust size char * mmap_ptr = (char *) mmap (NULL, size, PROT_READ | PROT_WRITE, MAP_FILE | MAP_SHARED, file_num, 0). Munmap (mmap_ptr, mmap_size); // Remove a map and write the contents back to disk msync(mmap_ptr, mmAP_size, MS_ASYNC); // Synchronous or asynchronous write back Disk Capacity expansion method: Disable Munmap, increase the file size, and reconfigure mMAPCopy the code

SatanWoo gobang master in xlog source analysis of the two points need to pay extra attention to the use of Mmap, article here

  • Note 1: If we try to open mmap successfully, but the corresponding data address of Mmap is NULL, then we must stop mapping. Because the address represented by NULL is in kernel state, once mapped, it is bound to cause Crash.
  • Note 2: In the case of mMAP, if the last application power failure or Crash, the log information still exists, but it may not be timely converted into the desired log file. So we first check if there is any data in the Mmap file, and then convert it to a log.

2.2.2 Transfer part: Rapid development of Web server using Spring Boot

There are many articles about Spring Boot on the Internet. A senior of Ant Financial wrote a quick start guide for Spring Boot, which I think is quite good. It introduces the knowledge points, circuit diagram and basic concepts of Spring Boot, as well as how to quickly create a Spring Boot application.

After developing my own student computer and debugging the interface with Postman, I went to the company’s operation and maintenance to ask for machine resources and deployed them to the company’s machine. I can’t introduce my experience in depth in this section because I just use Spring Boot to provide simple file upload and download functions, but I have put the corresponding Spring Boot part of the logging system on Github. Clone down and run github.com/HonchWong/H…

The name of the project is RDA, which stands for R&d Assistant, meaning the Spring-Boot application doesn’t stop there. I often encounter some problems that hinder efficiency in iOS business development and come up with a lot of “awesome” technical solutions to solve them. However, it is impossible to realize just being familiar with the Cocoa framework, and the support from the server side is indispensable. For example, I have been working on three requirements recently. Provide a visual interface to set up Mock network data without hard-coding Mock and reducing compilation time. [Client-side visualization generates bug lists with one click: provide a visual interface to input bug descriptions and generate bug lists to improve r&d efficiency.] Provide a platform to manage buried demand, verify buried point, buried point information custom display], this is also to set a flag, technical service business, business will be back to the technology, 19 years to learn the knowledge of the server, and then to improve this article 🙂

2.2.3 Management part: Directly fork “Ant Design Pro” for modification

Ant Design Pro is an enterprise-class back-end front-end/Design solution implemented by Ant Financial based on Ant Design’s Design specification and component library. This framework can be used to quickly develop back-end management systems. If you have experience developing React Native, you will be able to use Ant Design Pro based on React very quickly. The Ant Design Pro used in this logging system is available at github github.com/HonchWong/H…

How to use this log system

In addition to the server side and front-end Demo, I have submitted the iOS Demo to Github. Need to download two projects from Github web server project github.com/HonchWong/H… , iOS terminal project github.com/HonchWong/H…

  • Run the Web server locally
    • Build the environment, install JDK, Maven
    • HCRDA SpringBoot/SRC/main/resources/application. The properties in the configuration file is modified user log folder location, for example, this is my local store locationuserlog.dir.path=/Users/huanghongchang/Desktop/userLog
    • Run the MVN spring-boot:run command in the HCRDA -springboot directory
    • Go to http://localhost:9080
  • Running iOS projects
    • Pod Install Install dependencies
    • Example Change static NSString *_hostURL in hclogFileuploadManager. m to the IP address of localhost

3.1 XLog use

  • Initialize xlog. Four apis are involved. For details, see XLogHelper.m +setupXlog in Demo project

    • setxattr([[self xlogFileDirPath] UTF8String], attrName, &attrValue, sizeof(attrValue), 0, 0);This API is provided by the system library to prevent log files in this path from being synchronized by iCloud.
    • xlogger_SetLevel(kLevelDebug);Setting the Log Level
    • appender_set_console_log(true);Debug Specifies whether the console prints logs in the debug mode
    • appender_open(kAppednerAsync, [logPath UTF8String], "Test");Open the log file in the log directory and perform some initialization operations.

  • Type log, xlogger_Write(&info, message.utf8String), Demo encapsulates this API with macro definitions

  • Need to de-initialize appender_close() in APP termination method applicationWillTerminate

For more details, please refer to xlog wiki Mars iOS/OS X interface for details. In Demo, log logs are encrypted. Appender_open provides a public key for passing parameters. For details about how to use Xlog encryption, see the Xlog Encryption Guide

3.2 Uploading and downloading Logs

The log system uses a whitelist to upload logs. The whitelist is set in the background management system. The whitelist contains the unique user IDENTIFIER (UIN) and the corresponding log type (such as database or log log) to be uploaded.

After the MVN spring-boot:run command is executed in the HCRDA -springboot directory, access http://localhost:9080 in the browser, enter the admin account password 123, and eight whitellists can be found in user log-specified user. Currently, the Demo only implements “select log to upload a single file” and “upload all logs”.

After you set the whitelist, perform the following steps to upload the Demo file

You can view and download uploaded logs in the background management system

3.3 Viewing Logs

Instead of encrypting the xlog file, I encrypted the ZIP file, and used XOR to obfuscate the string to avoid using a disassembly tool like Hopper. Of course, this is not completely secure, but you can still use reverse App, dynamic debugging, and get the zip file password. It just makes it harder to crack. This script is used to decompress decode_mars_nocrypt_log_file

Run the python decode_mars_nocrypt_log_file.py xxx.xlog script

4. Refer to the article

  • Analysis of wechat High-performance Online Log System Xlog by Gobang masters
  • Xlog is a high-performance logging module for wechat Mars
  • Learn the logging system from the Xlog component of Mars
  • High-performance log recording mode – MMAP
  • Mars iOS/OS X Access Guide
  • Mars iOS/OS X interface description
  • Xlog Encryption Usage Guide