In-depth understanding of Swift code coverage

By introducing the generation principle of Swift code coverage, this paper supports combining the code coverage of each test in the scenario of distributed compilation and testing of CI, and finally restores the true coverage results.

📡 Tiktok Basic Technology continues to recruit in Beijing, Shanghai, Shenzhen and Hangzhou. Technology stack Swift, with assembly and C++ foundation bonus, can not be taught by hand. Welcome to add my wechat SevenkPlus consultation, free of charge to provide job introduction, team introduction, resume modification, progress tracking, interview counseling and other services, can also recommend excellent business team.

background

I am currently in charge of the development of a basic library within a Bytedance group. As a heavy-duty basic library open to all apps, we attach great importance to the construction of automated test capabilities and the accumulation of various cases. The accumulation of cases is not a r&d burden, but rather allows our team to do sufficient optimization, iteration, and refactoring within the base library without compromising delivery quality. At the same time, we strongly subscribe to the concept of precision testing, with test coverage as an indicator of ongoing concern, with the expectation that all externally delivered code should be tested.

We use XCTestplan introduced by WWDC2019 to manage various cases in groups. It is very simple to enable test coverage. As long as you turn on the switch in XCode, you can get coverage information as shown in the figure below:

The problem

As the underlying libraries become more powerful, the complexity of testing increases. We need more different types of cases to ensure the quality of the base library. Here are some common examples:

• Some low-level functions only need unit testing to ensure quality, some scenarios need interactive testing, and some layout related scenarios, we chose to use the tool iOSSnapshotTestCase for pixel comparison testing
• Tests that focus on logical correctness are generally compiled in Debug mode, while performance-sensitive and other special tests need to be compiled in Release mode or in more customized mode
• For some versioned logic, you need to run the test on the corresponding version of the device

As more and more cases accumulate, the challenge is the dual inflation of compile time and test execution time. Considering the total compilation time for the various compilation modes, as well as the execution time for all cases, which is currently over 30 minutes and may exceed an hour in the long term, this is clearly unacceptable.

Fortunately, the company has developed a set of Mac cluster system, which can assign tasks to different machines at the same time, and reduce the total CI time by parallel compilation and testing on multiple machines. This is a relatively long-term and reasonable technical solution to the root cause of CI taking too long.

However, this solution will also bring a problem: similar to the concept of K8S and Docker, every time CI scripts are executed, the scheduling management platform will assign the most appropriate machine from all Mac machines in the group to create a virtual environment for executing CI scripts. CI scripts are also executed in a highly isolated environment, with no knowledge of the host physical machine, in order to avoid interactions between tasks. Because the paths for each compilation are different, coverage from running some of the tests separately in different tasks cannot be combined. We might get the following data:

In this case, the same lock file has different compilation paths, so from the coverage tool’s perspective, it is actually three files with different ids that happen to have the same content. Our goal is to show the coverage tool that they are the same file and that the coverage of each line needs to be summed up exactly.

Principles of coverage statistics

Swift coverage

The first and simplest conclusion is that simple coverage summations must not be trusted because of the repetitive parts.

Coverage data that can be seen in Xcode is saved in the.xcresult file after the test is run. You can parse it with the xcrun xccov view –report path/to/xx.xcresult command, but you can only get the coverage percentage, still no more detailed data to merge. Moreover, Xcode’s data format is relatively more black box. For example, no good tools have been found to present the coverage information in the form of web pages.

So the author turned to more raw data, namely Swift’s own coverage calculation. In the early days of Swift, there were tools like SwiftCov that could be used to generate coverage data, but with Swift2’s native support for coverage information, there is actually one and only one official plan for Swift coverage, which is based on LLVM. Another benefit of an official based solution is that the tools are relatively mature, such as generating web pages that show exactly how each line of code is executed, and supporting statistics for complex scenarios such as generic specialization.

For specific information, please refer to this official document: Source-based Code Coverage. Here, the author will break down the process of generating Coverage into three stages, and separately introduce what is done in each stage and how Coverage data is finally generated.

MachO file compilation

The essence of enabling test coverage is the addition of two compilation parameters: -profile-generate and -profile-coverage-mapping. These two parameters can also be added manually to the Swift Compiler -> Custom Flags -> Other Swift Flags option.

For details about the two parameters, run the swiftc –help command

The explanation here may not be intuitive enough, so take a look at the compiled MachO file and see what’s special about it. The first thing you can see is that there are more sections, which are used for coverage statistics.

Using the previous lock function as an example, if you look at its assembly, you can see that the following dot logic has been inserted.

The ___profc_xxx symbol can be understood as a dot counter, and the specific address is stored in the __DATA and __llVM_prF_cnts sections of the MachO file. At the start of the program, all the counters have a value of zero, and each time the corresponding code is executed, the counters are incremented by one.

Now it’s easy to understand what the two compilation parameters mean. The extra code in the MachO file is generated by the -profile-generate parameter. The extra __LLVM_COV section is generated by the -profile-coverage-mapping parameter. The possible reason for this separation is that, in addition to coverage analysis, the piling information can also be used for PGO Optimization. For reference, Profile Guided Optimization can be used to improve Application performance.

It is worth mentioning that the number of studs is not a one-to-one relationship with the number of lines of code or the number of functions. In fact, each pile corresponds to a Basic Block (BB). A Basic Block is a piece of code that has only one entry and one exit and no other jump/return/if statements in between. The advantage of a Basic Block for a pile is that there is no need to pile every line, thus greatly reducing the size of the executable and increasing the speed of execution. At the same time, it can accurately analyze the execution of all code.

Note: There is no explicit inclusion relationship between BB and row. For example, a triplet operator actually contains two BB’s. A BB represents only a starting row and column number, and an ending row and column number.

Profraw and profData are generated

When the docked executable is finished running, LLVM extracts the execution data from the __DATA/ __LLVM_prF_cnts section, Generate a profraw and profdata file in the Build/ProfileData/ directory of DerivedData directory.

Profraw can be thought of as raw data extracted, completely binary and unreadable. Profdata, on the other hand, is structured data that has been aggregated. Although it is binary, you can already see some counter information in it. You can use the following command to convert the profRAW file to a profdata file.

You can use the llvm-profdata show command to view the contents of profdata:

In the figure above, you can see that this Basic Block has been executed 138 times.

In fact,.profdata can be thought of simply as an array of the above structures, which essentially records the number of times a buried point (Basic Block) is executed.

For full usage, see the official document LLVM -profdata – Profile data Tool

Coverage export

Process and effect

To finally get the coverage data, you need three files:

• Buried point count result:.profdata
• MachO executable
• Source directory

Then use the llvm-cov command to generate the coverage report in HTML format:

This enables coverage reports to be generated:

The index. HTML is the coverage data summary of all files, and the coverage information accurate to the line level of each file is saved in the Coverage folder, and each file corresponds to one HTML. Because the source code is confidential, here take Chromium coverage information as an example, demonstrate the results generated by the OFFICIAL LLVM tool:

Export principle

The most critical step in tracing the entire process is to understand how the llvm-cov command generates coverage reports.

First we know that the profdata file only has the number of calls to the counter, and that the source code in the coverage report must be obtained from the source path we passed in. Therefore, the counter information can be associated with the source code, must depend on the MachO file. After consulting the materials, the author’s conjecture was confirmed in the LLVM document: LLVM Code Coverage Mapping Format:

LLVM’s code coverage mapping format is designed to be a self contained data format that can be embedded into the LLVM IR and into object files. It’s described in this document as a mapping format because its goal is to store the data that is required for a code coverage tool to map between the specific source ranges in a file and the execution counts obtained after running the instrumented version of the program.

The mapping data is used in two places in the code coverage process:

1. When clang compiles a source file with -fcoverage-mapping, it generates the mapping information that describes the mapping between the source ranges and the profiling instrumentation counters. This information gets embedded into the LLVM IR and conveniently ends up in the final executable file when the program is linked.
2. It is also used by llvm-cov – the mapping information is extracted from an object file and is used to associate the execution counts (the values of the profile instrumentation counters), and the source ranges in a file. After that, the tool is able to generate various code coverage reports for the program.

This section clearly explains that the MachO file provides a mapping between a pile counter on one side and the source range corresponding to that counter on the other. To locate a piece of code, you need five more parameters: the source path, the starting line and column numbers, and the ending line and column numbers. This information is written to the __LLVM_COV section when compiling the MachO file, which is then parsed by the llvm-cov tool.

After observation, the author found that the path information of the source code can be directly seen in this section of __llVM_comap, and the information of the source code starting and ending range is Basic Block, which may be compressed due to the storage method, and can not be read intuitively, but this does not affect the understanding of the process.

The principle of summary

In summary, the calculation process of Swift code coverage can be roughly divided into two processes, as shown in the figure below:

The first process is the compile execution process. The source code is first compiled into an executable file using the -profile-generate and -profile-coverage-mapping parameters.

The code is staked in the compilation product. Executing code to a specific location increases the value of the counter. The counter and the source location information of the corresponding Basic Block are stored in the __LLVM_COV section of the compiled product.

The value of the counter increases as the code is executed and the __DATA section is written during the compilation. A.profraw file is generated and consolidated into a.profdata file, which records each counter and how many times it was called.

The other process is the coverage generation process. You need to use the executable files and profdata from the previous process and generate coverage reports in combination with the source code. The specific principle is: traversal each counter in profdata, according to the mapping stored in the executable file, to find the counter corresponding to the statistics of the section of source code, thus generating row level coverage information.

The solution

So far, we’ve figured out how Swift code coverage is calculated. In the current engineering situation, we have multiple XCTestPlans running on different CI machines, thus generating multiple profdata files and multiple identical executables.

Therefore, the overall idea is to merge multiple profdata files based on the LLVm-profdata merge command, and then arbitrarily select an executable file, combined with the source code, to generate coverage reports.

The reason profdata can be combined, or even simply added, is that it counts the number of Basic Block executions. Basic blocks correspond to the least granular flow control of a program. The same source code will compile the same Basic Block, and the code in the same Basic Block must be executed the same number of times.

Route map

The first thing to solve is the problem of random compilation paths. Suppose our working path is A at compile time and B at coverage report generation time. This results in A path A stored in the executable’s __LLVM_COV segment, and the source code that we get when we generate the coverage report is actually on the physical path B, so it is not properly associated with the source code. In particular, the generated coverage information is empty.

I used one of apple’s new compilation parameters in Swift 5.3: -coverage-prefix-map. For details, see this MR. In short, this parameter maps the source path in the __LLVM_COV segment from the current actual path to any virtual path. This virtual path is then restored to the real path in the llvm-cov command.

For example, we can add A compilation parameter to Other Swift Flags: -coverage- prefifiz-map $PWD=/ROOT, so that we can see in the __LLVM_COV section that the source code is in /ROOT instead of A. Then, when generating the coverage report,B told LLVM that /ROOT was the virtual path and B was the actual path. Llvm-cov-path-equivalence =/ROOT.

With the -coverage-prefix-map parameter of Swiftc and the -path-equivalence parameter of llvm-cov, we achieved the conversion of any compilation path A to the actual parsing path B.

In theory, if we could get the path A directly when we generated the coverage report, we could also use llVm-Cov-path-equivalence =A,B to achieve the goal. Path A can be obtained in A number of ways, such as parsing profdata or executable files, or passing parameters through A CI system. Due to time (laziness), the author did not try.

Profdata processing

Once you’ve solved the path-mapping problem, you’ve actually run through the process. The specific approach is:

1. The NxctestplanAssign it to N machines, each running a task
2. Let’s take each of these N tasksprofdataAs a product, it is provided to downstream coverage analysis tasks, and the first task also provides an executable file
3. In the coverage analysis task:
   1. First download the code with the same commit-id, along with the first N tasksprofdataAnd the first task executable
   2. usellvm-profdata data merge A.profdata B.profdata ... -output merge.profdataCommand, will allprofdataMerge into one
   3. usemerged.profdataAnd download down the source code and executable files, with-path-equivalenceParameter to generate coverage reports.

After testing, the coverage of the aggregate is indeed higher than that of any single run, and it works well.

However, after a period of operation, according to the feedback of students in the group, the author still found a case that did not meet the expectations. A section of a function that has zero coverage after explicitly adding test cases.

After investigation, it was found that the problem appeared in the name of the pile counter. For simple functions, the counter name can be simply understood as the mangle result of the function name, independent of the actual path, so the counter name will remain the same no matter what machine it is compiled on. In this way, when merging profdata, you can simply add the count for each run of the Basic Block to get the final count.

However, for some closures, even implicit closures, the counter name contains the path information. It is not clear whether it is a Feature or a Bug. For example, take the lock() function again.

The if determines the counter for the corresponding two Basic blocks, and the name carries the path information. In this way, the merged profdata will be determined to be different counters. The merged profdata will contain the following data:

Expect counter_name = /path/A/counter count = 6

Fortunately, after some research, the names of these counters were the same except for the path. Since the executable we are using corresponds to the first profdata, we only need to change the counter names in other profdata to the same path as the first profdata.

Since LLVM does not provide tools to modify profdata, the initial approach was to use SED to modify the binaries directly. If the path length changes, the profdata file format will be corrupted and the profdata file cannot be read. This may be due to the specific layout structure within the Profdata. The working directory of the CI environment is random and does not guarantee consistent length, so this solution fails.

The ultimate solution is to start with upstream data, and since profdata is generated from the profRAW file, the two data are completely equivalent. So turn your attention to the handling of the profRAW file. The llvm-profdata merge command enables you to select the format of the output results. In addition to the default binary format, the -text parameter is also supported to generate plain text profdata. If you change the path, the profdata format will not be broken.

The merge of profdata in text format also worked as expected, and the number of calls to the same counter was correctly added, resulting in the correct coverage report.

Project summary

By analyzing the implementation principle of Swift code coverage, this paper supports the distributed compilation and testing of Swift code, and combines the coverage rate to display, which fundamentally controls the time consuming of CI and provides feasibility for quality assurance in subsequent more complex scenarios.

The specific approach is:

• Will each test resultprofrawFile parsed to text formatprofdataAnd unify the path
• Merge multipleprofdataGet true coverage data
• usellvm-cov -path-equivalenceCommand to complete source path mapping

With the implementation of this scheme, we split five tests, and the overall CI time decreased from more than 30 minutes to about 10 minutes. At the same time, with certain technical accumulation and reserve in Swift coverage, more long-term iterations such as coverage Diff analysis and bayonet can be carried out in the future.

AD time

Douyin Basic technology continues to recruit in Beijing, Shanghai, Shenzhen and Hangzhou. Welcome to join us on wechatsevenkplusConsulting, free of charge to provide job introduction, team introduction, resume modification, progress tracking, interview counseling and other services. Or scan a code to send out a resume

Reference documentation

1. Source-based Code Coverage
2. 🚧 source-based Code Coverage for Swift Step by Step
3. Use Profile Guided Optimization to improve Application performance
4. llvm-profdata – Profile data tool
5. llvm-cov – emit coverage information
6. 🚧 Code Coverage in Chromium
7. LLVM Code Coverage Mapping Format
8. Add path remapping with -coverage-prefix-map to coverage data #32416