preface

Apple introduced Bitcode at WWDC 2015, and later added the ability to embed Bitcode in binaries (Enable Bitcode) in Xcode7, which is set to on by default. Many people have encountered bitcode errors when introducing third-party SDKS. After a search, they found that they just had to turn off the bitcode feature. In fact, most developers have no idea what bitcode is. In this article, we will take an in-depth look at Bitcode.

What is a Bitcode

LLVM is the compiler toolchain currently adopted by Apple. Bitcode is a code of the LLVM compiler’s intermediate code, which can be translated into the corresponding assembly instructions and machine code according to the target machine chip platform. While Bitcode is just an intermediate code that doesn’t run on any platform, it can be translated into any supported CPU architecture, including those that haven’t been invented yet. At what stage of LLVM’s work is Bitcode generated? To answer this question, we need a simple understanding of the LLVM workflow.

LLVM is a compiler framework system, written in C++, that optimizes compile-time, link-time, run-time, and idle-time of programs written in any programming language. Open to developers and compatible with existing scripts.

The LLVM program was initiated in 2000 by Dr. Chris Lattner of UIUC University. Chris Lattner joined Apple In 2006. And is committed to the application of LLVM in Apple development system. Apple is also a major funder of the LLVM program. LLVM has been adopted by Apple, Microsoft, Google, Facebook and other major companies. (This paragraph is from Baidu Encyclopedia)

To get a clearer idea of LLVM, look at this.

Since LLVM was just a compiler framework, it needed a front end to support the entire system, so Apple funded the development of Clang as a front end to compile C, C++, and Objective-C. You can think of Both Clang and LLD as part of LLVM. The framework means that you can develop your own modules based on the functionality provided by LLVM and add functionality to the LLVM system, or you can simply develop your own software tools and use LLVM to support the underlying implementation. LLVM is made up of libraries and tools, and because of the way it is designed, it is easy to integrate with an IDE (because IDE software can call libraries directly to implement functions such as static checking), and it is easy to build tools that generate various functions (because new tools only need to call the required libraries).

Below is a simple architecture for Clang/LLVM.

LLVM compile a source file process: preprocessing -> morphology -> Token -> syntax analysis -> AST -> (intermediate) code generation -> LLVM IR -> optimization -> generate assembly code -> Link -> object file

The front end of LLVM can use different compilation tools to lexical analyze the code files to form an abstract syntax tree (AST), and then convert the analyzed code into the INTERMEDIATE representation (IR) of LLVM. The optimizer in the middle part only operates on the intermediate expression IR and optimizes the IR through a series of passes. The back end is responsible for interpreting the optimized IR into machine code for the corresponding platform.

LLVM IR is the intermediate representation of LLVM, which is an important thing in LLVM. IR has three different representations. One is a readable IR, which is like assembly code, but it’s somewhere between higher language and assembly, and this representation is for people to see. The second is an unreadable binary IR, called bitcode, and the disk file has a.bc suffix. The third representation is a memory format that is kept only in memory, so there is no file format or file suffix. This format is one reason why LLVM compiles so fast. Unlike GCC, which generates intermediate procedure files at the end of each phase, the intermediate data it compiles is the IR of this third representation. The three formats are completely equivalent, we can specify the generation of these files in the arguments of the Clang/LLVM tool, and we can convert between the first two files using llVM-AS and LLVM-dis.

LLVM Backend is the real LLVM backend, also known as the LLVM core. It includes compilation, assembly, linking, and finally generating assembly files or object code.

From the introduction to LLVM above, we know that Bitcode is a binary representation of the file.bc that is generated when the front-end Clang processes the source code file. And it has two other representations, a readable intermediate between assembly and higher language.LL, and IR, which exists in memory.

Bitcode met

Since Bitcode is another representation of code, it must have its own syntax. We can find out by creating a simple demo.

Create the simplest c file and save it as bc.c. The code is as follows:

#include <stdio.h>

int main(a) {
    printf("hello world!");
    return 0;
}
Copy the code

You can then convert the source code file to the object file by using the following command.

O file bc_c.o //file command is used to identify the file type.Copy the code

Output:

bc_o.o: Mach-O 64-bit object x86_64

Clang -c is used to run only preprocessing, compilation, and assembly steps

The clang -o

command output to the target file file

The bitcode we want to explore happens to be in the clang -c process, LLVM provides us with the command to output bitcode.

// or clang-c-emit - LLVM bc.c file bc.bcCopy the code

Output:

c.bc: LLVM bitcode, wrapper x86_64

clang -c bc.bc -o bc.bc.o
file bc.bc.o
Copy the code

Output:

bc.bc.o: Mach-O 64-bit object x86_64

Compare the differences between two.o files:

XXD bc.bc.o > bc.bc.hex // Convert to readable format (hex file, Printf ("%02x") printf("%02x") printf("%02x") printf("%02x") printf("%02x") printf("%02x") printf("%02x") printf("%02x"Copy the code

From this we can determine that using the bitcode file as input to Clang, the resulting object file is the same as the source code.

Next we try to read the.bc file and output ASCII characters to see:

 hexdump -c bc.bc
Copy the code

-emit -llvm-s can be used to compile the source code into a text bitcode called LLVM Assembly Language, normally using.ll.

clang -emit-llvm -S bc.c -o bc.ll
file bc.ll
Copy the code

Output:

bc.ll: ASCII text, with very long lines

The complete document reads as follows:

; ModuleID = 'bc.c' source_filename = "bc.c" target datalayout = "E-m: O-p270:32:32 - P271:32:32 - P272:64:64 -i64:64- F80:128 - N8:16:32:64 -S128" target triple =" x86_64-apple-MacOSx11.0.0" @.str = private unnamed_addr constant [13 x i8] c"hello world! \00", align 1 ; Function Attrs: noinline nounwind optnone ssp uwtable define i32 @main() #0 { %1 = alloca i32, align 4 store i32 0, i32* %1, align 4 %2 = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([13 x i8], [13 x i8]* @.str, i64 0, i64 0)) ret i32 0 } declare i32 @printf(i8*, ...). #1 attributes #0 = { noinline nounwind optnone ssp uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "darwin-stkchk-strong-link" "disable-tail-calls"="false" "frame-pointer"="all" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "probe-stack"="___chkstk_darwin" "stack-protector-buffer-size"="8" "target-cpu"="penryn" "Target - the features" = "+ cx16, + cx8, + FXSR, + MMX, + sahf, + sse, + sse2, + sse3, + sse4.1, + ssse3, + x87" "unsafe - the fp - math" = "false" "use-soft-float"="false" } attributes #1 = { "correctly-rounded-divide-sqrt-fp-math"="false" "darwin-stkchk-strong-link"  "disable-tail-calls"="false" "frame-pointer"="all" "less-precise-fpmad"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "probe-stack"="___chkstk_darwin" "stack-protector-buffer-size"="8" "target-cpu"="penryn" "Target - the features" = "+ cx16, + cx8, + FXSR, + MMX, + sahf, + sse, + sse2, + sse3, + sse4.1, + ssse3, + x87" "unsafe - the fp - math" = "false" "use-soft-float"="false" } ! llvm.module.flags = ! {! 0,! 1,! 2}! llvm.ident = ! {! 3}! 0 =! {i32 2, !" SDK Version", [2 x i32] [i32 11, i32 0]} ! 1 =! {i32 1, !" wchar_size", i32 4} ! 2 =! {i32 7, !" PIC Level", i32 2} ! 3 =! {!" Apple clang version 12.0.0 (clang-1200.0.32.27)"}Copy the code

Refer to the documentation for the syntax of bicode.ll text.

Xcode的Enable Bitcode

Apple has this to say about it:

Enable Bitcode (ENABLE_BITCODE)

Activating this setting indicates that the target or project should generate bitcode during compilation for platforms and architectures that support it. For Archive builds, bitcode will be generated in the linked binary for submission to the App Store. For other builds, the compiler and linker will check whether the code complies with the requirements for bitcode generation, but will not generate actual bitcode.

Translation:

  • Enabling this setting means that the target or project should generate bitcode when compiling the platform and architecture that support it.
  • For Archive builds, bitcode is generated in the link binary for submission to the App Store.
  • For other versions, the compiler and linker check that the code meets the requirements for generating bitcode, but do not generate the actual bitcode.

That is, when we enable Bicode, bitcode will be generated in supported platforms and architectures, such as the current mainstream ARMV7 and ARM64. This is a limitation of Xcode. We can generate our own bitcode from the command line.

The generated bitcode is embedded in the binary of the final Archive and submitted to AppStrore along with the release. Turning this off will not generate the bitcode file and will not be included in the binaries when the package is built. This is why once this feature is turned on, you will find that the submitted packages are larger.

When carrying out the build of non-archive, Enable Bitcode will increase the compilation parameter -fembed-bitcode-marker. It is only marked in the object file, indicating that I can have Bitcode, but I have not brought it now. Because you don’t need bitcode for local compilation and debugging, only the AppStore does, removing this unnecessary step will speed up compilation.

Of course, you can set Enable Bitcode to NO and manually add -fembed-bitcode to the real architecture in Other Compiler Flags and Other Linker Flags. This way, any type of Build will come with bitcode.

-fembed-bitcode-maker: simply marks the location of bitcdoe in the binary archive.

-fembed-bitcode: Actually generates the bitcode directive, which is embedded in the binary. This setting should not only be set in the app, but also be used when compiling the static link library. This parameter is added only in archive mode by default

-optional from the clang command

Let’s take a look at how the file mutates with Enable Bitcode enabled.

clang -fembed-bitcode -c bc.c -o bc_bitcode.o
Copy the code

You can view the structure of an Object file using the otool tool

otool -l bc_bitcode.o
Copy the code

Output:

bc_bitcode.o: Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags 0xfeedfacf 16777223 3 0x00 1 4 680 0x00002000 Load command 0 cmd LC_SEGMENT_64 cmdsize 552 segname vmaddr 0x0000000000000000 vmsize 0x0000000000000d60 fileoff 712 filesize 3424 maxprot 0x00000007 initprot 0x00000007 nsects 6 flags 0x0 Section sectname __text segname __TEXT addr 0x0000000000000000 size 0x000000000000002a offset 712 align 2^4 (16) reloff 4136 nreloc 2 flags 0x80000400 reserved1 0 reserved2 0 Section sectname __cstring segname __TEXT addr 0x000000000000002a size 0x000000000000000d offset 754 align 2^0 (1) reloff 0 nreloc 0 flags 0x00000002 reserved1 0 reserved2 0 Section sectname __bitcode segname __LLVM addr 0x0000000000000040 size 0x0000000000000c60 offset 776 align 2^4 (16) reloff 0 nreloc 0 flags 0x00000000 reserved1 0 reserved2 0 Section sectname __cmdline segname __LLVM addr 0x0000000000000ca0 size 0x000000000000005a offset 3944 align 2^4 (16) reloff 0 nreloc 0 flags 0x00000000 reserved1 0 reserved2 0Copy the code

The file with Enable Bitcode enabled has two more sections, __LLVM,__bitcode and __LLVM,__cmdline, than the Bc. o object file.

Xcode provides the segedit command to export the specified Section directly, with only the given Section name interface.

segedit -extract __LLVM __bitcode bc_bitcode.o.bc -extract __LLVM __cmdline bc_bitcode.o.comdline bc_bitcode.o
Copy the code

Generate bc_bitcode.o.b.c. and bc_bitcode.o.comdline files. By this time, we should have the following documents (those not mentioned above need no attention).

(There is something wrong with the background of the picture, change it to another one, we will have to do with it ~)

Notice that the bc.bc we exported at the beginning is the same size as bc_bitcode.o.b.c. That’s a little suspicious. Compare the two files.

md5 bc.bc bc_bitcode.o.bc
Copy the code

Output:

MD5 (bc.bc) = 020389ec450aa7451bc105d7b010418c

MD5 (bc_bitcode.o.bc) = c0f0a511804418e167c3a74e1d816c21

Why, the exported bitcode file from the target file is different from the directly compiled bitcode?

We try to convert the format to make the bc_bitcode.o.b.c. file readable.

clang -emit-llvm -S bc_bitcode.o.bc -o bc_bitcode.o.ll
Copy the code

The contents of the bc_bitcode.o.l file are as follows:

; ModuleID = 'bc_bitcode.o.bc' source_filename = "bc.c" target datalayout = "E-m: O-p270:32:32 - P271:32:32 - P272:64:64 -i64:64- F80:128 - N8:16:32:64 -S128" target triple =" x86_64-apple-MacOSx11.0.0" @.str = private unnamed_addr constant [13 x i8] c"hello world! \00", align 1 ; Function Attrs: noinline nounwind optnone ssp uwtable define i32 @main() #0 { %1 = alloca i32, align 4 store i32 0, i32* %1, align 4 %2 = call i32 (i8*, ...) @printf(i8* getelementptr inbounds ([13 x i8], [13 x i8]* @.str, i64 0, i64 0)) ret i32 0 } declare i32 @printf(i8*, ...). #1 attributes #0 = { noinline nounwind optnone ssp uwtable "correctly-rounded-divide-sqrt-fp-math"="false" "darwin-stkchk-strong-link" "disable-tail-calls"="false" "frame-pointer"="all" "less-precise-fpmad"="false" "min-legal-vector-width"="0" "no-infs-fp-math"="false" "no-jump-tables"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "probe-stack"="___chkstk_darwin" "stack-protector-buffer-size"="8" "target-cpu"="penryn" "Target - the features" = "+ cx16, + cx8, + FXSR, + MMX, + sahf, + sse, + sse2, + sse3, + sse4.1, + ssse3, + x87" "unsafe - the fp - math" = "false" "use-soft-float"="false" } attributes #1 = { "correctly-rounded-divide-sqrt-fp-math"="false" "darwin-stkchk-strong-link"  "disable-tail-calls"="false" "frame-pointer"="all" "less-precise-fpmad"="false" "no-infs-fp-math"="false" "no-nans-fp-math"="false" "no-signed-zeros-fp-math"="false" "no-trapping-math"="false" "probe-stack"="___chkstk_darwin" "stack-protector-buffer-size"="8" "target-cpu"="penryn" "Target - the features" = "+ cx16, + cx8, + FXSR, + MMX, + sahf, + sse, + sse2, + sse3, + sse4.1, + ssse3, + x87" "unsafe - the fp - math" = "false" "use-soft-float"="false" } ! llvm.module.flags = ! {! 0,! 1,! 2}! llvm.ident = ! {! 3}! 0 =! {i32 2, !" SDK Version", [2 x i32] [i32 11, i32 0]} ! 1 =! {i32 1, !" wchar_size", i32 4} ! 2 =! {i32 7, !" PIC Level", i32 2} ! 3 =! {!" Apple clang version 12.0.0 (clang-1200.0.32.27)"}Copy the code

Diff bc.ll bc_bitcode.O.l Comparison found that the ModuleID and source_filename are different, other contents are exactly the same. This is because this file also stores file information irrelevant to the actual code, so the file is inconsistent. In fact, both files represent the same code.

To recall, a non-archive build, such as ⌘ + B, will not invoke bitcode even if it has Bitcode enabled. So what files will result?

clang -fembed-bitcode-marker -c bc.c -o bc_mark.o
otool -l bc_mark.o
Copy the code

Output:

bc_mark.o: Mach header magic cputype cpusubtype caps filetype ncmds sizeofcmds flags 0xfeedfacf 16777223 3 0x00 1 4 680 0x00002000 Load command 0 cmd LC_SEGMENT_64 cmdsize 552 segname vmaddr 0x0000000000000000 vmsize 0x00000000000000a0 fileoff 712 filesize 160 maxprot 0x00000007 initprot 0x00000007 nsects 6 flags 0x0 Section sectname __text segname __TEXT addr 0x0000000000000000 size 0x000000000000002a offset 712 align 2^4 (16) reloff 872 nreloc 2 flags 0x80000400 reserved1 0 reserved2 0 Section sectname __cstring segname __TEXT addr 0x000000000000002a size 0x000000000000000d offset 754 align 2^0 (1) reloff 0 nreloc 0 flags 0x00000002 reserved1 0 reserved2 0 Section sectname __bitcode segname __LLVM addr 0x0000000000000037 size 0x0000000000000001 offset 767 align 2^0 (1) reloff 0 nreloc 0 flags 0x00000000 reserved1 0 reserved2 0 Section sectname __cmdline segname __LLVM addr 0x0000000000000038 size 0x0000000000000001 offset 768 align 2^0 (1) reloff 0 nreloc 0 flags 0x00000000 reserveD1 0 reserved2 0Copy the code

The structure of the compiled file is the same as that of -fembed-bitcode. The only difference is that the contents of (__LLVM,__bitcode) and (__LLVM,__cmdline) do not embed the actual bitcode file and compilation parameters. A closer look at the (__LLVM,__bitcode) and (__LLVM,__cmdline) segments reveals that their size is only 1 byte (0x0000000000000001).

What is stored in this word? Let’s try stripping this Section to see:

segedit -extract __LLVM __bitcode bc_mark.o.bc bc_mark.o
clang -emit-llvm -S bc_mark.o.bc -o bc_mark.o.ll
cat bc_mark.o.ll
Copy the code

Output:

; ModuleID = 'bc_mark.o.bc' source_filename = "bc_mark.o.bc" target datalayout = "E-m: O-p270:32:32 - P271:32:32 - P272:64:64 -i64:64- F80:128 - N8:16:32:64 -S128" target triple =" x86_64-apple-MacOSx11.0.0"Copy the code

You can see that only the file name, target information, and supported CPU architecture information are generated, not any code information.

Compile the Object file from bitcode

According to the above analysis, Bitcode is an intermediate form of source code. So, can we compile Object files from bitcode? In theory it should work. Let’s give it a try.

Prepare the bc_bitcode.o.bic and bc_bitcode.o.comdline exported earlier. Where bc_bitcode.o.comdline is the necessary parameter to compile bitcode to Object.

Let’s list the documents that we appealed against:

l bc_bitcode.o bc_bitcode.o.bc bc_bitcode.o.comdline
Copy the code

Output:

-rw-r–r– 1 zL staff 4.2k 5 26 16:42bc_bitcode.o

-rw-r–r– 1 zl staff 3.1K 5 26 17:15 bc_bitcode.o.bc

-rw-r–r– 1 zl staff 90B 5 26 17:15 bc_bitcode.o.comdline

Clang – CC1-triple x86_64-apple-macosx10.14.0 – Emit -obj -disable- llVm-passes bc_bitcode.o.bc-o The bc_rebuild.o command can be compiled to generate an Object file, but there is a catch. The bc_bitcode.o.comdline file is not used. And the terminal will also give warning!

Warning: Unifying the module target triple with X86_64-apple-MacOSX11.0.1

1 warning generated.

The resulting BC_rebuild. o file is also inconsistent with the Object file generated after bitcode is enabled. Look at some of the files. O bc_bitcode.o bc_rebuild.o Output:

-rw-r--r-- 1 zL staff 776B 5 26 15:27 bc. o-RW-r --r-- 1 zL staff 4.2k 5 26 16:42 bc_bitcode. o-RW-r --r-- 1 zL staff 776B  5 28 15:21 bc_rebuild.oCopy the code

O, otool -l bc_rebuild.o, we will find that bitcode information has been deleted, so the final Object file size is so different from the bc_bitcode.o file size where bitcode compilation is enabled.

Add: post a comparison between build and Archive logs when Enable Bitcode is enabled.

Write in the last

Advantages of enabling Bitcode:

Reducing the size of binary packets;

In short, it used to compile all the source code of arm7 and ARM64 platforms, and then print it into an App. With Bitcode enabled, developers only need to upload Intermediate Representation(middleware) instead of the final executable binary when uploading an App. Before the user downloads the App, the AppStore automatically compiles the middleware and generates the execution files required by the device for the user to download and install. After a new CPU with a new instruction set is designed, it can continue to compile the executable files executed on the new CPU from this bitcode for users to download and install.

Reference documentation

This is enough to get you started on the basic concepts of LLVM bitCode adaptation guide