0. Guide language
Whether “QQ Music” or “National K Song”, its Android client is now a huge application with various functions, huge volume and more than 100,000 methods. The huge volume of the project brings a prominent problem of construction engineering: construction time is too long. Time issues affect both local development and continuous integration on the server, and as product functionality continues to iterate, the volume of applications is bound to rise further, resulting in a longer and longer full engineering build. Gradle daemons, incremental builds, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons, Gradle daemons Examples include LayoutCast, FreeLine, Instant Run, and Buck.
In general, Layout Cast and Instant Run are similar in that they both generate differential build packages and make them take effect at Run time. The difference is mainly in the implementation of the two. Layout Cast inserts differentiated code by reflection insertion dex, which is the same as many plug-in and patch package mechanisms. As for The Instant Run launched by Google, it inserts differentiated code by adding inserted code in the constructor of each class. (There are also some domestic development teams that use the idea of Instant Run for their hot patch solutions. For example, Meituan’s hot patch solution is close to the idea of Instant Run. However, this scheme is more intrusive to the project, and the cost is relatively high. Every class constructor needs to check the patch marker bit once. Although currently, both solutions have some drawbacks, such as API version limitations, dex limitations, or bugs that don’t take effect after modifying resources, But the incremental build approach greatly speeds up our debugging in most cases, and we can expect Google to address these issues in future versions of Instant Run. Unfortunately, neither of these approaches inherently speeds up builds, so neither of them provides a speed boost when we need a full build project.
FreeLine is an accelerated construction tool developed and open source by Ant Financial. Its core idea is the same as Buck, that is, it adopts multi-task and concurrent construction method, extracts and uses Buck’s DX, and replaces the original dex generation tool with DexMerge component tool to accelerate full construction. FreeLine also has an incremental build strategy similar to Instant Run and Layout Cast, with the disadvantages of not having enough applications to prove its reliability, and the subsequent maintenance of the tool is unclear.
The Buck build tool, the focus of this article, is nothing new. It is a powerful build tool developed, maintained and open sourced by Facebook. It is not only widely used in the full range of Facebook products, but also used by the wechat team in China. Its construction of the object code is quite extensive, and the Android project has been optimized, the core idea is the multi-task concurrent construction strategy, give full play to the advantages of multi-core. Compared with The Gradle build tool, its biggest advantage is that it can greatly speed up the full build of Android projects. It is the best choice in the current full build strategy of Android. This article is not intended to discuss the pros and cons of a modular architecture, but as far as Buck is concerned, a modular architecture that is as fine-grained as possible takes advantage of its “concurrent build” advantages.
1. Traditional Android engineering process
Building an Android project is a fairly complex process. The complexity is not only due to the complex steps and task dependencies of the build process itself, but also due to the fragmentation of the Android platform. Only one version of the code does not represent all versions of the build process. This is just a brief introduction to the dependencies between the main steps of a traditional Android engineering build, and the important cache files generated during the build.
The traditional construction method is understood here as Google’s Gradle script-based plug-in com.android.application and com.android.library as building tools of Android project. The difference between the two is that one is targeted at the main project. One for Module.
In the. Gradle script of the main project, access
apply plugin: 'com.android.application'Copy the codeIn Module, access
apply plugin: 'com.android.library'Copy the codeRead the source code, can be seen in the process of the Android engineering, the concrete execution of what tasks, the core task in groovy/com/Android/build/gradle/tasks, mainly includes:
Dex. Groovy / / -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- generated Dex file AidlCompile. Groovy / / -- -- -- -- -- -- -- -- -- -- -- -- -- -- to compile Aidl library GenerateBuildConfig. Groovy / / -- -- -- -- -- - analysis of compiler configuration Lint. Groovy / / -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - was scanned Lint MergeAsserts. Groovy / / -- -- -- -- -- -- -- -- -- -- -- -- -- integrating Asserts directory of resources MergeResources. Groovy / / -- -- -- -- -- -- -- -- -- -- - integrating resource file NdkCompile. Groovy / / -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- to compile the NDK libraries PackageApplication. Groovy / / -- -- -- -- -- -- -- packaged App PreDex. Groovy / / -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- generated Dex preparations ProcessAndroidResources. Groovy / / - processing resource files ProcessAppManifest. Groovy / / -- -- -- -- -- -- -- to deal with the Manifest file ZipAlign. Groovy / / -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- compression and its operationCopy the codeThese tasks eventually generate shell directives that call the build tools in the [Android SDK Home]/build-tools/[Build Tools version] directory.
Ignoring obfuscations, compilation and configuration, alignment, compression, signing, and other tasks that we don’t care about, we analyze the main process of building Gradle tools:
1. The resource file needs to be compiled first:
2. Then compile classes that depend on resource files:
3. Then compile classes that do not depend on resources:
4. The compiler then begins to consolidate.class files into dex files:
5. Finally, combine the compiled resource files to form an. Apk file
When building Gradle tools, you can use the –profile option to output a detailed report of build time in the [project floder]/build/ Report directory. This report allows you to check which Task takes the most time. Basically, the most time-consuming step is the dex generation step, mainly because the code file is too large. Because Gradle tool (Gradle 3.1) does not support concurrent multi-task building, and as mentioned above, the generation of Dex file is essentially realized by calling the Dex script of the Android SDK, so only from the perspective of speeding up Gradle building, to improve the speed of building. It’s hard to have a noticeable effect.
The Buck tool works from the following two aspects: first, it supports concurrent construction of multiple tasks. Each module generates an independent DEX file. Finally, it merges multiple independent dex files into one through dex Merge. The second is to redevelop dx and DexMerge components. According to the official document from Buck, the time complexity of Google’s native DEX script is O(N^2), while the time complexity of the improved component is only O(NlogN). According to the test data provided by the Freeline team, Buck’s DX component is about 40% faster than its native component.
2. Install the Buck tool
Before installing the Buck, make sure that the following support tools are properly installed on your computer:
-
Git
-
JDK 7+
-
Ant 1.8 +
-
Python 2.7 +
If you are building an Android project, you also need to install the Android SDK and NDK.
Domestic some articles about Buck is generally believed that the only can run on the Mac OS and Linux systems, but I’m in the website buckbuild.com/about/overv… Did not find this description, try to run on Windows system, also available, I used buck version:
>buck --version
buck version 97cdd2a490868a9dcf40148d8421ed27cf720410Copy the codeUnfortunately, watchman, one of Buck’s key enablers, doesn’t run on Windows yet. But even without a watchman doesn’t hurt, does not affect the normal operation of the Buck, but from the watchman’s website: facebook. Making. IO/watchman, you can see the development team is developing adaptation versions of Windows.
Instead of watchman, you can install Buck with git:
git clone https://github.com/facebook/buck.gitCopy the codeOnce the download is complete, you can check the correct installation by using the help command in the buck directory:
>buck --help
buck build tool
usage:
buck [options]
buck command --helpCopy the code3. A simple Android project using Buck
First, we need to create a new Android project. A buck-style project directory structure looks like this:
>Project Root -. Buckconfig // Build Project configuration -java // code directory -buck //BUCK script -com-tencent - xxx-res // resource directory -buck //BUCK script -layout-values -drawable -lib // third-party application directory -apps // Project directory -androidmanifest.xml -buck //BUCK script -debug. keystore / / the debug packet signature file - the debug. Keystore. The properties / / debug signature file configuration fileCopy the code3.1 buckconfig file
The. Buckconfig file is located in the root directory of the project and mainly configures the overall properties of the project. For example, the content of the file can be:
[java]
src_roots = /java/
[project]
default_android_manifest = //app/AndroidManifest.xml
[android]
target = Google Inc.:Google APIs:23
[alias]
app = //apps:appCopy the codeDetailed explanations of each parameter can be found on the official website. Here is a brief explanation.
The [Java] parameter specifies the source path of the project. The source path configured here is/Java /. In all buck scripts, slash/indicates the same path as the current script, and double slash // indicates the root directory of the current project.
The [project] parameter specifies some of the project’s core configuration items, such as the path to the project’s Androidmanifest.xml file.
The [Android] parameter specifies some information about the Version of Android the project is running on, such as the Target API=23 specified here.
The [alias] parameter represents the alias of the build project, as configured here:
[alias]
app = //apps:appCopy the codeIn this project, we set an alias :app for the //apps:app Buck task. So build or install an Android project with Buck in this project, using:
>buck build app
>buck install appCopy the codeIt has the same effect as the following statement:
>buck build //apps:app
>buck install //apps:appCopy the code3.2 BUCK Files and BUCK rules
In the above directory structure, you can see that a project can have multiple BUCK files, and each BUCK file consists of a single BUCK Rule. There are many kinds of BUCK rules, including compiling source code, compiling AAR package, compiling NDK, compiling AIDL, compiling resources, integrating packages, signing files, and so on. For detailed explanation, please refer to the explanation on the official website. Here, take the “BUCK” file of //apps/BUCK as an example. The content of the file is as follows:
android_binary(
name = 'app',
manifest = 'AndroidManifest.xml',
keystore = ':debug_keystore',
deps = [
'//java:activity',
],
)
keystore(
name = 'debug_keystore',
store = 'debug.keystore',
properties = 'debug.keystore.properties',
)
project_config(
src_target = ':app',
)Copy the codeThe BUCK script is written in Python, so there is no need to go over Python syntax here, but it is important to pay attention to the indentation of each line.
Start with the first Buck Rule: Android_binary. This Rule represents the build goal of an Android project, which is to produce an.apk file. It contains attributes such as name, manifest, and keystore whose meaning is obvious, while the deps attribute indicates that this Rule depends on the completion of other rules. As mentioned earlier, the double slash // indicates the root directory of the project. For simplicity’s sake, you don’t need to specify a BUCK file, while the colon: indicates a Rule in the BUCK file. Therefore, from the // Java :activity attribute, you can see that the execution of the android_binary Rule, The activity Rule that relies on the activity Rule in [Project Root]/ Java /BUCK completes first.
[Project Root]/ Java /BUCK: For this activity Rule, we need to specify the keystore Rule manually because android-Gradle automatically adds a default signature when typing Debug packages. BUCK does not. Sign the debug configuration file. The keystore. The properties are as follows:
key.alias=my_alias
key.store.password=android
key.alias.password=androidCopy the codeWithout going over the implications of these configurations, let’s jump right to the last Rule:project_config, whose main job is to give our build project a name. Configuration items like Gradle’s buildType can be used to implement things like multi-channel packaging. Combined with the [alias] parameter in the previous section, we designed the alias app for this Rule.
[Project Root]/ Java /BUCK [Project Root]/ Java /BUCK]
android_library(
name = 'activity',
srcs = glob(['*.java']),
deps = [
'//res:res',
],
visibility = [ 'PUBLIC' ],
)
project_config(
src_target = ':activity',
)Copy the codeAs you can see from the last BUCK file, this file no longer contains android_binary and instead uses the android_library Rule, This is because a build type can contain only one Android_binary, whereas android_library can have multiple. Given the architecture of the Android project, android_binary is equivalent to the main project, while android_library corresponds to multiple modules. As mentioned earlier, the Buck tool encourages a modular architecture of sufficient granularity, with each module corresponding to an Android_library, to take full advantage of concurrent builds. As for the meaning of the parameters in this document, you can find the authoritative explanation on the official website, which will not be repeated here.
4. Why does Buck speed builds
The Buck tool generates three important files at different stages of construction: r.txt,.jar, and.apk, corresponding to three types of Rule: android_resource, android_library, and android_binary. In modules, each module would have an R.teddy and a.jar(if the module was unrelated to the UI element, there would be no R.teddy file), and Buck eventually aggregated the R.teddy files into r.Ava files and the.jar files into dex files.
Analyzing the Buck tool build process, you can see:
1. First, it will concurrently start compiling resources for multiple modules:
The contents of the r.xt file are as follows:
int anim fade_in_seq 0x7f04000b
int attr actionBarSize 0x7f0100af
int color fbui_bg_dark 0x7f060071
int dimen title_bar_height 0x7f07000f
int drawable map_placeholder 0x7f0204ca
int id embeddable_map_frame 0x7f0a00e8
int layout splash_screen 0x7f030172
int string add_members_button_text 0x7f0900f8
int[] styleable ThreadTitleView { 0x7f01018a }Copy the codeUnlike R.java, the descriptor for the resource property is not static final Int but int, because in the last step we need to aggregate all r.tx files into one R.java, which may need to change the conflicting resource ids.
2. After that, Buck compiled the source files of each module and generated the dex file:
3. Finally, merge resource files and dex files respectively to generate APK in packaging:
BuckThe tool construction is complete, and the unchanged Module will be used directly when we modify the existing logicThe cacheData, which also greatly increases the speed at which we build projects. In this sectionBuckFrom the analysis of the tool construction process, it can be seen that only the modular structure with sufficient particle size can be fully utilizedBuckMultitasking concurrency and the benefits of caching.
5. Practice of accessing Buck tool in the National Karaoke Project
In version 3.7, The National Karaoke Project tried to access Buck tool. In order to ensure the stability of the Internet version, Buck tool was only used for local debugging to speed up the full construction. The intrusion to the project is mainly reflected in the following aspects:
-
Buck does not support remote access to maven libraries to download third-party dependencies. You need to manually download them and add them to the buck-libs directory. When Buck is compiled, the dependency files for this directory will be included
-
Buck does not support switch(view.getid ()). Use if-else instead.
-
The buildConfig. Java file generated by Gradle compilation needs to be manually copied and placed in a specified location to be included when Buck builds.
-
The Buck does not automatically sign the Debug package. You need to manually configure the signature file.
-
Windows does not support case-sensitive file names. Therefore, resource files, code files, and third-party dependent libraries cannot be case-sensitive files.
Compare the time spent on a full build with Buck and Gradle:
Hardware environment: Windows7 sp1(64bit),Intel i7-4790,16GB RAMCopy the codeAs you can see, the build speed of The K-Song project is about 40% faster with the addition of the Buck tool, but how much faster a project can be built with the addition of the Buck tool depends on the modular partitioning strategy of the project.
6. Summary
The Buck tool is a build tool that effectively speeds up full builds, encouraging developers to divide projects into smaller, granular blocks as much as possible to maximize the benefits of concurrent builds. It was developed by the Facebook team and has been proven to be reliable and stable for large volume applications, without requiring major modifications to existing projects. All in all, it’s a good build acceleration strategy to try. The above is my personal understanding, there may be mistakes or mistakes, welcome to correct communication.