The company’s project code is quite large, and it takes nearly 2 minutes to run after debugging and changing Java files, which is really unbearable. I found a lot of configuration parameters on the Internet, but there was no obvious effect. I tried using Instant Run, and the effect was not good. Then I tried using Freeline, and the compilation speed was ok but unstable.

Project address: github.com/typ0520/fas…

Note: The instructions for Gradle Task in this article are based on the premise that Instant Run is turned off
Note: All of the code in this article, gradle task names, and task output paths are explained using the debug buildType

To optimize the build speed, first of all, you need to find out which parts are causing the slow build speed. Put the following code into app/build.gradle and print out any tasks that take more than 50ms

 public class BuildTimeListener implements TaskExecutionListener, BuildListener {
    private Clock clock
    private times = []

    @Override
    void beforeExecute(Task task) {
        clock = new org.gradle.util.Clock()
    }

    @Override
    void afterExecute(Task task, TaskState taskState) {
        def ms = clock.timeInMs
        times.add([ms, task.path])

        //task.project.logger.warn "${task.path} spend ${ms}ms"
    }

    @Override
    void buildFinished(BuildResult result) {
        println "Task spend time:"
        for (time in times) {
            if (time[0] >= 50) {
                printf "%7sms %s\n", time
            }
        }
    }

    ......
}

project.gradle.addListener(new BuildTimeListener())Copy the code

Execute./gradlew assembleDebug and, after a long wait, get the following output

Total time: 1 mins 39.566 secs
Task spend time:
     69ms  :app:prepareComAndroidSupportAnimatedVectorDrawable2340Library
    448ms  :app:prepareComAndroidSupportAppcompatV72340Library
     57ms  :app:prepareComAndroidSupportDesign2340Library
     55ms  :app:prepareComAndroidSupportSupportV42340Library
     84ms  :app:prepareComFacebookFrescoImagepipeline110Library
     69ms  :app:prepareComSquareupLeakcanaryLeakcanaryAndroid14Beta2Library
     60ms  :app:prepareOrgXutilsXutils3336Library
     68ms  :app:compileDebugRenderscript
    265ms  :app:processDebugManifest
   1517ms  :app:mergeDebugResources
    766ms  :app:processDebugResources
   2897ms  :app:compileDebugJavaWithJavac
   3117ms  :app:transformClassesWithJarMergingForDebug
   7899ms  :app:transformClassesWithMultidexlistForDebug
  65327ms  :app:transformClassesWithDexForDebug
    151ms  :app:transformNative_libsWithMergeJniLibsForDebug
    442ms  :app:transformResourcesWithMergeJavaResForDebug
   2616ms  :app:packageDebug
    123ms  :app:zipalignDebugCopy the code

From the above output can be found that the total build time for 100 seconds (the above output is not in accordance with the real execution sequence output), transformClassesWithDexForDebug task is the slowest took 65 seconds, it is the task we need to focus on optimizing, First, let’s talk about the role of the main tasks in the build process to understand the following hook points

MergeDebugResources task’s role is to extract all the aar package output to the app/build/intermediates/exploded – the aar, And combine all the resource file to the app/build/intermediates/res/merged/debug directory

The processDebugManifest task is to merge all the nodes in the ANDROIDmanifest.xml aar package into the project’s Androidmanifest.xml. And replace the placeholder in the manifest file with the manifestplaceholder configuration of the current buildType in app/ built. gradle, The final output to the app/build/intermediates/manifests/full/debug/AndroidManifest. XML

The role of processDebugResources

  • 1, call aapt generates all aar depend on the project and R.j ava, output to the app/build/generated/source/r/debug directory
  • 2, generate resources index file app/build/intermediates/res/resources – debug. Ap_
  • 3, the symbol table output to the app/build/intermediates/symbols/debug/R.t xt

CompileDebugJavaWithJavac this task is used to compile the Java file into the class file, the output path is app/build/intermediates/classes/debug directory has compiled input

  • 1, the project source directory, the default path is app/ SRC /main/ Java, can be configured by sourceSets DSL, Allow multiple (print project. Android. SourceSets. Main. Java srcDirs can view the source code of all current path, specific configuration can be reference for android – doc
  • 2, app/build/generated/source/aidl
  • 3, app/build/generated/source/buildConfig
  • 4, app/build/generated/source/apt (inherit javax.mail. The annotation. Processing. AbstractProcessor do some dynamic code generation library, output in this directory, See the code for Butterknife and Tinker

TransformClassesWithJarMergingForDebug the role of the output of the task is to put the compileDebugJavaWithJavac app/build/intermediates/classes/debug, And app/build/intermediates/exploded – all classes in the aar. The jar in the jar and libs packages as input, Combined to output to the app/build/intermediates/transforms/jarMerging/debug/jars/f / 1/1 combined. The jar, We sometimes report duplicate Entry: XXX error when we rely on third-party libraries in development because we found classes in the same path in different JARS during merging

TransformClassesWithMultidexlistForDebug the task time is also very long nearly 8 seconds, it has two functions

  • 1. Scan the dependencies between the project’s Androidmanifest.xml file and the analysis class. Calculate those classes must be placed inside the first dex, finally the results of the analysis to app/build/intermediates/multi – dex/debug/maindexlist. TXT file inside
  • 2, generate confusion configuration item is output to the app/build/intermediates/multi – dex/debug/manifest_keep. TXT file

The code entry in the project is the application node attribute android.name configured from the application class. Before Android 5.0, the system only loads a dex(classes.dex). classes2.dex ……. ClassesN. Dex in general is the use of the android. Support. Multidex. Multidex loaded, so if the entrance to the Application class is not in the classes. In 5.0 the following dex will hang up, In addition, when the entry Application depends on a class that is not classes.dex, it will fail because the class cannot be found when it is initialized, and it will fail because the class name is changed when it is confused

TransformClassesWithDexForDebug this task is to put the jar package containing all of the class files into dex, Class file the conversion of the slower input jar package path is app/build/intermediates/transforms/jarMerging/debug/jars/f / 1/1 combined. The jar The output directory of dex is build/intermediates/transforms/dex 1000/1 / debug/folders/f/main

* Note that if you need to use the above paths to write gradle plugins, it is best to obtain the paths from the Android Gradle API to prevent future changes

Combined with the above the information focus needs to optimize transformClassesWithDexForDebug this task, my train of thought is the first time that the whole quantity packaging after execution transformClassesWithDexForDebug task generated dex cache down, Before performing this task, take a snapshot of all the current Java source files. In the future patch packaging, compare the information of all the current Java files with the previous snapshot to find out the changed Java files and then find out which class files have changed. Then the app/build/intermediates/transforms/jarMerging/debug/jars/f / 1/1 combined. There is no change in the jar class remove, just send the change class generated dex, And then I’m going to pick a hot fix and I’m going to load this dex as a patch and I’m going to think about it and then I’m going to work on all the technical points

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

# # # # to get transformClassesWithDexForDebug before and after the mission from the Tinker project life cycle of the reference code, find the following implementation

public class ImmutableDexTransform extends Transform { Project project DexTransform dexTransform def variant ...... @Override void transform(TransformInvocation transformInvocation) throws TransformException, IOException, InterruptedException {def outputProvider = transformInvocation. GetOutputProvider () / / dex output directory File outputDir = outputProvider.getContentLocation("main", dexTransform.getOutputTypes(), dexTransform.getScopes(), Format.DIRECTORY);
        if (outputDir.exists()) {
            outputDir.delete()
        }
        println("=== Clear the dex output directory before performing transform:${project.projectDir.toPath().relativize(outputDir.toPath())}")
        dexTransform.transform(transformInvocation)
        if (outputDir.exists()) {
            println("=== after executing transform dex output directory is not empty:${project.projectDir.toPath().relativize(outputDir.toPath())}")
            outputDir.listFiles().each {
                println("=== after executing transform:${it.name}")
            }
        }
    }
}

project.getGradle().getTaskGraph().addTaskExecutionGraphListener(new TaskExecutionGraphListener() {
    @Override
    public void graphPopulated(TaskExecutionGraph taskGraph) {
        for (Task task : taskGraph.getAllTasks()) {
            if (task instanceof TransformTask && task.name.toLowerCase().contains(variant.name.toLowerCase())) {

                if(((TransformTask) task).getTransform() instanceof DexTransform && ! (((TransformTask) task).getTransform() instanceof ImmutableDexTransform)) { project.logger.warn("find dex transform. transform class: " + task.transform.getClass() + " . task name: " + task.name)

                    DexTransform dexTransform = task.transform
                    ImmutableDexTransform hookDexTransform = new ImmutableDexTransform(project,
                            variant, dexTransform)
                    project.logger.info("variant name: " + variant.name)

                    Field field = TransformTask.class.getDeclaredField("transform")
                    field.setAccessible(true)
                    field.set(task, hookDexTransform)
                    project.logger.warn("transform class after hook: " + task.transform.getClass())
                    break; }}}}});Copy the code

Run the above code in app/build.gradle./gradlew assembleDebug

:app:transformClassesWithMultidexlistForDebug ProGuard, Version 5.2.1 Reading Program JAR [/Users/tong/Projects/fastdex/app/build/intermediates/transforms/jarMerging/debug/jars/1/1f/combined.jar] Reading Library jar [/ Users/tong/Applications/android SDK - macosx/build - the tools / 23.0.1 / lib/shrinkedAndroid jar] Preparing the output jar [/Users/tong/Projects/fastdex/app/build/intermediates/multi-dex/debug/componentClasses.jar] Copying resources from program jar [/Users/tong/Projects/fastdex/app/build/intermediates/transforms/jarMerging/debug/jars/1/1f/combined.jar] App: transformClassesWithDexForDebug = = = before performing the transform to empty dex output directory: build/intermediates/transforms/dex/debug/folders/1000/1f/main ...... = = = after performing the transform dex output directory is not empty: build/intermediates/transforms/dex/debug/folders / 1000/1 / f main = = = : after performing the transform classes. DexCopy the code

The above log output proves that this hook point is effective. Before the transform is executed in full packaging, you can make a snapshot of Java source code, and cache dex after the execution. Before patch package and transform, compare the snapshot to remove the unchanged class. After the transform is executed, merge the cached dex and put it into the dex output directory

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

#### How to take a snapshot and compare a snapshot and get a list of changed classes execute the following code to get all the project source directory

project.android.sourceSets.main.java.srcDirs.each { srcDir->
    println("==srcDir: ${srcDir}")}Copy the code

The sample project is not configured with sourceSets, so the output is app/ SRC /main/ Java

Copy all Java files from the srcDir directory to the snapshot directory by copying files directly. (There is a pit here. Do not use project.copy {}. Use stream copy directly and overwrite the lastModified of the destination file with the lastModified of the source file.

Through the length of the Java files and comparison of two basic elements of the last modification time can know whether the same file changes, through the snapshot directory in the current directory has a file without a file that increased the files, through the snapshot directory has a file but not in the current directory to delete the file (to efficiency can not delete, There are only some classes in the cache that are not needed. For example, if the project source code path for/Users/tong/fastdex/app/SRC/main/Java, when doing a snapshot copy the directory to/Users/tong/fastdex/app/build/fastdex snapshoot, The file tree in the current snapshot is

├── Java ├─ Java ├─ Java ├─ Java ├─ Java ├─ JavaCopy the code

If the contents of the current source path change, the current file tree is

└── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── ── SampleApplication.javaCopy the code

A list of relative paths for this change can be obtained by comparing the file traversal

  • com/dx168/fastdex/sample/MainActivity.java
  • com/dx168/fastdex/sample/New.java

From this list, you can see that the changed classes are

  • com/dx168/fastdex/sample/MainActivity.class
  • com/dx168/fastdex/sample/New.class

However, Java files with inner classes will also have some other class output, such as R file to compile, its compilation output is as follows

➜  sample git:(master) ls
R.java
➜  sample git:(master) javac R.java 
➜  sample git:(master) ls
R$attr.class      R$dimen.class     R$id.class        R$layout.class    R$string.class    R$styleable.class R.java
R$color.class     R$drawable.class  R$integer.class   R$mipmap.class    R$style.class r.lass ➜ sample git:(master)Copy the code

If you use the butterknife, also generates the binder class, such as compiling MainActivity. Java generated when the com/dx168 fastdex/sample/MainActivity? ViewBinder.class

Combine the above points to get a matching pattern for all the changing classes

  • com/dx168/fastdex/sample/MainActivity.class
  • com/dx168/fastdex/sample/MainActivity$*.class
  • com/dx168/fastdex/sample/New.class
  • com/dx168/fastdex/sample/New$*.class

With the above patterns can before the patch package execution transform the app/build/intermediates/transforms/jarMerging/debug/jars/f / 1/1 combined. Remove class all of no change in the jar

project.copy {
    from project.zipTree(combinedJar)
        for (String pattern : patterns) {
            include pattern
        }
    }
    into tmpDir
}
project.ant.zip(baseDir: tmpDir, destFile: patchJar)Copy the code

You can then use patchJar as the input JAR to generate the patch dex

Note: This mapping scheme will not work if obfuscation is enabled and will need to parse the mapping file generated after obfuscation, but we don’t need to develop and debug in buildType with obfuscation enabled, so we don’t need to do this for now

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = with dex patch, can choose a kind of hot fix the patch load in dex, here there are several kinds of, In order to select android. The simple and direct support. Multidex. Multidex with dex pile to load, just need to put the dex according to Google’s standard (classes. Dex, classes2. Dex, classesN. Dex) the arrangement is good, There are two technical points here

Because there are duplicate classes in patch.dex and cached dex, pre-Verify errors will be caused when loading classes that reference duplicate classes. For details, please refer to the introduction of Hot patch dynamic repair technology for Android App written by qzone team. The solution given in the article is to insert a code into all the classes that reference the repaired class, and the dex of the class where the code is inserted must be a separate dex, which we have prepared in advance, called fastdex-Runtime. dex, and its code structure is

├── unbreakable, unbreakable, unbreakable, unbreakable, unbreakable, unbreakable, unbreakable, unbreakable, unbreakable, unbreakable, unbreakable, unbreakable, unbreakable, unbreakable, unbreakable Multidex ├ ─ ─ multidex. Java ├ ─ ─ MultiDexApplication. Java ├ ─ ─ MultiDexExtractor. Java └ ─ ─ ZipUtil. JavaCopy the code

AntilazyLoad. Java is the package used to load classes2.dex-classesn. dex, in case the project does not rely on MultiDex. So copy the MultiDex code into our package fastdexApplication.java

To meet the needs of our project before full quantity packaging app/build/intermediates/transforms/jarMerging/debug/jars/f / 1/1 combined. The jar all project code in the class all dynamic insert code (third party libraries due to no Within the scope of our repair so to ignore efficiency), the specific approach is to add to all the construction method of com. Dx168. Fastdex. Runtime. Antilazyload. Antilazyload dependence, shown in the code below

//source class:
public class MainActivity {
}

==>

//dest class:
import com.dx168.fastdex.runtime.antilazyload.AntilazyLoad;
public class MainActivity {
    public MainActivity() { System.out.println(Antilazyload.str); }}Copy the code

Dynamically insert code into the class file using ASM, I did the test found some relevant information and code are put on Github above click ME to view, the code is more only posted part, please see ClassInject. Groovy

private static class MyClassVisitor extends ClassVisitor { public MyClassVisitor(ClassVisitor classVisitor) { super(Opcodes.ASM5, classVisitor); } @Override public MethodVisitor visitMethod(int access, String name, String desc, String signature, String[] exceptions) {// Check if it is a constructorif ("<init>".equals(name)) {
            MethodVisitor mv = cv.visitMethod(access, name, desc, signature, exceptions);
            MethodVisitor newMethod = new AsmMethodVisit(mv);
            return newMethod;
        } else {
            return super.visitMethod(access, name, desc, signature, exceptions);
        }
    }
}

static class AsmMethodVisit extends MethodVisitor {
    public AsmMethodVisit(MethodVisitor mv) {
        super(Opcodes.ASM5, mv);
    }

    @Override
    public void visitInsn(int opcode) {
        if(opcode == opcodes.return) {// Access the static constant of Java /lang/System out mv.visitFieldinsn (GETSTATIC,"java/lang/System"."out"."Ljava/io/PrintStream;"); // Access the AntilazyLoad static variable mv.visitFieldinsn (GETSTATIC,"com/dx168/fastdex/runtime/antilazyload/AntilazyLoad"."str"."Ljava/lang/String;"); // Call out println to print the value of mv.visitmethodinsn (INVOKEVIRTUAL,"java/io/PrintStream"."println"."(Ljava/lang/String;) V".false); } super.visitInsn(opcode); }}Copy the code

If there are two (class.dex classes2.dex) in the cache, there are two (class.dex classes2.dex) in the cache. The order of the merged dex is fastdex-Runtime. dex, patch.dex, classes2.dex (patch.dex must be placed before the cached dex to be repaired).

fastdex-runtime.dex  => classes.dex
patch.dex            => classes2.dex
classes.dex          => classes3.dex
classes2.dex         => classes4.dexCopy the code

In explaining transformClassesWithMultidexlistForDebug task has said that the program entrance Application problems, if the patch. Do not include the entrance Application in dex, apk startup will surely quote class can’t find the mistake, So how to solve this problem

    1. The first plan:

      thetransformClassesWithMultidexlistForDebugAll classes in the maindexlist. TXT output in this task are involved in patch.dex generation
    1. The second option is to proxy the entry of the project, Application, and put the proxy class in the first dex, followed by the dex of the project in sequence

In the first scheme, a large number of classes in maindexlist. TXT must be involved in the generation of patches, which conflicts with the previous idea of minimizing the participation of class files in the generation of dex. Compared with the second scheme, the efficiency is lower. Another reason is that there is no guarantee that MultiDex is used in the project’s Application;

The second solution does not have the above problems, but can cause problems if the project code is strengthened with getApplication() (see issue#2). Instant run also has the same problems. It does this by restoring the Application during the hook system API runtime. So hard turns should not be a problem, see monkeypatcher. Java (need to be turned over the wall to open, if you can’t see the fastdexApplication. Java monkeyPatchApplication method)

The following is the code for the agent Application in Fastdex-Runtime. dex

public class FastdexApplication extends Application {
    public static final String LOG_TAG = "Fastdex"; private Application realApplication; Private String getOriginApplicationName(Context Context) {ApplicationInfo; // Get the real project Application class from the manifest file appInfo = null; try { appInfo = context.getPackageManager().getApplicationInfo(getPackageName(), PackageManager.GET_META_DATA); } catch (PackageManager.NameNotFoundException e) { e.printStackTrace(); } String msg = appInfo.metaData.getString("FASTDEX_ORIGIN_APPLICATION_CLASSNAME");
        return msg;
    }

    private void createRealApplication(Context context) {
        String applicationClass = getOriginApplicationName(context);
        if(applicationClass ! = null) { Log.d(LOG_TAG, new StringBuilder().append("About to create real application of class name = ").append(applicationClass).toString());

            try {
                Class realClass = Class.forName(applicationClass);
                Constructor constructor = realClass.getConstructor(new Class[0]);
                this.realApplication = ((Application) constructor.newInstance(new Object[0]));
                Log.v(LOG_TAG, new StringBuilder().append("Created real app instance successfully :").append(this.realApplication).toString()); } catch (Exception e) { throw new IllegalStateException(e); }}else {
            this.realApplication = new Application();
        }
    }

    protected void attachBaseContext(Context context) {
        super.attachBaseContext(context);
        MultiDex.install(context);
        createRealApplication(context);

        if(this.realApplication ! = null) try { Method attachBaseContext = ContextWrapper.class .getDeclaredMethod("attachBaseContext", new Class[]{Context.class});

                attachBaseContext.setAccessible(true);
                attachBaseContext.invoke(this.realApplication, new Object[]{context});
            } catch (Exception e) {
                throw new IllegalStateException(e);
            }
    }

    public void onCreate() {
        super.onCreate();

        if(this.realApplication ! = null) { this.realApplication.onCreate(); }}... }Copy the code

According to the previous task description, the task to generate the manifest file is processDebugManifest. We only need to process the manifest file after it is executed and create a task whose implementation class is FastdexManifestTask. The core code is as follows

def ns = new Namespace("http://schemas.android.com/apk/res/android"."android")
def xml = new XmlParser().parse(new InputStreamReader(new FileInputStream(manifestPath), "utf-8"))
def application = xml.application[0]
if (application) {
    QName nameAttr = new QName("http://schemas.android.com/apk/res/android".'name'.'android');
    def applicationName = application.attribute(nameAttr)
    if (applicationName == null || applicationName.isEmpty()) {
        applicationName = "android.app.Application"} // Replace application android.name with application.attributes().put(nameAttr,"com.dx168.fastdex.runtime.FastdexApplication")
    def metaDataTags = application['meta-data'] // remove any old FASTDEX_ORIGIN_APPLICATION_CLASSNAME elements def originApplicationName = metaDataTags.findAll { it.attributes()[ns.name].equals(FASTDEX_ORIGIN_APPLICATION_CLASSNAME) }.each { it.parent().remove(it) } // Add the new FASTDEX_ORIGIN_APPLICATION_CLASSNAME element // Write the original Application to meta-data. AppendNode ('meta-data', [(ns.name): FASTDEX_ORIGIN_APPLICATION_CLASSNAME, (ns.value): applicationName])
    // Write the manifest file
    def printer = new XmlNodePrinter(new PrintWriter(manifestPath, "utf-8"))
    printer.preserveWhitespace = true
    printer.print(xml)
}
File manifestFile = new File(manifestPath)
if (manifestFile.exists()) {
    File buildDir = FastdexUtils.getBuildDir(project,variantName)
    FileUtils.copyFileUsingStream(manifestFile, new File(buildDir,MANIFEST_XML))
    project.logger.error("fastdex gen AndroidManifest.xml in ${MANIFEST_XML}")}Copy the code

Use the following code to add this task and ensure that it is executed after the processDebugManifest task has finished executing

project.afterEvaluate { android.applicationVariants.all { variant -> def variantOutput = variant.outputs.first() def VariantName = variant. The name. Capitalize () / / replace project. The Application of com dx168. Fastdex. Runtime. FastdexApplication FastdexManifestTask manifestTask = project.tasks.create("fastdexProcess${variantName}Manifest". FastdexManifestTask) manifestTask.manifestPath = variantOutput.processManifest.manifestOutputFile manifestTask.variantName = variantName manifestTask.mustRunAfter variantOutput.processManifest variantOutput.processResources.dependsOn manifestTask } }Copy the code

Processing after the manifest file android application node. The name attribute value becomes com. Dx168. Fastdex. Runtime. FastdexApplication, Write the Application name of the original project into meta-data, which is read by FastdexApplication at runtime

<meta-data android:name="FASTDEX_ORIGIN_APPLICATION_CLASSNAME" android:value="com.dx168.fastdex.sample.SampleApplication"/>Copy the code

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

##### after the development of the above functions, do the following four times of packaging for time comparison (in fact, only one time is not too accurate, do dozens of tests to take the average time so that the most accurate)

  • 1. Delete build directory for the first full package (fastdex is not enabled)

    BUILD SUCCESSFUL Total time: 1 mins 46.678 secS Task 437ms :app:prepareComAndroidSupportAppcompatV72340Library 50ms :app:prepareComAndroidSupportDesign2340Library 66ms :app:prepareComAndroidSupportSupportV42340Library 75ms :app:prepareComFacebookFrescoImagepipeline110Library 56ms :app:prepareOrgXutilsXutils3336Library 870ms :app:mergeDebugResources 93ms :app:processDebugManifest 777ms :app:processDebugResources 1200ms :app:compileDebugJavaWithJavac 3643ms :app:transformClassesWithJarMergingForDebug 5520ms :app:transformClassesWithMultidexlistForDebug 61770ms :app:transformClassesWithDexForDebug 99ms :app:transformNative_libsWithMergeJniLibsForDebug 332ms :app:transformResourcesWithMergeJavaResForDebug 2083ms :app:packageDebug 202ms :app:zipalignDebugCopy the code

  • 2. Delete build directory for first full package (enable Fastdex)

      BUILD SUCCESSFUL

    Total time: 1 mins 57.764 secs
    Task spend time:
      106ms  :app:prepareComAndroidSupportAnimatedVectorDrawable2340Library
      107ms  :runtime:transformClassesAndResourcesWithSyncLibJarsForDebug
      416ms  :app:prepareComAndroidSupportAppcompatV72340Library
       67ms  :app:prepareComAndroidSupportSupportV42340Library
       76ms  :app:prepareComFacebookFrescoImagepipeline110Library
       53ms  :app:prepareOrgXutilsXutils3336Library
      111ms  :app:processDebugManifest
      929ms  :app:mergeDebugResources
      697ms  :app:processDebugResources
     1227ms  :app:compileDebugJavaWithJavac
     3237ms  :app:transformClassesWithJarMergingForDebug
     6225ms  :app:transformClassesWithMultidexlistForDebug
    78990ms  :app:transformClassesWithDexForDebug
      122ms  :app:transformNative_libsWithMergeJniLibsForDebug
      379ms  :app:transformResourcesWithMergeJavaResForDebug
     2050ms  :app:packageDebug
       77ms  :app:zipalignDebugCopy the code

  • 3. After fastdex is enabled for the first full package, close the MainActivity. Java of the Fastdex change sample project

      BUILD SUCCESSFUL

  Total time: 1 mins 05.394 secs
  Task spend time:
     52ms  :app:mergeDebugResources
   2583ms  :app:compileDebugJavaWithJavac
  60718ms  :app:transformClassesWithDexForDebug
    101ms  :app:transformNative_libsWithMergeJniLibsForDebug
    369ms  :app:transformResourcesWithMergeJavaResForDebug
   2057ms  :app:packageDebug
     75ms  :app:zipalignDebugCopy the code

  • 4. After the fastdex full package is enabled for the first time, mainActivity. Java of the Fastdex Change Sample project is still enabled

    BUILD SUCCESSFUL Total time: 16.5 SecS Task Spend time: 142ms :app:processDebugManifest 1339ms :app:compileDebugJavaWithJavac 3291ms :app:transformClassesWithJarMergingForDebug  4865ms :app:transformClassesWithMultidexlistForDebug 1005ms :app:transformClassesWithDexForDebug 2112ms :app:packageDebug 76ms :app:zipalignDebugCopy the code
Package number The total time The transform of time
1 1 mins 46.678 s 61770 ms
2 1 mins 57.764 s 78990 ms
3 1 mins 05.394 s 60718 ms
4 16.5 s 1005 ms

According to the comparison of 1 and 2, it takes about 10 seconds longer when Fastdex is enabled for the first full packaging than when it is not enabled. This is mainly due to the overhead of the injection code and IO

By comparison of 2 and 3, it was found that the time to enable Fastdex for patch packaging was about 60 seconds faster than not to enable Fastdex. This is the long-awaited build speed

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = just excited for a while and then was a mistake of nima when modifying activity_main. XML to add a control

<TextView
    android:id="@+id/tv2"
    android:layout_width="wrap_content"
    android:layout_height="wrap_content" />Copy the code

The package crashed when it started

Caused by: java.lang.IllegalStateException: 
Required view 'end_padder' with ID 2131493007 for field 'tv1' was not found.
If this view is optional add '@Nullable' (fields) or '@Optional' (methods) annotation.
     at butterknife.internal.Finder.findRequiredView(Finder.java:51)
     at com.dx168.fastdex.sample.CustomView$$ViewBinder.bind(CustomView$$ViewBinder.java:17)
     at com.dx168.fastdex.sample.CustomView$$ViewBinder.bind(CustomView$$ViewBinder.java:12) at butterknife.ButterKnife.bind(ButterKnife.java:187) at butterknife.ButterKnife.bind(ButterKnife.java:133) at  com.dx168.fastdex.sample.CustomView.<init>(CustomView.java:20) ...... at dalvik.system.NativeStart.main(Native Method)Copy the code

Error message: CustomView tv1 (id=2131493007) : CustomView tv1 (id=2131493007) : CustomView ViewBinder.bind

public class CustomView$$ViewBinder<T extends CustomView>
        implements ViewBinder<T>
{
    public CustomView$$ViewBinder()
    {
        System.out.println(AntilazyLoad.str);
    }

    public Unbinder bind(Finder paramFinder, T paramT, Object paramObject)
    {
        InnerUnbinder localInnerUnbinder = createUnbinder(paramT);
        paramT.tv1 = ((TextView)paramFinder.castView((View)paramFinder
                .findRequiredView(paramObject, 2131493007, "field 'tv1'"), 2131493007, "field 'tv1'"));
        paramT.tv3 = ((TextView)paramFinder.castView((View)paramFinder
                .findRequiredView(paramObject, 2131493008, "field 'tv3'"), 2131493008, "field 'tv3'"));
        return localInnerUnbinder; }... }Copy the code

CustomView? The ViewBinder class is dynamically generated by ButterKnife. The value comes from the comments above the Tv1 field of CustomView, which is decomcompiled as follows

public class CustomView extends LinearLayout { @BindView(2131493007) TextView tv1; @BindView(2131493008) TextView tv3; public CustomView(Context paramContext, AttributeSet paramAttributeSet) { super(paramContext, paramAttributeSet); inflate(paramContext, 2130968632, this); ButterKnife.bind(this); this.tv3.setText(2131099697); MainActivity.aa(); System.out.println(AntilazyLoad.str); }}Copy the code

See here is not strange, CustomView source is clearly

public class CustomView extends LinearLayout { @BindView(R.id.tv1) TextView tv1; @BindView(R.id.tv3) TextView tv3; public CustomView(Context context, AttributeSet attrs) { super(context, attrs); inflate(context,R.layout.view_custom,this); ButterKnife.bind(this); tv3.setText(R.string.s3); MainActivity.aa(); }}Copy the code

After compiling, r.i.T.v1 is changed to the number 2131493007 because the Java compiler made a performance optimization. If the source file references a constant with a final descriptor, it will copy the value directly

Decompiling last compilation of success of R.c lass, the result is as follows (app/build/intermediates/classes/debug/com/dx168 / fastdex/sample/R.c lass)

public static final R { public static final class id { ...... public static final int tv1 = 2131493008; public static final int tv2 = 2131492977; public static final int tv3 = 2131493009; . publicid() {}}}Copy the code

After analysis, r.i.T.v1 = 2131493007 when fully packaged. Since all ids in R files are final, references to R.I.t.v1 are replaced with its corresponding value 2131493007. When I add the control named Tv2 to activity_layout. XML and pack the patch, the value of R.D.t.v1 changes to 2131493008, while the value of the cached dex node remains 2131493007. Therefore, I failed to find the corresponding control with ID 2131493007

My first thought was that if I removed the final descriptor for all fields of the ID class in the R file after executing the processDebugResources task, I could get rid of the compilation optimization =>

public static final R { public static final class id { ...... public static int tv1 = 2131493008; public static int tv2 = 2131492977; public static int tv3 = 2131493009; . publicid() {}}}Copy the code

Get rid of later when performing compileDebugJavaWithJavac compilation errors

2.png

The reason for the error is that annotations can only refer to constants with final descriptors. In addition, the switch statement case must also refer to constants

If you take this approach, references to ids will not be able to use constant expressions, and a framework for view dependency injection like ButterKnife will not be able to use it, so the idea is too restrictive

Another idea is to modify the source code of AAPT, so that the value of the same name and ID can be kept the same for multiple packaging. This can definitely be solved, but the workload is too heavy, so we did not do this. Then we adopted a compromise, that is, every time all the classes in the project (except the third-party library) participate in the generation of dex. The problem was solved but the efficiency dropped dramatically. It took nearly 40 seconds to run and it was still slow

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = this problems for a long time, until the source TinkerResourceIdTask tinker open source after reading it, groovy, found they also encountered the same problem, and have a solution, Our scenario and Tinker scenario are exactly the same in this problem, just copy the code to solve the problem, important things say three times, thank Tinker, thank Tinker, thank Tinker!!

Tinker solution is, according to user’s configuration while patching resourceMapping file (output after each building a successful app/build/intermediates/symbols/debug/R.t xt). Generate public. XML and ids. XML and then into the app/build/intermediates/res/merged/debug/values catalog, aapt when processing according to the configuration of the file to generate rules, See Luo’s article analyzing the compilation and packaging process of Android application resources (search public. XML) for details on this

Same as above and combined with our scenario, the first full package its success after the app/build/intermediates/symbols/debug/R.t xt cache, Patch package before processResources missions, according to the symbol table of cache R.t xt to generate public. The XML and ids. The XML and then into the app/build/intermediates/res/merged/debug/values catalog, This twice before and after the id of the same name to build value can be consistent, the following code FastdexResourceIdTask. Groovy

public class FastdexResourceIdTask extends DefaultTask {
    static final String RESOURCE_PUBLIC_XML = "public.xml"
    static final String RESOURCE_IDX_XML = "idx.xml"

    String resDir
    String variantName

    @TaskAction
    def applyResourceId() {
        File buildDir = FastdexUtils.getBuildDir(project,variantName)
        String resourceMappingFile = new File(buildDir,Constant.R_TXT)
        // Parse the public.xml and ids.xml
        if(! FileUtils.isLegalFile(resourceMappingFile)) { project.logger.error("==fastdex apply resource mapping file ${resourceMappingFile} is illegal, just ignore")
            return
        }
        File idsXmlFile = new File(buildDir,RESOURCE_IDX_XML)
        File publicXmlFile = new File(buildDir,RESOURCE_PUBLIC_XML)
        if (FileUtils.isLegalFile(idsXmlFile) && FileUtils.isLegalFile(publicXmlFile)) {
            project.logger.error("==fastdex public xml file and ids xml file already exist, just ignore")
            return
        }
        String idsXml = resDir + "/values/ids.xml";
        String publicXml = resDir + "/values/public.xml";
        FileUtils.deleteFile(idsXml);
        FileUtils.deleteFile(publicXml);
        List<String> resourceDirectoryList = new ArrayList<String>()
        resourceDirectoryList.add(resDir)

        project.logger.error("==fastdex we build ${project.getName()} apk with apply resource mapping file ${resourceMappingFile}")
        Map<RDotTxtEntry.RType, Set<RDotTxtEntry>> rTypeResourceMap = PatchUtil.readRTxt(resourceMappingFile)

        AaptResourceCollector aaptResourceCollector = AaptUtil.collectResource(resourceDirectoryList, rTypeResourceMap)
        PatchUtil.generatePublicResourceXml(aaptResourceCollector, idsXml, publicXml)
        File publicFile = new File(publicXml)

        if (publicFile.exists()) {
            FileUtils.copyFileUsingStream(publicFile, publicXmlFile)
            project.logger.error("==fastdex gen resource public.xml in ${RESOURCE_PUBLIC_XML}")
        }
        File idxFile = new File(idsXml)
        if (idxFile.exists()) {
            FileUtils.copyFileUsingStream(idxFile, idsXmlFile)
            project.logger.error("==fastdex gen resource idx.xml in ${RESOURCE_IDX_XML}") } } } project.afterEvaluate { android.applicationVariants.all { variant -> def variantOutput = variant.outputs.first() Def variantName = variant.name.capitalize() // Keep the same node in the R file when the patch is packaged as it was the first time. FastdexResourceIdTask applyResourceTask = project.tasks.create("fastdexProcess${variantName}ResourceId". com.dx168.fastdex.build.task.FastdexResourceIdTask) applyResourceTask.resDir = variantOutput.processResources.resDir applyResourceTask.variantName = variantName variantOutput.processResources.dependsOn applyResourceTask } }Copy the code

If the project has a lot of resources, the first patch package will take some time to generate public.xml and ids.xml. It is best to do a cache, and later patch package will directly use cached public.xml and ids.xml

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = after solved the problem of the above, then continue to do optimization, it has talked about transformClassesWithMultidexlistForDebug task, Because of the isolation of the Application, all project code is not in classes.dex, and the task of analyzing the classes in the project that need to be in classes.dex is meaningless

project.afterEvaluate {
    android.applicationVariants.all { variant ->
        def variantName = variant.name.capitalize()

        def multidexlistTask = null
        try {
            multidexlistTask = project.tasks.getByName("transformClassesWithMultidexlistFor${variantName}"} catch (Throwable e) {// If multiDexEnabled is not enabled, an exception will be reported for this task.if(multidexlistTask ! = null) { multidexlistTask.enabled =false}}}Copy the code

When disabled, executing./gradle assembleDebug failed during build

:app:transformClassesWithMultidexlistForDebug SKIPPED
:app:transformClassesWithDexForDebug
Running dex in- Process requires build tools 23.0.2. For faster builds update this project to use the latest build tools. UNEXPECTED TOP-LEVEL ERROR: java.io.FileNotFoundException: /Users/tong/Projects/fastdex/app/build/intermediates/multi-dex/debug/maindexlist.txt (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at java.io.FileReader.<init>(FileReader.java:58) at com.android.dx.command.dexer.Main.readPathsFromFile(Main.java:436) at com.android.dx.command.dexer.Main.runMultiDex(Main.java:361) at com.android.dx.command.dexer.Main.run(Main.java:275) at com.android.dx.command.dexer.Main.main(Main.java:245) at com.android.dx.command.Main.main(Main.java:106) :app:transformClassesWithDexForDebug FAILED FAILURE: Build failed with an exception. ...... BUILD FAILEDCopy the code

From the above log the first line of the found transformClassesWithMultidexlistForDebug task indeed banned off, followed by a SKIPPED the output, But transformClassesWithDexForDebug mission times app/build/intermediates/multi – dex/debug/maindexlist. TXT (No to file the or Directory), the reason is transformClassesWithDexForDebug task will check if the file exists, in this case just before transformClassesWithDexForDebug mission to create an empty file, see if still complains, The following code

public class FastdexCreateMaindexlistFileTask extends DefaultTask {
    def applicationVariant

    @TaskAction
    void createFile() {
        if(applicationVariant ! = null) { File maindexlistFile = applicationVariant.getVariantData().getScope().getMainDexListFile() File parentFile = maindexlistFile.getParentFile()if(! parentFile.exists()) { parentFile.mkdirs() }if(! maindexlistFile.exists() || maindexlistFile.isDirectory()) { maindexlistFile.createNewFile() } } } } project.afterEvaluate { android.applicationVariants.all { variant -> def variantName = variant.name.capitalize() def multidexlistTask = null try { multidexlistTask = project.tasks.getByName("transformClassesWithMultidexlistFor${variantName}"} catch (Throwable e) {// If multiDexEnabled is not enabled, an exception will be reported for this task.if(multidexlistTask ! = null) { FastdexCreateMaindexlistFileTask createFileTask = project.tasks.create("fastdexCreate${variantName}MaindexlistFileTask". FastdexCreateMaindexlistFileTask) createFileTask.applicationVariant = variant multidexlistTask.dependsOn createFileTask multidexlistTask.enabled =false}}}Copy the code

Execute./gradle assembleDebug again

:app:transformClassesWithJarMergingForDebug UP-TO-DATE
:app:collectDebugMultiDexComponents UP-TO-DATE
:app:fastdexCreateDebugMaindexlistFileTask
:app:transformClassesWithMultidexlistForDebug SKIPPED
:app:transformClassesWithDexForDebug UP-TO-DATE
:app:mergeDebugJniLibFolders UP-TO-DATE
:app:transformNative_libsWithMergeJniLibsForDebug UP-TO-DATE
:app:processDebugJavaRes UP-TO-DATE
:app:transformResourcesWithMergeJavaResForDebug UP-TO-DATE
:app:validateConfigSigning
:app:packageDebug UP-TO-DATE
:app:zipalignDebug UP-TO-DATE
:app:assembleDebug UP-TO-DATE

BUILD SUCCESSFUL

Total time: 16.201 secsCopy the code

This successful build demonstrates that this approach to creating empty files works

= = = = = = = = =

Our company’s project in the process of use, although have a little to find patches a Java class, but when building compileDebugJavaWithJavac tasks or spent 13 seconds

BUILD SUCCESSFUL Total time: 28.222 SECS Task 554ms :app:processDebugManifest 127ms :app:mergeDebugResources 3266ms :app:processDebugResources 13621ms :app:compileDebugJavaWithJavac 3654ms :app:transformClassesWithJarMergingForDebug 1354ms :app:transformClassesWithDexForDebug 315ms :app:transformNative_libsWithMergeJniLibsForDebug 220ms :app:transformResourcesWithMergeJavaResForDebug 2684ms :app:packageDebugCopy the code

After analysis because we use the butterknife and tinker, both inside with the javax.mail. The annotation. Processing. AbstractProcessor do code dynamically generated this interface, so the project if many Java file, Scanning all the Java files one by one and doing the same thing would be a huge waste of time, but they generate almost the same code every time, so if the patch was packaged with its own implementation instead of just compiling and snapshot the changed Java files, And output the results to the app/build/intermediates/classes/debug, covering the original class, can greatly improve the efficiency, the part of the code is as follows, for details see FastdexCustomJavacTask. Groovy

public class FastdexCustomJavacTask extends DefaultTask {
    ......

    @TaskAction
    void compile() {... File androidJar = new File("${project.android.getSdkDirectory()}/platforms/${project.android.getCompileSdkVersion()}/android.jar")
        File classpathJar = FastdexUtils.getInjectedJarFile(project,variantName)
        project.logger.error("==fastdex androidJar: ${androidJar}")
        project.logger.error("==fastdex classpath: ${classpathJar}")
        project.ant.javac(
                srcdir: patchJavaFileDir,
                source: '1.7',
                target: '1.7',
                encoding: 'UTF-8',
                destdir: patchClassesFileDir,
                bootclasspath: androidJar,
                classpath: classpathJar
        )
        compileTask.enabled = false
        File classesDir = applicationVariant.getVariantData().getScope().getJavaOutputDir()
        Files.walkFileTree(patchClassesFileDir.toPath(),new SimpleFileVisitor<Path>(){
            @Override
            FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {
                Path relativePath = patchClassesFileDir.toPath().relativize(file)
                File destFile = new File(classesDir,relativePath.toString())
                FileUtils.copyFileUsingStream(file.toFile(),destFile)
                returnFileVisitResult.CONTINUE } }) } } project.afterEvaluate { android.applicationVariants.all { variant -> def variantName =  variant.name.capitalize() Task compileTask = project.tasks.getByName("compile${variantName}JavaWithJavac")
        Task customJavacTask = project.tasks.create("fastdexCustomCompile${variantName}JavaWithJavac". com.dx168.fastdex.build.task.FastdexCustomJavacTask) customJavacTask.applicationVariant = variant customJavacTask.variantName = variantName customJavacTask.compileTask = compileTask compileTask.dependsOn customJavacTask } }Copy the code

Execute./gradlew assembleDebug, again

BUILD SUCCESSFUL

Total time: 17.555 secs
Task spend time:
   1142ms  :app:fastdexCustomCompileDebugJavaWithJavac
     59ms  :app:generateDebugBuildConfig
    825ms  :app:processDebugManifest
    196ms  :app:mergeDebugResources
   3540ms  :app:processDebugResources
   3045ms  :app:transformClassesWithJarMergingForDebug
   1505ms  :app:transformClassesWithDexForDebug
    391ms  :app:transformNative_libsWithMergeJniLibsForDebug
    253ms  :app:transformResourcesWithMergeJavaResForDebug
   3413ms  :app:packageDebugCopy the code

That’s about 10 seconds faster, Good

========= Since there is a cache, there is a cache expiration problem. If we add a dependency of a third party library and reference it in the project code, if we do not clear the cache, the package class will not be found after running, so we need to deal with this problem. How do you get dependencies in the first place? You can get a list of dependencies with the following code

project.afterEvaluate { project.configurations.all.findAll { ! it.allDependencies.empty }.each { c ->if (c.name.toString().equals("compile")
                || c.name.toString().equals("apt")
                || c.name.toString().equals("_debugCompile".toString())) {
            c.allDependencies.each { dep ->
                String depStr =  "$dep.group:$dep.name:$dep.version"
                println("${depStr}")}}}}Copy the code

Enter the following

Com. Dialonce: dialonce - android: 2.3.1 com. Facebook. The fresco ": fresco" : 1.1.0 com. Google. : guava guava: 18.0... Com. Android. Support: design: 23.4.0 com. Bigkoo: alertview: 1.0.2 com. Bigkoo: pickerview: mid-atlantic movedCopy the code

A list of current dependencies can be retrieved and saved at the first full package at the same point in time as a snapshot of the project’s source directory was generated. A list of current dependencies can be retrieved when the patch is packaged, compared to the previously saved list, and the cache can be cleared if changes occur

It is also best to provide an active task to clear the cache

public class FastdexCleanTask extends DefaultTask {
    String variantName

    @TaskAction
    void clean() {
        if (variantName == null) {
            FastdexUtils.cleanAllCache()
        }
        else {
            FastdexUtils.cleanCache(project,variantName)
        }
    }
}Copy the code

Start with a task to clear all caches

project.tasks.create("fastdexCleanAll", FastdexCleanTask)Copy the code

Build the cleanup task based on buildType and flavor

Android. ApplicationVariants. All {variant - > def variantName = variant. The name. Capitalize () / / create clear specified variantName caching task (user) FastdexCleanTask cleanTask = project.tasks.create("fastdexCleanFor${variantName}", FastdexCleanTask)
    cleanTask.variantName = variantName
}Copy the code

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

### Subsequent optimization plan

  • 1, improve stability and fault tolerance, this is the most critical
  • 2, the patch package, is no change of class from the app/build/intermediates/transforms/jarMerging/debug/jars/f / 1/1 combined. The jar is removed, If transformClassesWithJarMergingForDebug can hook off the task, only to change the class to participate in combined. The generation of the jar, can save a lot of time on the IO
  • 3, at present to the project source directory snapshot, the use of file copy, if you can only write the required information in the text file, can save some time in IO
  • 4. Currently, changes in liBS directory have not been monitored, and this should be made up later
  • 5. The installation speed of APK is relatively slow (especially for ART, it is used for AOT compilation during installation, so the installation speed is very slow. For details, please refer to Zhang Shaowen’s article Android N Mixed Compilation and Analysis of the Impact of Hot patches)

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

### Here is a summary of the packaging process

Packaging process

The process of full packaging:
  • 1. Merge all class files to generate a JAR package
  • 2, code scanning all of the project, and add to fastdex in the constructor. The runtime. Antilazyload. Antilazyload class depends on the purpose is to solve a class of verify, please The android App hotfixes dynamic repair technology is introduced
  • 3. Take a snapshot of the project code to compare those Java files for later patch packaging
  • 4. Make snapshots of all dependencies of the current project, in order to compare whether the dependencies have changed when the patch is packaged later, and clear the cache if they have changed
  • 5. Call the real transform to generate dex
  • 6. Cache the generated dex and insert the fastdex-Runtime. dex into the dex list. Fastdex-runtime. dex => classes.dex classes2.dex => classes2.dex => classes2.dex => classes2.dex => classes2.dex => Classes3. Dex and then run the entrance Application (fastdex. Runtime. FastdexApplication) using MultiDex all the dex loaded in
  • @see fastdex.build.transform.FastdexTransform
  • 7. Save the resource mapping table. In order to keep the values of ids consistent, see details
  • @see fastdex.build.task.FastdexResourceIdTask
Process of patch packaging
  • Check the validity of the cache
  • @ see fastdex. Build. The variant. FastdexVariant prepareEnv method
  • 2. Scan all changed Java files and compile to class
  • @see fastdex.build.task.FastdexCustomJavacTask
  • 3. Merge all changed classes and generate jar packages
  • 4. Generate the dex patch
  • 5, all the dex according to certain rules on transformClassesWithMultidexlistFor ${variantName} task output directory fastdex – runtime. Dex = > classes. Dex patch. Dex => classes2.dex dex_cache.classes.dex => classes3.dex dex_cache.classes2.dex => classes4.dex dex_cache.classesN.dex => classes(N + 2).dex

= = = = = = = = = = = = =

The entire project code is now open sourceGithub.com/typ0520/fas…

If you enjoyed this post, send us star

============= Speed up apK build speed, how to reduce the build time from 130 seconds to 17 seconds (ii)

Reference projects and articles

Instant Run

Tinker

Introduction to hot patch dynamic repair technology of Android App

Android application resource compilation and packaging process analysis

Key words: Faster APK compilation speed Faster Android App compilation speed Faster Android Studio compilation speed Slow Android Studio compilation speed optimized Android Studio Gradle compiles slowly

This article from typ0520 Jane books blog www.jianshu.com/p/53923d8f2…