As an Android developer, in addition to writing Java code, have you ever thought about playing with class files? There are a lot of fun things we can do directly with the bytecode of a class file, such as:

  • Global all class staking, do UI, memory, network and other aspects of performance monitoring
  • You find a third-party dependency that you don’t like to use, but you don’t want to modify its source code and recompile it. You want to do something about its class directly
  • If you want the TAG to be generated automatically every time you log, it will default to the name of the current class. If you want the current line of code to be added to the log, it will be easier to locate the log
  • Java’s native dynamic proxies are too weak to do dynamic proxies only for interface classes, whereas we want to do dynamic proxies for any class

In order to realize the above ideas, it may be, we started the first reaction is whether through code generation, APT, reflection, or dynamic proxy to implement, but it want to, as these solutions can well meet the requirements of the above, moreover, some problems can not be from Java file, and should be a breakthrough from the class file. Starting with class files, we have to take a closer look at bytecode!

From ORM frameworks (Hibernate, MyBatis) to Mock frameworks (Mockio), to the evergreen Spring framework in the Java Web, to the compiler for the emerging JVM language Kotlin, And then there’s the famous Cglib project, which has bytecode in it.

The power of bytecode technology goes without saying, and Android development, whether using Java development and Kotlin development, is the JVM platform language, so if we use bytecode technology hack Android development, also can naturally compatible with Java and Kotlin language.

Recently, I have done some research and practice on the application of bytecode technology on Android. Incidentally, I have made several little wheels. Project address: Hunter

  • Hunter: A plug-in framework that allows you to quickly develop concurrent, incremental bytecode compilation plug-ins that help developers hide most of the logic of Transform and ASM. Developers can write a small amount of ASM code to develop a compilation plug-in that changes the bytecode of Android projects.

Based on the above framework, I also developed several small tools

  • The okHTTP-plugin allows you to set the global Interceptor/Eventlistener for all okHttpClients in your application (including okHttpClients in third-party dependencies).
  • Timing-plugin: Allows you to monitor the execution time of all UI threads, and provides an algorithm to print a stack with the execution time of each step. You can also customize how to analyze the stack.
  • Logline-plugin: Add line numbers to your logs
  • Debug-Plugin: Just add an annotation to a given method, and you can print out the values of all the input parameters of that method, the return values, and the execution time. (In fact, JakeWharton’s Hugo implements this functionality in AspectJ, and I do it using ASM, which processes bytecode faster.)
  • You can check out some of the TODO plugins I want to continue developing here, and you are welcome to provide your valuable ideas

Writing this article today, I share some of my shortcomings in the process of exploring relevant technologies and developing this project.

The main technologies used in this project are Android Gradle plugin, Transform, ASM and bytecode foundation. This article will focus on the following technical points:

  • The application, principle and optimization of Transform
  • ASM application, development flow, and adaptation with Android project
  • Several specific application cases

So reading this article, you should have a background in Android development and writing simple Gradle plug-ins.

Without further ado, let’s begin.

A, the Transform

The introduction of the Transform

Transform is a concept introduced in Android Gradle Plugin 1.5.

Let’s start with how to introduce the Transform dependency. First we need to write a custom plug-in, and then register a custom Transform in the plug-in. The Transform library is independent from gradle. Since version 2.0.0, the Transform library has been incorporated into the Gradle-API that the Android build system relies on. Let’s take a look at a current version of Transform on JCenter.

So, a long, long time ago I introduced transform dependencies like this

The compile 'com. Android. Tools. Build: transform - API: 1.5.0'Copy the code

Right now

/ / since 2.0.0 version is in the gradle - API implementation 'com. Android. View the build: gradle - API: 3.1.4'Copy the code

Next, let’s register a custom Transform in our custom plugin. The Gradle plugin can be written in Java, Groovy, or Kotlin. I chose Java here.

public class CustomPlugin implements Plugin<Project> { @SuppressWarnings("NullableProblems") @Override public void apply(Project project) { AppExtension appExtension = (AppExtension)project.getProperties().get("android"); appExtension.registerTransform(new CustomTransform(), Collections.EMPTY_LIST); }}Copy the code

So how do you write a custom Transform?

Principle and application of Transform

Before we introduce how to apply Transform, let’s first introduce the principle of Transform. A picture is worth a thousand words

Each Transform is actually a Gradle Task. The TaskManager in the Android compiler concatenates each Transform. The first Transform receives the result from the Javac compilation. As well as third-party dependencies that have been pulled locally (jar.aar), and resource resources, note that the resource is not the RES resource in the Android project, but the asset directory. These compiled intermediates flow along a chain of transforms, and each Transform node can process the class and pass it on to the next Transform. Our common confusions, such as Desugar logic, are now implemented in a single Transform, and our custom Transform is inserted at the top of the Transform chain.

However, the above image is just one example of a Transform. A Transform can have two types of input: consumption, which the current Transform needs to output to the next Transform, and reference, which the current Transform can read without output to the next Transform. Instant Run, for example, checks diff between compilations in this way. Examples of how to declare both inputs in a Transform and how to handle both inputs follow.

In order to confirm the working principle and application of the Transform way, we can also from Android gradle plugin source of finding evidence, in TaskManager createPostCompilationTasks have a method. To avoid Posting too long source code, here is the link

TaskManager#createPostCompilationTasks

The context of this method is very clear, we can see Jacoco, Desugar, MergeJavaRes, AdvancedProfiling, Shrinker, Proguard, JarMergeTransform, MultiDex, and Dex are all connected in series in the form of Transform. It also inserts our custom Transform.

After talking about the principle of data flow of Transform, we will introduce the filtering mechanism of input data of Transform. The data input of Transform can be filtered through Scope and ContentType.

ContentType, as the name implies, is a data type. In plug-in development, we can only use CLASSES and RESOURCES. Note that CLASSES already contain class files and JAR files

As you can see from the figure, in addition to CLASSES and RESOURCES, there are other types that we can’t use during development, such as DEX files. These hidden types are in a separate enumerated class, ExtendedContentType, which is only available to the Android compiler. In addition, we generally use several common sets of contentTypes and scopes provided in TransformManager. If we want to process all the bytecodes of classes and jars, ContentType We normally use transformManager.content_class.

Scope is another dimension filter compared to ContentType,

As you can see, there are several types available on the left, and we use them in combination. TransformManager has several common Scope sets for developers to use. To handle all class bytecodes, Scope is normally transformmanager.scope_full_project.

So far, we have introduced the principle of the data flow of the Transform, the types of input, and the filtering mechanism. Let’s write a simple custom Transform to give us a more concrete understanding of the Transform

public class CustomTransform extends Transform { public static final String TAG = "CustomTransform"; public CustomTransform() { super(); } @Override public String getName() { return "CustomTransform"; } @Override public void transform(TransformInvocation transformInvocation) throws TransformException, InterruptedException, IOException { super.transform(transformInvocation); / / if the current is incremental compilation Boolean isIncremental = transformInvocation. IsIncremental (); // Consumer input, from which you can get the JAR package and the class folder path. Need to output to the next task Collection < TransformInput > inputs. = transformInvocation getInputs (); // Reference input, no output required. Collection<TransformInput> referencedInputs = transformInvocation.getReferencedInputs(); //OutputProvider manages the output path. If the consumption input is empty, You will find OutputProvider = = null TransformOutputProvider OutputProvider = transformInvocation. GetOutputProvider (); for(TransformInput input : inputs) { for(JarInput jarInput : input.getJarInputs()) { File dest = outputProvider.getContentLocation( jarInput.getFile().getAbsolutePath(), jarInput.getContentTypes(), jarInput.getScopes(), Format.JAR); Fileutils.copyfile (jarinput.getFile (), dest); fileutils.copyfile (jarinput.getfile ()); } for(DirectoryInput directoryInput : input.getDirectoryInputs()) { File dest = outputProvider.getContentLocation(directoryInput.getName(), directoryInput.getContentTypes(), directoryInput.getScopes(), Format.DIRECTORY); Fileutils.copydirectory (directoryInput.getFile(), dest); fileutils.copydirectory (directoryInput.getFile); } } } @Override public Set<QualifiedContent.ContentType> getInputTypes() { return TransformManager.CONTENT_CLASS; } @Override public Set<? super QualifiedContent.Scope> getScopes() { return TransformManager.SCOPE_FULL_PROJECT; } @Override public Set<QualifiedContent.ContentType> getOutputTypes() { return super.getOutputTypes(); } @Override public Set<? super QualifiedContent.Scope> getReferencedScopes() { return TransformManager.EMPTY_SCOPES; } @Override public Map<String, Object> getParameterInputs() { return super.getParameterInputs(); } @Override public boolean isCacheable() { return true; } @Override public boolean isIncremental() { return true; // Enable incremental compilation}}Copy the code

As you can see, in the transform method, we copy each jar package and class file to the dest path, which is the input data for the next transform. The bytecodes of jar packages and class files can be modified before copying. As for how to modify bytecodes, we will use ASM later. And if the development process depends on your current transform processing after class/jar package, can go to/build/intermediates/transforms/CustomTransform/view, you will find all the jar package name is 123456, This is normal, the naming rules can be OutputProvider. GetContentLocation found in specific implementation

public synchronized File getContentLocation( @NonNull String name, @NonNull Set<ContentType> types, @NonNull Set<? super Scope> scopes, @NonNull Format format) { // runtime check these since it's (indirectly) called by 3rd party transforms. checkNotNull(name); checkNotNull(types); checkNotNull(scopes); checkNotNull(format); checkState(! name.isEmpty()); checkState(! types.isEmpty()); checkState(! scopes.isEmpty()); // search for an existing matching substream. for (SubStream subStream : subStreams) { // look for an existing match. This means same name, types, scopes, and format. if (name.equals(subStream.getName()) && types.equals(subStream.getTypes()) && scopes.equals(subStream.getScopes()) && format == subStream.getFormat()) { return new File(rootFolder, subStream.getFilename()); }} // increment by position!! // didn't find a matching output. create the new output SubStream newSubStream = new SubStream(name, nextIndex++, scopes, types, format, true); subStreams.add(newSubStream); return new File(rootFolder, newSubStream.getFilename()); }Copy the code

Optimization of Transform: incremental versus concurrent

So far, it seems that Transform is not difficult to use, but if you use it directly, it will greatly slow down the compilation time. In order to solve this problem, after exploring for a while, we also learned from several Transform implementations such as Desugar in the Android compiler, and found that we can use incremental compilation. The transform method above iterates through the process of processing each JAR /class, which can be processed concurrently, plus the general compilation process is on the PC, so we can try to extant resources from the machine.

To enable incremental compilation, we need to override the Transform interface and return true.

@Override 
public boolean isIncremental() {
    return true;
}Copy the code

Although incremental compilation is enabled, not every build process supports deltas. After all, a clean Build has no basis for deltas at all, so we need to check whether the current build is incremental.

If it is not an incremental compilation, it clears the output directory and does the same class/jar by class as before. If it is an incremental compilation, it checks the Status of each file. There are four types of Status and different operations are performed for each file

  • NOTCHANGED: The current file does not need to be processed, or even copied;
  • ADDED, CHANGED: Normally processed, output to the next task.
  • REMOVED: Removes the file obtained by the outputProvider.

Perhaps the implementation can take a look at the code below

@Override public void transform(TransformInvocation transformInvocation){ Collection<TransformInput> inputs = transformInvocation.getInputs(); TransformOutputProvider outputProvider = transformInvocation.getOutputProvider(); boolean isIncremental = transformInvocation.isIncremental(); // If it is not incremental, empty the old output if(! isIncremental) { outputProvider.deleteAll(); } for(TransformInput input : inputs) { for(JarInput jarInput : input.getJarInputs()) { Status status = jarInput.getStatus(); File dest = outputProvider.getContentLocation( jarInput.getName(), jarInput.getContentTypes(), jarInput.getScopes(), Format.JAR); if(isIncremental && ! emptyRun) { switch(status) { case NOTCHANGED: continue; case ADDED: case CHANGED: transformJar(jarInput.getFile(), dest, status); break; case REMOVED: if (dest.exists()) { FileUtils.forceDelete(dest); } break; } } else { transformJar(jarInput.getFile(), dest, status); } } for(DirectoryInput directoryInput : input.getDirectoryInputs()) { File dest = outputProvider.getContentLocation(directoryInput.getName(), directoryInput.getContentTypes(), directoryInput.getScopes(), Format.DIRECTORY); FileUtils.forceMkdir(dest); if(isIncremental && ! emptyRun) { String srcDirPath = directoryInput.getFile().getAbsolutePath(); String destDirPath = dest.getAbsolutePath(); Map<File, Status> fileStatusMap = directoryInput.getChangedFiles(); for (Map.Entry<File, Status> changedFile : fileStatusMap.entrySet()) { Status status = changedFile.getValue(); File inputFile = changedFile.getKey(); String destFilePath = inputFile.getAbsolutePath().replace(srcDirPath, destDirPath); File destFile = new File(destFilePath); switch (status) { case NOTCHANGED: break; case REMOVED: if(destFile.exists()) { FileUtils.forceDelete(destFile); } break; case ADDED: case CHANGED: FileUtils.touch(destFile); transformSingleFile(inputFile, destFile, srcDirPath); break; } } } else { transformDir(directoryInput.getFile(), dest); }}}}Copy the code

This provides incremental features for our compiled plug-in.

Once incremental compilation is implemented, it is best to support concurrent compilation as well. The implementation of concurrent compilation is not complicated, just need to process the logic for a single JAR /class, process it concurrently, and finally block until all tasks are finished.

private WaitableExecutor waitableExecutor = WaitableExecutor.useGlobalSharedThreadPool(); Class waitableExecutor.execute(() -> {bytecodeWeaver. WeaveJar (srcJar, destJar); return null; }); waitableExecutor.execute(() -> { bytecodeWeaver.weaveSingleClassToFile(file, outputFile, inputDirPath); return null; }); / / wait for all task end waitableExecutor. WaitForTasksWithQuickFail (true);Copy the code

Next, we make a comparison on compilation speed. Each experiment is compiled 10 times under the same conditions for 5 times, and the average time is taken after removing the maximum size value

First of all, in the QQ Mailbox Android client project, we will do a cleanbuild first

./gradlew clean assembleDebug --profileCopy the code

To add UI time statistics to the project, globally insert the first and last lines of each method (including ordinary class files and all classes in third-party JAR packages), using Transform+ASM. The time of Tranform step

As you can see, concurrent compilation is basically 80% faster than non-concurrent compilation. The effect is remarkable.

Then, in another experiment, we simulate a routine modification of a class file in a project that is compatible with incremental compilation. Then do the same piling logic on the basis of the previous, and compare the difference between the incremental Transform and the full Transform.

./gradlew assembleDebug --profileCopy the code

It can be seen that the incremental speed is more than 3 times faster than the full speed, and this speed optimization becomes more significant as the project gets larger.

The data shows that increments and concurrency have a significant impact on compilation speed. Some of them are written in Kotlin, some are written in Java, some support delta, some don’t. And the code comments write a large FIXME To support incremental build. So, to be fair, there is still room for improvement in Android compilation speed.

We’ve talked about Transform and how to efficiently process all the bytecodes at compile time, but what about bytecodes? Let’s take a look at the JVM platform bytecode processing magic tool, ASM!

Second, the ASM

The ASM website is here asm.ow2. IO /, and you can see how powerful it is

On the JVM platform, the most common frameworks for processing bytecode are ASM, Javasist and AspectJ. I tried Javasist, and AspectJ knew a little bit about it, and chose ASM because it could process each command of bytecode at a lower level, with speed, memory footprint, and better than the other two frameworks.

We can make a comparison. Based on the above experiment of calculating compilation time, we do the following experiment. In the case that ASM and Javasist are respectively used to fully process all the classes of the project and concurrent processing is not enabled, the time comparison of transform in a clean build is as follows

ASM has significant advantages over Javasist, and the efficiency and performance advantages of ASM over other bytecode operation libraries should be beyond doubt. After all, it is a bytecode generation library authorized by many JVM languages.

We will introduce ASM in this part in the future, but due to space problems, not from the basis of bytecode introduction, will introduce some bytecode knowledge through the implementation of several examples, in addition, will introduce the use of ASM, as well as ASM parsing class file structure principle, as well as the application of Android plug-in development, encountered problems. And its solutions.

The introduction of the ASM

Gradle’s custom plugin + transform + ASM dependencies are listed below. Note the differences between the two Gradle apis

Dependencies {// Use the gradle Wrapper version specified in the project Implementation gradleApi(); implementation localGroovy(); implementation localGroovy(); Such as said above TaskManager implementation 'com. Android. View the build: gradle: 3.1.4' / / this dependence is mainly the reliance of the transform, pay attention to, This dependence is different from the above gradleApi () implementation 'com. Android. View the build: gradle - API: 3.1.4' / / ASM related implementation 'org.ow2.asm:asm:5.1' implementation 'org.ow2.asm:asm:5.1' implementation 'org.ow2.asm: asm-Commons :5.1'}Copy the code

The application of ASM

ASM designs two API types, one is Tree API, one is based on Visitor API(Visitor Pattern),

The Tree API reads the class structure into memory and builds a Tree structure. Then, when processing elements such as Method and Field, the Tree locates an element in the Tree structure, performs operations, and writes the operations to a new class file.

The Visitor API separates the logic that reads and writes classes by interface, typically through a ClassReader that reads the class bytecode, and then through a ClassVisitor interface that reads the class bytecode. Each detail of the bytecode is passed in order through the interface to the ClassVisitor (you’ll find multiple visitXXXX interfaces in the ClassVisitor), just as the ClassReader takes the ClassVisitor through each instruction of the class bytecode.

The above two kinds of parsing the file structure of the way in many dealing with structured data are common, general background to choose the appropriate scheme depends on demand, and our demand is such, for a certain purpose, looking for a hook point in the class files, to bytecode modification, this background, the way we choose the Visitor API is more appropriate.

Let’s write a simple demo that reads the contents of a class through the Visitor API and saves them to another file

private void copy(String inputPath, String outputPath) { try { FileInputStream is = new FileInputStream(inputPath); ClassReader cr = new ClassReader(is); ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES); cr.accept(cw, 0); FileOutputStream fos = new FileOutputStream(outputPath); fos.write(cw.toByteArray()); fos.close(); } catch (IOException e) { e.printStackTrace(); }}Copy the code

First, we read a class file with a ClassReader, and then we define a ClassWriter, which we can look at the source code for, which is basically a ClassVisitor implementation, Write ClassReader data to a byte stream, which is triggered by ClassWriter accept.

public void accept(ClassVisitor classVisitor, Attribute[] attributePrototypes, Int parsingOptions) {// Read bytecode information of the current class int accessFlags = this.readUnsignedShort(currentOffset); String thisClass = this.readClass(currentOffset + 2, charBuffer); String superClass = this.readClass(currentOffset + 4, charBuffer); String[] interfaces = new String[this.readUnsignedShort(currentOffset + 6)]; //classVisitor is the ClassWriter that was passed by the accept method. Visit (this.readint (this.cpInfooffsets [1] -7), accessFlags, thisClass, Signature, thisClass, thisClass, thisClass, thisClass superClass, interfaces); // /visit Attribute while(attributes! = null) { Attribute nextAttribute = attributes.nextAttribute; attributes.nextAttribute = null; classVisitor.visitAttribute(attributes); attributes = nextAttribute; } /** omit a lot of visit logic */ classvisit.visitend (); }Copy the code

Finally, we export the bytecode passed from ClassReader to ClassWriter via toByteArray() of ClassWriter and write it to a new file. This completes the copy of the class file. This demo, while very simple, covers ASM using the Visitor API to modify the lowest level of bytecode

If we want to modify bytecode, we should start from ClassWriter. As mentioned above, each visitXXX in ClassWriter (these interfaces are implemented from ClassVisitor) will save bytecode information and can be exported eventually. So if we can proxy the ClassWriter interface, we can interfere with the final bytecode generation.

So the diagram above should look something like this

We just have to look a little bit at the ClassVisitor code and see that its constructor can accept another ClassVisitor and delegate all the methods through that ClassVisitor. Let’s look at an example of inserting a line of code at the beginning and end of each method call statement in a class

The method before modification looks like this


private static void printTwo() {
    printOne();
    printOne();
}Copy the code

The modified method looks like this


private static void printTwo() {
    System.out.println("CALL printOne");
    printOne();
    System.out.println("RETURN printOne");
    System.out.println("CALL printOne");
    printOne();
    System.out.println("RETURN printOne");
}Copy the code

So let’s see how do we do that with ASM

private static void weave(String inputPath, String outputPath) { try { FileInputStream is = new FileInputStream(inputPath); ClassReader cr = new ClassReader(is); ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES); CallClassAdapter adapter = new CallClassAdapter(cw); cr.accept(adapter, 0); FileOutputStream fos = new FileOutputStream(outputPath); fos.write(cw.toByteArray()); fos.close(); } catch (IOException e) { e.printStackTrace(); }}Copy the code

The only difference between this code and the code that implements the copy class above is the use of a CallClassAdapter, which is a custom ClassVisitor that passes the ClassWriter to the constructor of the CallClassAdapter. Let’s look at the implementation

//CallClassAdapter.java public class CallClassAdapter extends ClassVisitor implements Opcodes { public CallClassAdapter(final ClassVisitor cv) { super(ASM5, cv); } @Override public void visit(int version, int access, String name, String signature, String superName, String[] interfaces) { super.visit(version, access, name, signature, superName, interfaces); } @Override public MethodVisitor visitMethod(final int access, final String name, final String desc, final String signature, final String[] exceptions) { MethodVisitor mv = cv.visitMethod(access, name, desc, signature, exceptions); return mv == null ? null : new CallMethodAdapter(name, mv); } } //CallMethodAdapter.java class CallMethodAdapter extends MethodVisitor implements Opcodes { public CallMethodAdapter(final MethodVisitor mv) { super(ASM5, mv); } @Override public void visitMethodInsn(int opcode, String owner, String name, String desc, boolean itf) { mv.visitFieldInsn(GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;" ); mv.visitLdcInsn("CALL " + name); mv.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;) V", false); mv.visitMethodInsn(opcode, owner, name, desc, itf); mv.visitFieldInsn(Opcodes.GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;" ); mv.visitLdcInsn("RETURN " + name); mv.visitMethodInsn(Opcodes.INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;) V", false); }}Copy the code

The visitMethod in the CallClassAdapter uses a custom MethodVisitor — the CallMethodAdapter, which also proxies for the original MethodVisitor, in the same way as the proxy for the ClassVisitor.

See here, using the modified ASM bytecode appeared to be about routines are covered, so how to write the above visitMethodInsn method to insert the logic of printing the method name, which requires some basic knowledge of the bytecode, we said that this will not introduce bytecode, but we can introduce some learn quickly bytecode way, It is also a useful tool for developing bytecode related projects.

Before we do that, let’s talk about line numbers

How do I verify line numbers

We inserted a line of log print before and after each method call, then did not think that this is not the number of lines of code, so that if crash, positioning stack is not chaotic. In fact, this is not the case, visitMethodInsn above do things, in fact, in the same line of code, we posted above the code looks like this


private static void printTwo() {
    System.out.println("CALL printOne");
    printOne();
    System.out.println("RETURN printOne");
    System.out.println("CALL printOne");
    printOne();
    System.out.println("RETURN printOne");
}Copy the code

If you open the above class file with IDEA or Eclipse, it is displayed in line after line, but the actual number of lines inside the class should be like this


private static void printTwo() {
    System.out.println("CALL printOne"); printOne(); System.out.println("RETURN printOne");
    System.out.println("CALL printOne"); printOne(); System.out.println("RETURN printOne");
}Copy the code

Idea allows you to enable an option to keep the actual number of lines when viewing class content

When you turn it on, what you see is this

We can see that lines 17 and 18 each contain three lines of code.

This is before you turn on the option

So how do you turn this on? Mac CMD + Shift + A enter Registry and check both options

All code in class, whether it’s bytecode code or ASM code, is going to declare line number X, and then start several bytecode instructions, and those bytecode instructions are going to be in line number X until the next line number is declared.

ASM code

Parse to show how to write the logic that generates the code above. First, let’s imagine that if we were to make changes to a class, what specific changes would be required to the bytecode? The most straightforward way to do this is to compile the target class and see how its bytecode differs from the original class’s bytecode (you can use javap tools to see the bytecode), but this is not enough. Instead of reading and writing the bytecode, we use ASM to modify the bytecode. Bytecode is the JVM bytecode, and ASM code is the ASM bytecode. In fact, the two are very similar, but ASM code is Java code. So instead of comparing bytecode, we should compare ASM Code. Compared to ASM Code diff, that’s basically what we’re going to do.

ASM also provides a class like this: ASMifier, it can generate ASM code, but there are more efficient tools, Intellij IDEA plug-ins ASM Bytecode Outline, you can view a class file Bytecode and ASM code.

So far, seems to use contrast ASM code, to achieve the bytecode changes is not difficult, but this way only can implement some scenarios, the basis of modified bytecode still has a lot of scenarios is the need to have some basic knowledge on byte code to do this, moreover, to read to understand the ASM code, also requires some knowledge of byte code. So, if you want to develop bytecode engineering, you still need to learn bytecode.

ClassWriter pit on Android

If we apply ASM directly to the Android compiler, we will run into a pit from ClassWriter, because one of the logic in ClassWriter is to find the common parent of both classes. Look at the method getCommonSuperClass in ClassWriter,

/** * Returns the common super type of the two given types. The default * implementation of this method <i>loads</i> the  two given classes and uses * the java.lang.Class methods to find the common super class. It can be * overridden to compute this common super type in other ways, in particular * without actually loading any class, or to take into account the class * that is currently being generated by this ClassWriter, which can of * course not be loaded since it is under construction. * * @param type1 * the internal name of a class. * @param type2 * the internal name of another class. * @return the internal name of the common super class of the two given * classes. */ protected String getCommonSuperClass(final String type1, final String type2) { Class<? > c, d; ClassLoader classLoader = getClass().getClassLoader(); try { c = Class.forName(type1.replace('/', '.'), false, classLoader); d = Class.forName(type2.replace('/', '.'), false, classLoader); } catch (Exception e) { throw new RuntimeException(e.toString()); } if (c.isAssignableFrom(d)) { return type1; } if (d.isAssignableFrom(c)) { return type2; } if (c.isInterface() || d.isInterface()) { return "java/lang/Object"; } else { do { c = c.getSuperclass(); } while (! c.isAssignableFrom(d)); return c.getName().replace('.', '/'); }}Copy the code

This method is used to find the common parent of the two classes. We can see that it is the type that gets the current class and loads the two inputs. The classLoader used during compilation does not load the code in the Android project, so we need a custom classLoader. Add all the jars received in the Transform mentioned above, along with the class and android.jar, to the custom ClassLoader. (Some of the problems with this method are hinted at in the above method comments.)

The following


public static URLClassLoader getClassLoader(Collection<TransformInput> inputs,
                                            Collection<TransformInput> referencedInputs,
                                            Project project) throws MalformedURLException {
    ImmutableList.Builder<URL> urls = new ImmutableList.Builder<>();
    String androidJarPath  = getAndroidJarPath(project);
    File file = new File(androidJarPath);
    URL androidJarURL = file.toURI().toURL();
    urls.add(androidJarURL);
    for (TransformInput totalInputs : Iterables.concat(inputs, referencedInputs)) {
        for (DirectoryInput directoryInput : totalInputs.getDirectoryInputs()) {
            if (directoryInput.getFile().isDirectory()) {
                urls.add(directoryInput.getFile().toURI().toURL());
            }
        }
        for (JarInput jarInput : totalInputs.getJarInputs()) {
            if (jarInput.getFile().isFile()) {
                urls.add(jarInput.getFile().toURI().toURL());
            }
        }
    }
    ImmutableList<URL> allUrls = urls.build();
    URL[] classLoaderUrls = allUrls.toArray(new URL[allUrls.size()]);
    return new URLClassLoader(classLoaderUrls);
}Copy the code

However, if we just replace the Classloader in getCommonSuperClass, there is still a further hole. We can look at the implementation of getCommonSuperClass. It loads a Class via class.forname and then looks for its parent Class. However, classes in Android.jar cannot be loaded casually. Android.jar is a compile-time dependency for android projects. The runtime uses the Android machine’s own Android.jar. And all of the android.jar methods, including constructors, are empty implementations with only one line of code

throw new RuntimeException("Stub!" );Copy the code

When a class is loaded, its static field is fired, and if a static variable is initialized at declaration time and only a RuntimeException is initialized, an exception is thrown.

So, we can’t get the parent class this way, but can we get the parent class without loading the class? The parent class is also an item in the bytecode of a class, so we can query the parent class from the bytecode. The final implementation looks like this.

public class ExtendClassWriter extends ClassWriter { public static final String TAG = "ExtendClassWriter"; private static final String OBJECT = "java/lang/Object"; private ClassLoader urlClassLoader; public ExtendClassWriter(ClassLoader urlClassLoader, int flags) { super(flags); this.urlClassLoader = urlClassLoader; } @Override protected String getCommonSuperClass(final String type1, final String type2) { if (type1 == null || type1.equals(OBJECT) || type2 == null || type2.equals(OBJECT)) { return OBJECT; } if (type1.equals(type2)) { return type1; } ClassReader type1ClassReader = getClassReader(type1); ClassReader type2ClassReader = getClassReader(type2); if (type1ClassReader == null || type2ClassReader == null) { return OBJECT; } if (isInterface(type1ClassReader)) { String interfaceName = type1; if (isImplements(interfaceName, type2ClassReader)) { return interfaceName; } if (isInterface(type2ClassReader)) { interfaceName = type2; if (isImplements(interfaceName, type1ClassReader)) { return interfaceName; } } return OBJECT; } if (isInterface(type2ClassReader)) { String interfaceName = type2; if (isImplements(interfaceName, type1ClassReader)) { return interfaceName; } return OBJECT; } final Set<String> superClassNames = new HashSet<String>(); superClassNames.add(type1); superClassNames.add(type2); String type1SuperClassName = type1ClassReader.getSuperName(); if (! superClassNames.add(type1SuperClassName)) { return type1SuperClassName; } String type2SuperClassName = type2ClassReader.getSuperName(); if (! superClassNames.add(type2SuperClassName)) { return type2SuperClassName; } while (type1SuperClassName ! = null || type2SuperClassName ! = null) { if (type1SuperClassName ! = null) { type1SuperClassName = getSuperClassName(type1SuperClassName); if (type1SuperClassName ! = null) { if (! superClassNames.add(type1SuperClassName)) { return type1SuperClassName; } } } if (type2SuperClassName ! = null) { type2SuperClassName = getSuperClassName(type2SuperClassName); if (type2SuperClassName ! = null) { if (! superClassNames.add(type2SuperClassName)) { return type2SuperClassName; } } } } return OBJECT; } private boolean isImplements(final String interfaceName, final ClassReader classReader) { ClassReader classInfo = classReader; while (classInfo ! = null) { final String[] interfaceNames = classInfo.getInterfaces(); for (String name : interfaceNames) { if (name ! = null && name.equals(interfaceName)) { return true; } } for (String name : interfaceNames) { if(name ! = null) { final ClassReader interfaceInfo = getClassReader(name); if (interfaceInfo ! = null) { if (isImplements(interfaceName, interfaceInfo)) { return true; } } } } final String superClassName = classInfo.getSuperName(); if (superClassName == null || superClassName.equals(OBJECT)) { break; } classInfo = getClassReader(superClassName); } return false; } private boolean isInterface(final ClassReader classReader) { return (classReader.getAccess() & Opcodes.ACC_INTERFACE) ! = 0; } private String getSuperClassName(final String className) { final ClassReader classReader = getClassReader(className); if (classReader == null) { return null; } return classReader.getSuperName(); } private ClassReader getClassReader(final String className) { InputStream inputStream = urlClassLoader.getResourceAsStream(className + ".class"); try { if (inputStream ! = null) { return new ClassReader(inputStream); } } catch (IOException ignored) { } finally { if (inputStream ! = null) { try { inputStream.close(); } catch (IOException ignored) { } } } return null; }}Copy the code

So far, we introduce Transform+ASM, the two basic technologies of implementing bytecode modification on Android, introduce its principle and application, and analyze performance optimization and adaptation on Android platform. From there, I abstracted a wheel that would allow developers to write a section code plug-in with a small amount of ASM code and not care much about the details behind Transform and ASM. As shown in the

Github.com/Leaking/Hun…

All we need to do is write a plugin to play around with, so let’s take a look at some examples.

The application case

First of all, there are ways to modify bytecode, one is hack code call, one is hack code implementation.

For example, if you change the implementation of the Android Framework (Android.jar), you won’t be able to do this at compile time because the final Classes of the Android Framework are on the Android device. I (TAG, “hello”). You can’t hack the implementation, but you can hack it as hacklog. I (TAG, “seeyou”).

However, if you need to insert a lot of bytecode, you can also reduce the amount of WRITING ASM code by using some tricks. You can abstract most of the logic that can be abstracted into a written class. ASM code then simply writes the statement that calls the written class.

Of course, the above is only a summary based on my experience so far, but there are still some more complex cases to be analyzed on a case-by-case basis. For example, when implementing a function like JakeWharton’s Hugo, I had a difficult problem getting method parameter names at the beginning of the code (a secondary scan was used to solve this problem. You can refer to the project home page for details.

The implementation of okHTTP-Plugin is selected to analyze and demonstrate how to develop a bytecode compilation plug-in using the Huntet framework.

People using OkHttp know that in every OkHttp OkHttp can set their own independent Intercepter/Dns/EventListener (EventListener okhttp3.11 new), But all the need for global OkHttp unified Intercepter/Dns/EventListener is very troublesome, need these Settings, and some of the third party relies on the OkHttp big may not be able to set up. One of the authors thinks that if he were the one, he would use dependency injection to achieve uniform Okhttp configuration. However, this method is only feasible but not ideal. When Posting on Reddit in the background, JakeWharton personally answered the question, face to face, but face to face! After the argument, summarize his position, which goes something like this

He felt as if I said this is okhttp pot, this is a feature of okhttp, however, he felt the global state is a bad coding, so did not provide global Intercepter okhttp the design/Dns/EventListener interface. And the third party dependent libraries cannot set custom Intercepter/Dns/EventListener this is their pot.

I don’t entirely agree with him, however, that while global state is certainly bad design, it’s hard to avoid more or less global intrusion if you’re doing something like performance monitoring. (Although I did misphrase it as if it were an Okhttp pot)

Without further ado, let’s see what we can do with OkHttp


public Builder(){
    this.dispatcher = new Dispatcher();
    this.protocols = OkHttpClient.DEFAULT_PROTOCOLS;
    this.connectionSpecs = OkHttpClient.DEFAULT_CONNECTION_SPECS;
    this.eventListenerFactory = EventListener.factory(EventListener.NONE);
    this.proxySelector = ProxySelector.getDefault();
    this.cookieJar = CookieJar.NO_COOKIES;
    this.socketFactory = SocketFactory.getDefault();
    this.hostnameVerifier = OkHostnameVerifier.INSTANCE;
    this.certificatePinner = CertificatePinner.DEFAULT;
    this.proxyAuthenticator = Authenticator.NONE;
    this.authenticator = Authenticator.NONE;
    this.connectionPool = new ConnectionPool();
    this.dns = Dns.SYSTEM;
    this.followSslRedirects = true;
    this.followRedirects = true;
    this.retryOnConnectionFailure = true;
    this.connectTimeout = 10000;
    this.readTimeout = 10000;
    this.writeTimeout = 10000;
    this.pingInterval = 0;
    this.eventListenerFactory = OkHttpHooker.globalEventFactory;
    this.dns = OkHttpHooker.globalDns;
    this.interceptors.addAll(OkHttpHooker.globalInterceptors);
    this.networkInterceptors.addAll(OkHttpHooker.globalNetworkInterceptors);
}Copy the code

This is the inner class Builder OkhttpClient constructors, our goal is at the end of the method combined with four lines of code, so that all OkhttpClient Shared a common Intercepter/Dns/EventListener. Let’s look at the implementation of OkHttpHooker again

public class OkHttpHooker { public static EventListener.Factory globalEventFactory = new EventListener.Factory() { public EventListener create(Call call) { return EventListener.NONE; }}; public static Dns globalDns = Dns.SYSTEM; public static List<Interceptor> globalInterceptors = new ArrayList<>(); public static List<Interceptor> globalNetworkInterceptors = new ArrayList<>(); public static void installEventListenerFactory(EventListener.Factory factory) { globalEventFactory = factory; } public static void installDns(Dns dns) { globalDns = dns; } public static void installInterceptor(Interceptor interceptor) { if(interceptor ! = null) globalInterceptors.add(interceptor); } public static void installNetworkInterceptors(Interceptor networkInterceptor) { if(networkInterceptor ! = null) globalNetworkInterceptors.add(networkInterceptor); }}Copy the code

In this way, only need to OkHttpHooker advance install several global Intercepter/Dns/EventListener.

Insert four lines of code into the OkhttpClient internal Builder.

First, with Hunter’s framework, we can hide most of the details of the Transform and ASM, and we can just focus on writing the ClassVisitor and MethodVisitor. We need to do the following steps altogether

Create a custom Transform and add it to a custom Gradle Plugin. Inherit the HunterTransform to implement the custom Transform. Implement the custom ClassVisitor and implement the custom MethodVisitor as appropriate

The first step is described in the transform section, which is basically the same short writing method. Let’s start with the second step

Inherit HunterTransform to make your Transform concurrent and incremental.

final class OkHttpHunterTransform extends HunterTransform { private Project project; private OkHttpHunterExtension okHttpHunterExtension; public OkHttpHunterTransform(Project project) { super(project); this.project = project; / / it depends, see if you need a plug-in extension project. GetExtensions (). The create (" okHttpHunterExt, "OkHttpHunterExtension. Class); This.bytecodeweaver = new OkHttpWeaver(); } @Override public void transform(Context context, Collection<TransformInput> inputs, Collection<TransformInput> referencedInputs, TransformOutputProvider outputProvider, boolean isIncremental) throws IOException, TransformException, InterruptedException { okHttpHunterExtension = (OkHttpHunterExtension) project.getExtensions().getByName("okHttpHunterExt"); super.transform(context, inputs, referencedInputs, outputProvider, isIncremental); } // used to control which debug or release packages are used to modify bytecode, Or fully open/close @ Override protected RunVariant getRunVariant () {return okHttpHunterExtension. RunVariant; }} public final class OkHttpWeaver extends BaseWeaver {Override protected ClassVisitor wrapClassWriter(ClassWriter classWriter) { return new OkHttpClassAdapter(classWriter); }} public class OkHttpHunterExtension {public RunVariant RunVariant = runvariable. ALWAYS; @Override public String toString() { return "OkHttpHunterExtension{" + "runVariant=" + runVariant + '}'; }}Copy the code

Okay, so the Transform is that easy to write, so let’s look at the custom ClassVisitor, which is returned in OkHttpWeaver.

Let’s create a new ClassVisitor(custom ClassVisitor for proxy ClassWriter)

public final class OkHttpClassAdapter extends ClassVisitor{ private String className; OkHttpClassAdapter(final ClassVisitor cv) { super(Opcodes.ASM5, cv); } @Override public void visit(int version, int access, String name, String signature, String superName, String[] interfaces) { super.visit(version, access, name, signature, superName, interfaces); this.className = name; } @Override public MethodVisitor visitMethod(final int access, final String name, final String desc, final String signature, final String[] exceptions) { MethodVisitor mv = cv.visitMethod(access, name, desc, signature, exceptions); if(className.equals("okhttp3/OkHttpClient$Builder")) { return mv == null ? null : new OkHttpMethodAdapter(className + File.separator + name, access, desc, mv); } else { return mv; }}}Copy the code

Okhttp3 /OkHttpClient$Builder will have a custom MethodVisitor to handle it

Let’s look at the implementation of this MethodVisitor

public final class OkHttpMethodAdapter extends LocalVariablesSorter implements Opcodes { private boolean defaultOkhttpClientBuilderInitMethod = false; OkHttpMethodAdapter(String name, int access, String desc, MethodVisitor mv) { super(Opcodes.ASM5, access, desc, mv); if ("okhttp3/OkHttpClient$Builder/<init>".equals(name) && "()V".equals(desc)) { defaultOkhttpClientBuilderInitMethod = true; } } @Override public void visitInsn(int opcode) { if(defaultOkhttpClientBuilderInitMethod) { if ((opcode >= IRETURN && opcode <= RETURN) || opcode == ATHROW) { //EventListenFactory mv.visitVarInsn(ALOAD, 0); mv.visitFieldInsn(GETSTATIC, "com/hunter/library/okhttp/OkHttpHooker", "globalEventFactory", "Lokhttp3/EventListener$Factory;" ); mv.visitFieldInsn(PUTFIELD, "okhttp3/OkHttpClient$Builder", "eventListenerFactory", "Lokhttp3/EventListener$Factory;" ); //Dns mv.visitVarInsn(ALOAD, 0); mv.visitFieldInsn(GETSTATIC, "com/hunter/library/okhttp/OkHttpHooker", "globalDns", "Lokhttp3/Dns;" ); mv.visitFieldInsn(PUTFIELD, "okhttp3/OkHttpClient$Builder", "dns", "Lokhttp3/Dns;" ); //Interceptor mv.visitVarInsn(ALOAD, 0); mv.visitFieldInsn(GETFIELD, "okhttp3/OkHttpClient$Builder", "interceptors", "Ljava/util/List;" ); mv.visitFieldInsn(GETSTATIC, "com/hunter/library/okhttp/OkHttpHooker", "globalInterceptors", "Ljava/util/List;" ); mv.visitMethodInsn(INVOKEINTERFACE, "java/util/List", "addAll", "(Ljava/util/Collection;) Z", true); mv.visitInsn(POP); //NetworkInterceptor mv.visitVarInsn(ALOAD, 0); mv.visitFieldInsn(GETFIELD, "okhttp3/OkHttpClient$Builder", "networkInterceptors", "Ljava/util/List;" ); mv.visitFieldInsn(GETSTATIC, "com/hunter/library/okhttp/OkHttpHooker", "globalNetworkInterceptors", "Ljava/util/List;" ); mv.visitMethodInsn(INVOKEINTERFACE, "java/util/List", "addAll", "(Ljava/util/Collection;) Z", true); mv.visitInsn(POP); } } super.visitInsn(opcode); }}Copy the code

Okhttp3 /OkHttpClient$Builder (); OkHttpClient$Builder (); OkHttpClient$Builder ();

At this point, we just need to publish the plug-in and apply it to our project.

Framework with the help of a Hunter, we easily successful hacks Okhttp, we can use the global unification Intercepter/Dns/EventListener to monitor our web APP. Also, as an aside, one might say that not all network requests are written through Okhttp, but also through URLConnection, as I mentioned in another article

Quinnchen. Me / 2017/11/18 /…

This article explains how to direct URLConnection requests to your own OkhttpClient.

At this point, I’ve covered the entire process of using the Hunter framework to develop a bytecode compilation plug-in that can do whatever it wants with third-party dependencies. If you are confused about the code, go to the project home page for the complete code, as well as the implementation of several other plug-ins. I’ll write another article about the implementation of the other plug-ins when I have time.

conclusion

This is the end of the article, which focuses on Hunter, analyzes how to develop an efficient compiler plug-in to modify bytecode, and some of the workflow and development routines associated with ASM bytecode technology.

Welcome to use the Hunter framework to develop plug-ins and use the existing plug-ins. Welcome to mention issues and star/fork.

References and article recommendations

In-depth understanding of the Java virtual machine

ASM GUIDE

A good bytecode technology PPT