1. Status quo of hotfix framework

At present, the HotFix framework mainly includes QZONE patch, HotFix, Tinker, Robust and so on. According to the principle, the hotfix framework can be roughly divided into three categories:

  1. Interacted with ClassLoader to load dex based on multidex mechanism
  2. Native replacement method structure
  3. Instant-run piling scheme

Qzone patch and Tinker are both used in scheme one; Alibaba’s AndFix uses solution two; Meituan’s Robust uses scheme 3.

1. QQ space patch principle

Generate patch.dex for the patch class. When the app is started, use reflection to obtain the ClassLoader of the current application, that is, BaseDexClassLoader. [] dexElements, denoted as elements1; Then the current application ClassLoader is used as the parent ClassLoader to construct a DexClassLoader of patch.dex. In general, the corresponding Element[] dexElements can be obtained by reflection, denoted as elements2. Put elements2 before elementS1, and then call the method loadClass that loads the class.

The hidden technical difficulty, the CLASS_ISPREVERIFIED problem

The DEX file is verified and optimized during apK installation. This operation can directly load the Odex file when the app is running, which can reduce the memory usage and accelerate the startup speed. If there is no ODEX operation, you need to extract the DEX from the APK package before running.

During the verification process, if all the calling relationships of a class are in the same DEX file, the class will be marked CLASS_ISPREVERIFIED, indicating that the class has been pre-verified. If the class is verified by CLASS_ISPREVERIFIED but the called classes are not in the same dex file, an exception is thrown.

To solve this problem, qzone’s solution is to prepare an AntilazyLoad class that is packaged separately as a hack.dex, and then add code like this to all class constructors:

if (ClassVerifier.PREVENT_VERIFY) {
   System.out.println(AntilazyLoad.class);
}
Copy the code

In this way, each class will have the problem of AntilazyLoad in another DEX file during the oDEX process, so the oDEX verification process will not continue, which will sacrifice the DVM’s dex optimization effect.

2. Tinker principle

For Tinker, the apK before and after repair are defined as APK1 and APk2 respectively. Tinker has developed a set of dex file differential merge algorithm. When the patch package is generated, a subcontract patch.dex is generated. Tinker will start a thread to merge the class.dex and patch.dex of the old APK to generate a new class.dex and store it in the local directory.

3. The principle of AndFix

AndFix fixes by replacing the structure of the method. In the native layer, we get Pointers to the pre-fixed class and the post-fixed class, and then point the old method’s property pointer to the new method. Because the method structures are different in different system versions and the processing methods of Davilk and ART virtual machines are different, it is necessary to replace the method structures for different systems.

├─ art │ ├─ art. H │ ├─ art_4_4.h │ ├─ art_5_0.h │ ├─ art_5_0.h │ ├─ bass exercises Art_5_1. H │ ├ ─ art_6_0. H │ ├ ─ art_7_0. H │ ├ ─ art_method_replace. CPP │ ├ ─ art_method_replace_4_4. CPP │ ├ ─ │ ├─ Art_method_replace_5_0.cpp │ ├─ art_method_replace_5_0.cpp │ ├─ Art_method_replace_6_0.cpp │ ├─ 7_0. CPP ├─ common. H ├─ dalvik ├─ dalvikCopy the code

Second, meituan Robust thermal repair program principle

Now, let’s move on to today’s topic, Robust hotfixes. First, introduce the implementation principle of Robust.

Take the State class as an example

public long getIndex(a) {
    return 100L;
}
Copy the code

State class after piling

public static ChangeQuickRedirect changeQuickRedirect;
public long getIndex(a) {
    if(changeQuickRedirect ! =null) {
        // The PatchProxy package contains the logic to get the current className and methodName, and finally calls the changeQuickRedirect function within it
        if(PatchProxy.isSupport(new Object[0].this, changeQuickRedirect, false)) {
            return ((Long)PatchProxy.accessDispatch(new Object[0].this, changeQuickRedirect, false)).longValue(); }}return 100L;
}
Copy the code

We generate a StatePatch class, create an instance and reflect the changeQuickRedirect variable assigned to State.

public class StatePatch implements ChangeQuickRedirect {
    @Override
    public Object accessDispatch(String methodSignature, Object[] paramArrayOfObject) {
        String[] signature = methodSignature.split(":");
        // The getIndex method corresponds to a
        if (TextUtils.equals(signature[1]."a")) {//long getIndex() -> a
            return 106;
        }
        return null;
    }

    @Override
    public boolean isSupport(String methodSignature, Object[] paramArrayOfObject) {
        String[] signature = methodSignature.split(":");
        if (TextUtils.equals(signature[1]."a")) {//long getIndex() -> a
            return true;
        }
        return false; }}Copy the code

When we execute the code getState in question, we execute the logic in StatePatch instead. This is the core principle of Robust. Since there is no interference in the dex loading process of the system, this scheme is the most compatible.

Robust implementation details

The implementation of Robust is very simple. If we just have a simple understanding of it, there are many details that we will not realize without contact. The implementation of Robust can be divided into three parts: piling, generating patch package, and loading patch package. Let’s start with piling.

1. The pile

Robust defines a configuration file called RobustTransform that specifies whether to enable pilings, which packages need pilings, and which packages do not need pilings. The RobustTransform plug-in automatically traverses all classes when compiling the Release package. And perform the following operations on the class according to the rules specified in the configuration file:

  1. Class to add a static variableChangeQuickRedirect changeQuickRedirect
  2. Insert a piece of code in front of the method, executing the method in the fix pack if it needs to be patched, or executing the original logic if it is not.

Common bytecode manipulation frameworks are:

  • ASM
  • AspectJ
  • BCEL
  • Byte Buddy
  • CGLIB
  • Cojen
  • Javassist
  • Serp

Meituan Robust uses ASM and Javassist respectively to implement the operation of inserting piles to modify bytecode. Personally, I find JavaAssist a little easier to understand. The following code analysis uses JavaAssist manipulating bytecode as an example.

for (CtBehavior ctBehavior : ctClass.getDeclaredBehaviors()) {
    Step 1: add the static variable changeQuickRedirect
    if(! addIncrementalChange) {//insert the field
        addIncrementalChange = true;
        // Create a static variable and add it to ctClass
        ClassPool classPool = ctBehavior.getDeclaringClass().getClassPool();
        CtClass type = classPool.getOrNull(Constants.INTERFACE_NAME);  // com.meituan.robust.ChangeQuickRedirect
        CtField ctField = new CtField(type, Constants.INSERT_FIELD_NAME, ctClass);  // changeQuickRedirect
        ctField.setModifiers(AccessFlag.PUBLIC | AccessFlag.STATIC);
        ctClass.addField(ctField);
    }
    // Determine that this method needs fixing
    if(! isQualifiedMethod(ctBehavior)) {continue;
    }
    // Step 2: Insert a piece of code before the method...
}
Copy the code

For inserting a piece of code before a method,

// Robust gives each method a unique ID
methodMap.put(ctBehavior.getLongName(), insertMethodCount.incrementAndGet());
try {
    if (ctBehavior.getMethodInfo().isMethod()) {
        CtMethod ctMethod = (CtMethod) ctBehavior;
        booleanisStatic = (ctMethod.getModifiers() & AccessFlag.STATIC) ! =0;
        CtClass returnType = ctMethod.getReturnType();
        String returnTypeString = returnType.getName();
        // The body is the logic that precedes the method
        String body = "Object argThis = null;";
        // In Javaassist, $0 represents the current instance object and equals this
        if(! isStatic) { body +="argThis = $0;";
        }
        String parametersClassType = getParametersClassType(ctMethod);
        // In Javaassist, the $args expression represents an array of method arguments. You can see that the isSupport method passed these arguments: All parameters of the method, current object instance, changeQuickRedirect, whether the method is static, current method ID, type of all parameters of the method, return type of the method
        body += " if (com.meituan.robust.PatchProxy.isSupport($args, argThis, " + Constants.INSERT_FIELD_NAME + "," + isStatic +
                "," + methodMap.get(ctBehavior.getLongName()) + "," + parametersClassType + "," + returnTypeString + ".class)) {";
        // getReturnStatement returns the code that executes the methods in the patch pack
        body += getReturnStatement(returnTypeString, isStatic, methodMap.get(ctBehavior.getLongName()), parametersClassType, returnTypeString + ".class");
        body += "}";
        // Finally, insert the body we wrote into the pre-execution logic of the methodctBehavior.insertBefore(body); }}catch (Throwable t) {
    //here we ignore the error
    t.printStackTrace();
    System.out.println("ctClass: " + ctClass.getName() + " error: " + t.getMessage());
}
Copy the code

Now look at the getReturnStatement method,

 private String getReturnStatement(String type, boolean isStatic, int methodNumber, String parametersClassType, String returnTypeString) {
        switch (type) {
            case Constants.CONSTRUCTOR:
                return " com.meituan.robust.PatchProxy.accessDispatchVoid( $args, argThis, changeQuickRedirect, " + isStatic + "," + methodNumber + "," + parametersClassType + "," + returnTypeString + "); ";
            case Constants.LANG_VOID:
                return " com.meituan.robust.PatchProxy.accessDispatchVoid( $args, argThis, changeQuickRedirect, " + isStatic + "," + methodNumber + "," + parametersClassType + "," + returnTypeString + "); return null;";
            // Omitted other return type processing}}Copy the code

PatchProxy. AccessDispatchVoid finally call changeQuickRedirect accessDispatch.

This is the end of the piling process.

2. Generate the patch package

Robust defines an Modify annotation,

@Target({ElementType.FIELD, ElementType.METHOD, ElementType.TYPE, ElementType.CONSTRUCTOR})
@Retention(RetentionPolicy.CLASS)
@Documented
public @interface Modify {
    String value(a) default "";
}
Copy the code

For the method you want to fix, add the Modify annotation directly to the method declaration

@Modify
public String getTextInfo() {
    getArray();
    //return "error occur " ;
    return "error fixed";
}
Copy the code

During compilation, the Robust iterates through all classes one by one. If the class has methods that need fixing, the Robust generates an xxPatch class:

  1. The first step is to clone the Patch class according to the bug class, and then delete the classes that do not need to be patched. Why use the delete method instead of the add method? Easier to delete)
  2. The second step is to create a constructor for Patch that will receive the instance object of the bug class.
  3. Walk through all the methods in the Patch class, using ExprEditor + reflection to modify the expression.
  4. Delete all variables and superclasses in Patch.

So here’s an example of why this is such a hassle.

public class Test {
    private int num = 0;
    public void increase(a) {
        num += 1;
    }
    public void decrease(a) {
        // This subtraction is wrong
        num -= 2;
    }
    public static void main(String[] args) {
        Test t1 = new Test();
        // Num =1 is executed
        t1.increase();
        // Num =2
        t1.increase();
        Decrease (num=0); decrease (num=0)t1.decrease(); }}Copy the code

Therefore, when we deliver the patch, the num operation is also the NUM operation on the T1 object. This is why we need to create a constructor that accepts bug class instance objects. Let’s look at how we migrate all calls to TestPatch variables, methods, and so on from TestPatch class to Test. This requires the use of ExprEditor (expression editor).

// This method is the same as TestPatch
method.instrument(
    new ExprEditor() {
        // Handle variable access
        public void edit(FieldAccess f) throws CannotCompileException {
            if (Config.newlyAddedClassNameList.contains(f.getClassName())) {
                return;
            }
            Map memberMappingInfo = getClassMappingInfo(f.getField().declaringClass.name);
            try {
                // If the variable is read, then replace f with the return expression in parentheses using the replace method
                if (f.isReader()) {
                    f.replace(ReflectUtils.getFieldString(f.getField(), memberMappingInfo, temPatchClass.getName(), modifiedClass.getName()));
                }
                // Write data to variables
                else if(f.isWriter()) { f.replace(ReflectUtils.setFieldString(f.getField(), memberMappingInfo, temPatchClass.getName(), modifiedClass.getName())); }}catch (NotFoundException e) {
                e.printStackTrace();
                throw newRuntimeException(e.getMessage()); }}})Copy the code

ReflectUtils. GetFieldString method invocation result is to generate a list of string like this:

\$_=(\$r) com.meituan.robust.utils.EnhancedRobustUtils.getFieldValue(fieldName, instance, clazz)

In this way, calls to the num variable in TestPatch will be converted to calls to the num variable in the original bug-class object T1 through reflection during compilation.

In addition to variable access to FieldAccess, ExprEditor has these cases that require special handling.

public void edit(NewExpr e) throws CannotCompileException {}public void edit(MethodCall m) throws CannotCompileException {}public void edit(FieldAccess f) throws CannotCompileException {}public void edit(Cast c) throws CannotCompileException {}Copy the code

There are so many situations to deal with that the author of Robust can’t help laughing: shit!! too many situations need take into consideration

After the Patch class is generated, the Robust will generate a ChangeQuickRedirect class from the template class. The template class code is as follows:

public class PatchTemplate implements ChangeQuickRedirect {
    public static final String MATCH_ALL_PARAMETER = "(\\w*\\.) *\\w*";

    public PatchTemplate(a) {}private static final Map<Object, Object> keyToValueRelation = new WeakHashMap<>();

    @Override
    public Object accessDispatch(String methodName, Object[] paramArrayOfObject) {
        return null;
    }

    @Override
    public boolean isSupport(String methodName, Object[] paramArrayOfObject) {
        return true; }}Copy the code

For example, the Test class generates a ChangeQuickRedirect class named TestPatchController, which will add filtering logic before isSupport methods during compilation.

// Check whether the patch method is executed based on the method ID
public boolean isSupport(String methodName, Object[] paramArrayOfObject) {
    return "23:".contains(methodName.split(":") [3]);
}
Copy the code

After these two classes are generated, a mapping to maintain the bug class -> ChangeQuickRedirect class is generated

public class PatchesInfoImpl implements PatchesInfo {
    public List getPatchedClassesInfo(a) {
        ArrayList arrayList = new ArrayList();
        arrayList.add(new PatchedClassInfo("com.meituan.sample.Test"."com.meituan.robust.patch.TestPatchControl"));
        EnhancedRobustUtils.isThrowable = false;
        returnarrayList; }}Copy the code

For example, one method fix for a class generates a patch. The fix pack contains three files:

  • TestPatch
  • TestPatchController
  • PatchesInfoImpl

The generated patch package is in JAR format, and we need to use Jar2dex to convert the JAR package into dex package.

3. Load the patch package

When the online app produces bugs, you can notify the client to pull the corresponding patch package. After the patch package is downloaded, a thread will be opened to perform the following operations:

  1. Use the DexClassLoader to load the external dex file, which is the patch package we generated.
  2. Reflection for PatchesInfoImpl patches in the mapping relationship, such as PatchedClassInfo (” com. At meituan. Sample. The Test “, “com. At meituan. Robust. Patch. TestPatchControl”).
  3. TestPatchControl is instantiated and assigned to changeQuickRedirect

At this point, the bug is fixed and takes effect without a restart.

4. Some questions

A. Robust causes the Proguard method to fail inline

Proguard is a code optimizer and obfuscation tool. Proguard optimizes your program so that if a method is short or only called once, it inline the method’s internal logic to the calling place. The solution of Robust is to find the inline method and not to insert the pile in the inline method.

B. Lambada expression repair

For lambada expressions that cannot be annotated directly, the Robust provides a class RobustModify. The modify method is null. During recompilation, use ExprEditor to check if the RobustModify class is invoked. I think it needs fixing.

new Thread(
        () -> {
            RobustModify.modify();
            System.out.print("Hello");
            System.out.println(" Hoolee");
        }
).start();
Copy the code

C. Robust generation of method ids is achieved by incrementing ids through all classes and methods during compilation

A method that can be uniquely identified by class name + method name + parameter type. My own scheme is to assemble these three data into class name @ method name # parameter type MD5, support lambada Expressions (com) orzangleli) demo. Test# lambda $execute $0 @ 2 ab6d5a5d73bad3848b7be22332e27ea). Based on the core principles of Robust, I copied a hot fix framework called Anivia.

Github.com/hust2010107…

Four,

The first step is to recognize the work of various hotfix developers and organizations in the country. It is not easy to do a good job with hotfix solutions. Secondly, it is worth learning from other people’s problems in the implementation of hotfixes. When these developers encounter a problem, they will go back to the root cause of the problem and then think about a solution.