Editor’s note: This article is for ctrip wireless Foundation team to introduce their open source Android dynamic loading solution DynamicAPK. One of the authors of this article, Chen Haoran, Ctrip wireless RESEARCH and development director, will share relevant content on architecture optimization at ArchSummit 2015 Beijing Architect Conference, welcome to follow.

The plug-in and dynamic loading framework of Ctrip Android App has been online for half a year. After initial exploration and continuous polishing and optimization, the new framework and engineering configuration have withstood the test of production practice. This paper will introduce the principle and implementation details of plug-in development and dynamic loading technology of Android platform in detail, review the evolution process of Ctrip Android App architecture, and hope that our experience can help more Android engineers.

Demand driven

In 2014, with the need of business development and the division of Ctrip’s wireless department, each business product module was assigned to each business BU, and the original Ctrip wireless App development team was divided into several development teams, including basic framework, hotel, air ticket and train ticket, thus the development and release of Ctrip App entered a new mode. In this mode, the cost of development communication is greatly increased, and the previous collaboration mode is not sustainable. New development mode and technology are needed to solve the demand problem.

On the other hand, from the technical point of view, Ctrip as early as 2012 hit the Android platform in the history of the most pit ceiling (no one) : 65535 method number problems. The old scheme was to put all the third-party libraries into the second dex and use the hack method discovered by Facebook to expand the point LinearAllocHdr allocation space (from 5M to 8M), but with the expansion of the code, the old scheme gradually became inadequate. To dismantle or not to dismantle, is not even a consideration, continue to dismantle DEX is our only way out. The question is: what’s smarter?

Secondly, with the impact of organizational structure adjustment, the quality control of our App has been greatly challenged. Such tension and pressure has made our development team exhausted. Is there no way to solve this fundamental flaw in Native architecture, except to envy the ability of front-end colleagues to publish updates online? NO! The added benefit of plug-in dynamic loading is the client’s hot deployment capability.

These fundamental requirements show how much we can benefit from a plug-in dynamic loading architecture solution, and there are many other benefits:

  • Improved compilation speed

    After the project was broken down into dozens of sub-projects, the shortcomings of Android Studio compilation process were quickly amplified, in Win7 mechanical hard disk development machine compilation time once exceeded 1 hour, the appalling slow compilation let developers complain (of course now switch to Mac+SSD much faster).

  • Start speed increase

    In the MultiDex solution provided by Google, all dex decompression, dexopt and loading operations will be performed in the main thread. This is a very long process, and users will obviously see a long black screen, which is more likely to cause the ANR of the main thread and lead to initial startup failure.

  • A/B Testing

    The AB version of the module can be developed independently, rather than the AB version code in the same module.

  • Optional modules are downloaded on demand

    For example, modules used for debugging functions can be downloaded and loaded when necessary to reduce the Size of App

With so many pain points on the list, the kids are already feeling overwhelmed, aren’t they? Without further ado, let’s begin our exploration of plug-in dynamic loading architecture.

The principle of

The software industry has had plenty of user education about the idea of plug-ins. Whether it is the daily use of the browser, or to accompany programmers countless days and nights of Eclipse, even QQ behind, have the support of plug-in technology. There are two main issues to consider when implementing plug-ins on Android:

  • Compile time: Compilation of resources and code
  • Runtime: Loading of resources and code

After the above two key issues are solved, how to implement the specific interface of the plug-in becomes a matter of personal technical preferences or specific requirement scenarios. Now we will solve the above key problems one by one, the most troublesome is the compilation and loading of resources.

How is Android compiled?

First, let’s review how Android compiles. See the picture below:

View image (click image to enlarge)

The whole process is huge and complex. We mainly focus on several key links: AAPT, JAVAC, ProGuard and DEX. Relevant links involved in the input and output are highlighted in bold on the map.

Compilation of resources

Android resource compilation relies on a powerful command line tool, aapt, which is located at /build-tools//aapt and has a number of command-line arguments, several of which are worth paying special attention to:

  • -I add an existing package to base include set

    This parameter appends an existing package to the dependent path. In Android, resource compilation also requires dependencies, and the most common dependency is the Android.jar that comes with the SDK itself. Jar is not a common JAR package. It not only contains the existing SDK class library, but also contains the SDK’s own compiled resources and resource index table resources.arsc file. In daily development, we often refer to SDK resources in the form of @Android :color/opaque_red. This all comes from aapt’s reference to android.jar dependencies during compilation. Similarly, we can use this parameter to reference an existing APK package as a dependent resource for compilation.

  • -G A file to output proguard options into.

    In resource compilation, class names and method references to components will result in run-time reflection calls, so these symbolic quantities should not be obfuscated or whittled down during code obfuscation, otherwise the classes and methods referenced in the layout file will not be found at runtime. The -g method exports the classes and interfaces found during resource compilation that must be kept, which will participate in the later obfuscation phase as an append configuration file.

  • -J specify where to output R.java resource constant definitions

In Android, all resources generate constant ids at the Java source level, and these ids are recorded in the R.Java file for subsequent code compilation. In the R.java file, Android resources generate ids for all resources during compilation and are stored as constants in the R class for reference by other code. Each four-byte int resource ID generated in class R actually consists of three fields. The first byte represents the Package, the second byte is the classification, and the third and fourth bytes are the in-class ids. Such as:

// Android.jar, whose PackageID is 0x01 public static final int Cancel = 0x01040000; PackageID = 0x7F public static final int zip_code = 0x7F090f2e;Copy the code

After we modify aAPT, we can assign different header packageIDS to the resources in each child APK, so that there is no conflict with each other.

Compilation of code

You should be familiar with compiling Java code, just pay attention to the following:

  • classpath

    The classpath is used to specify which directories, files, and JARS to look for dependencies in Java source compilation.

  • Confusion.

    Most Android projects are obfuscated for security purposes. Refer to the Proguard manual for confusion principles and configurations.

With this background, we can think about and design the fundamentals and main flow of a plug-in dynamic loading framework.

implementation

The implementation is divided into two categories: 1. Compilation process transformation for plug-in sub-project, 2. Dynamic loading transformation at runtime (host program dynamic loading plug-in, there are two barriers to break: how to access resources, how to access code).

Plug-in resource compilation

For resource compilation of plug-ins, we need to consider the following points:

  • The host apK is referenced with the -i parameter.

    From this, you can use the host’s resources and controls and layout classes in the plug-in’s resources and XML layout.

  • Add –apk-module parameter to AAPT.

    As mentioned earlier, the resource ID actually has an internal field for PackageID. We specify a unique PackageID field for each plug-in project so that it is easy to tell by the resource ID which plug-in APK the resource needs to be found and loaded from. More on this later in the resource loading section.

  • Add — public-r-path to aAPT.

    Referring to a system resource can refer to a specific ID using the fully qualified name of its R class android.r to distinguish it from the R class in the current project, as is usual with resources in the Android.jar package. Plugin references to host resources can also be done using base.package.name.r. However, due to historical reasons, the “plug-in” codes of each sub-BU are decoupled and independent from the main APP, and R of the current project is still directly used for resource reference. If the standard mode is changed, R in the current large amount of legacy code needs to be changed to base.r as appropriate, which is large and error-prone, and the future use of BU developers is also a little less “transparent”. So we made a concession in the design, and added the — public-r-path parameter to specify the location of the base.R for AAPT, let it define the base resource ID in the R class of the plug-in during compilation, so that the plug-in project can be as before, no matter whether the resource comes from the host or itself. Just use it. Of course, the side effect of this is that the host and plug-in resources should not have the same name, which is relatively easy to understand through the development specification.

Plug-in code compilation

When compiling code for plug-ins, consider the following:

  • classpath

    For the compilation of the plug-in, in addition to relying on android.jar and its own third-party libraries, it also needs to rely on the exported base.jar class library of the host. There is also a requirement for host obfuscation: all public/protected hosts can be dependent on plug-ins, so none of these interfaces can be obfuscated.

  • Confusion.

    When the plug-in project is confused, of course, it also imports the host’s confused jar package as a reference library.

At this point, the technical plan for all the important steps at compile time has been decided, and all that remains is to import the plug-in APK into the base-.apk that was generated earlier and re-align the signature.

All we need is the show. Let’s take a look at how plug-ins come to the fore at runtime.

Loading of runtime resources

Normally we use Resources through the AssetManager class and Resources class to access. The methods that get them are in the Context class.

Context.java

/** Return an AssetManager instance for your application's package. */
public abstract AssetManager getAssets();

/** Return a Resources instance for your application's package. */
public abstract Resources getResources();Copy the code

They are two abstract methods, implemented in the ContextImpl class. After the Resources object is initialized in the ContextImpl class, subsequent subclasses of the Context, including activities, services, and other components, can read Resources using these two methods.

ContextImpl.java

private final Resources mResources;

@Override
public AssetManager getAssets() {
   return getResources().getAssets();
}

@Override
public Resources getResources() {
   return mResources;
}Copy the code

Now that we know from which APK a resource ID should be read (previously at compile time we marked the package to which the resource belongs in the first byte of the resource ID), we can simply rewrite these two abstract methods to guide the application to the right place to read the resource.

As for reading resources, AssetManager has a hidden method, addAssetPath, to add resource paths to AssetManager.

/**
* Add an additional set of assets to the asset manager.  This can be
* either a directory or ZIP file.  Not for use by applications.  Returns
* the cookie of the added asset, or 0 on failure.
* {@hide}
*/
public final int addAssetPath(String path) {
   synchronized (this) {
       int res = addAssetPathNative(path);
       makeStringBlocks(mStringBlocks);
       return res;
   }
}Copy the code

We simply call this method by reflection, and then tell the AssetManager class the location of the plug-in APK, and it will complete the task of loading resources based on resources. Arsc and compiled resources in the APK.

We can already load plugin resources, but with a bunch of custom classes. One more step is needed for a “seamless” experience: use Instrumentation to take over the creation of all activities, services, and other components (including, of course, the Resources classes they use).

Saying the Activity, Service, such as system components, can from android. The app. ActivityThread class in the main thread of execution. The ActivityThread class has a member called the mInstrumentation, which is responsible for creating activities and so on, and this is the perfect time to inject our modified resource class. InstrumentationHook by tampering the mInstrumentation to our own InstrumentationHook, and then accidentally replacing its mResources class with our DelegateResources each time we create an Activity, Every Activity created in the future has a resource loading class that understands plug-ins and delegates.

Of course, all of the above substitutions work on the Application Context.

Loading of runtime classes

Loading classes is relatively simple. Similar to the runtime classpath concept for Java programs, the Android system default class loader, PathClassLoader, also has a member, pathList, which is essentially a List, as the name implies, and the runtime looks for classes to load from each dex path in between. Since it’s a List, you might want to append a bunch of dex paths to it. In fact, Google’s official MultiDex library is based on this principle. The following code snippet shows the details of modifying the pathList path:

MultiDex.java

private static void install(ClassLoader loader, List additionalClassPathEntries,
     File optimizedDirectory)
             throws IllegalArgumentException, IllegalAccessException,
             NoSuchFieldException, InvocationTargetException, NoSuchMethodException {
    /* The patched class loader is expected to be a descendant of
    * dalvik.system.BaseDexClassLoader. We modify its
    * dalvik.system.DexPathList pathList field to append additional DEX
    * file entries.
    */
    Field pathListField = findField(loader, "pathList");
    Object dexPathList = pathListField.get(loader);
    expandFieldArray(dexPathList, "dexElements", makeDexElements(dexPathList,
         new ArrayList(additionalClassPathEntries), optimizedDirectory));
}Copy the code

Of course, for different Android versions, the class loading method is slightly different, you can refer to the MultiDex source code to do the specific difference processing.

So far, the four fundamental problems put forward before, have had specific solutions. All that’s left is coding!

Coding is mainly divided into three parts:

  • Changes to the AAPT tool.
  • Gradle package script implementation.
  • Implementation of runtime load code.

See DynamicAPK, our open source project on GitHub.

Benefits and Costs

Everything has its two sides, especially for dynamic loading, which uses unofficial Hack technology, benefits and costs need to be considered clearly in the planning stage, so as to facilitate the review after completion.

earnings

  • Plug-in architecture ADAPTS to the needs of existing organizational structure and development pace. BU achieves high cohesion and low coupling not only from the code level but also from the project control level, which greatly reduces communication cost and improves work efficiency.
  • After splitting into several small plug-ins, Dex said goodbye to the ceiling.
  • HotFix is the last guarantee for app quality, there is no irreparable loss, and now the level of HotFix granularity is controllable, that is, the traditional class level (directly using the pathClassLoader implementation), can also be apK level with resources.
  • ABTesting breaks away from the ugly old if/else implementation and loads multiple solutions on demand.
  • The compilation speed is greatly improved, and each BU only needs to use the compilation results of the host to update and compile its own sub-project, which can be done in minutes.
  • The APK of App host is greatly reduced, the background loading or lazy loading of each business module as required, and the startup speed is optimized, so as to bid farewell to the black screen and start ANR.
  • Each BU plug-in apK independent, who is fat and thin at a glance, app size control targeted.

The above income has basically reached or even exceeded the expected goal of the project: D

The price

  • Resources alias

    Android provides powerful resource alias rules for more details. Unfortunately, using the resource alias on some models like Samsung S6 will cause the problem that the resource cannot be found due to the confusion of host resource and plug-in resource ID. The only option is to ban the use of this technology. Fortunately, abandoning this advanced feature will not cause fundamental loss.

  • The nuptial resources

    For the reasons mentioned earlier, the host resource ID is copied in the plug-in in its entirety. Without the protection of the package name namespace, resources with same names can directly cause conflicts. For the time being, avoid naming conventions. Fortunately, good naming habits should also be done by each developer, so the solution costs less.

  • Many controls use enumerations to constrain the value range of properties. Unfortunately, Android enumerations uniquely identify id constants generated in R using names, regardless of namespace or control ownership. For the same reason mentioned above, enumerations of the same name within the host and plug-in cause ID conflicts. For the time being, it is also circumvented by naming conventions.

  • External access to resources.

    For the rare occasions that apK resources need to be accessed from the outside (such as sending delay notification), the App has not been started at this time, and the resource acquisition is done by the system on behalf of the system, so it is naturally impossible to know the resource location and acquisition method of the internal plug-in. There is nothing that can be done about this situation, so we specifically allow such resources to be placed directly within the host APK.

These costs, either innocuous or very low cost alternatives, are within acceptable limits.

The future optimization

There are also some advanced features that have not yet been implemented due to prioritization, but have been added to the optimization agenda as development requirements for each line of business have been added, such as:

  • The plug-in project supports the SO library.
  • The plugin project supports lib project dependencies, AAR dependencies, Maven remote dependencies and other advanced dependencies.
  • IDE friendly, so that developers can more easily generate plug-in APK.

Open source

After the above introduction, I believe that you have a preliminary understanding of Ctrip Android plug-in development and dynamic loading scheme. For details, visit GitHub open source project DynamicAPK. Ctrip wireless basic RESEARCH and development team will continue to work hard in the future to share more practical experience of the project.