Use BsDiff for incremental updates

On Android, we usually download the complete installation package for in-app updates and install it. However, when the installation package is large, each update will make users uncomfortable, because not only will it consume a lot of traffic, but when the user’s network is not very good, the update will be slow, and the user experience will be affected, such as the bandwidth consumption during the download, resulting in the slow loading of images. As a result, users are likely to reject updates.

Bsdiff is a difference algorithm that generates a difference file based on the difference between two files, and then regenerates a new file based on the old file and the difference file. The app looks like this on Android: The user installs v1.0 version, and when updating V2.0 version, the server generates a differential package patch according to V1.0 and V2.0. Then the user downloads patch when prompted to update, and then installs and updates the V2.0 version based on the installed v1.0 and patch locally.

Compile the Bsdiff used by the server

On the server side, it is possible to install bsdiff directly, but to keep the bsdiff version consistent with the version in the application, you compile it yourself.

Download the source code

Download bsdiff source code: the official website address, but the official website download time actually prompt 403. So I uploaded a copy to Github, which you can download from Github or here.

Then download the source code for bzip2: from SourceForge, as BsDiff needs to use bzip2.

Begin to compile

Windows compilation is cumbersome, the environment and tools are lacking, and bSdiff references some Linux header files. So compile in Linux.

Start by unpacking bsdiff and bzip2 and placing them in the same directory.

. ├ ─ ─ bsdiff 4.3 │ ├ ─ ─ bsdiff. 1 │ ├ ─ ─ bsdiff. C │ ├ ─ ─ bspatch. 1 │ ├ ─ ─ bspatch. C │ └ ─ ─ a Makefile ├ ─ ─ bzip2-1.0.6...Copy the code

Then modify the Makefile in bsdiff, since bsdiff references the bzip2 header and library files, we need to point the search path to our unzipped bzip2-1.0.6. At the same time, there are some formatting issues in the Makefile that also need to be fixed. The modified Makefile looks like this:

BZIP2PATH=.. /bzip2-1.0.6 CC= GCC CFLAGS += -o3 -lbz2 -l ${BZIP2PATH} -i ${BZIP2PATH} PREFIX? = /usr/local INSTALL_PROGRAM ? = ${INSTALL} -c -s -m 555 INSTALL_MAN ? = ${INSTALL} -c -m 444all: bsdiff bspatch
bsdiff: bsdiff.c
	$(CC) bsdiff.c $(CFLAGS) -o bsdiff
bspatch: bspatch.c
	$(CC) bspatch.c $(CFLAGS) -o bspatch

install:
        ${INSTALL_PROGRAM} bsdiff bspatch ${PREFIX}/bin
        .ifndef WITHOUT_MAN
        ${INSTALL_MAN} bsdiff.1 bspatch.1 ${PREFIX}/man/man1
        .endif
Copy the code

Not much changed, first add a BZIP2PATH parameter and point to the bzip2 path, then specify the library file search directory -l ${BZIP2PATH} and header file search path -i ${BZIP2PATH} as bzip2 path in CFLAGS. The second is to specify the compiler as GCC and add explicit generated commands to bsdiff and bspatch. Finally, a TAB indent was added before the. Ifndef and. Endif commands in install.

In CFLAGS, the bz2 library is linked using -lbz2, so it needs to be libbz2.a. Cut to the bzip2-1.0.6 directory and execute the command:

# since only libbz2.a is required, there is no need to compile the rest
make libbz2.a
Copy the code

In bzip2-1.0.6, you can see that libbz2.a is generated, and then cut back to the bsdiff-4.3 directory and execute the command:

make
Copy the code

At this point, the bsdiff and bspatch executables are generated in the bsDIFF-4.3 directory. In fact, we do not need the executable file bspatch, because the composition step is done on the phone, and the server only needs to use BSdiff to generate the Patch difference file.

So you can use the command: make bsdiff to generate only bsdiff executables.

Generating difference files

Use the compiled Bsdiff to generate the difference file, followed by three parameters, the first is the old version of the file, the second is the new version of the file, and the third is the generated difference file:

./bsdiff app-v1.apk app-v2.apk patch
Copy the code

After the above command is executed, the patch file will be generated, which should be smaller than app-v2.apk. When updating, users only need to download the patch file. That’s all the server needs to do, compile BSdiff and generate the difference file.

Use the Bspatch composite installation package in Android

Bspatch is the executable used to compose the installation package. The patch file is generated by comparing the old version with the new version using Bsdiff. Here, BSPatch is to merge the old version and patch into the new version file, which is a corresponding process with BSDIff and is also the main method used on Android.

Parameter order is the same as bsdiff
./bspatch apk-v1.apk apk-v2.apk patch
Copy the code

Importing source files

Using Android is also relatively easy. Start by creating a new Native project or Nativelib. Then create a directory bzip2-1.0.6 under the SRC /main/ CPP directory. Place the corresponding bzip2 source file here.

Note that you don’t need to put all the files extracted from bzip2 in, but instead generate the source files associated with libbz2.a. You can view the Makefile in the unzipped directory of bzip2-1.0.6:

OBJS= blocksort.o  \
      huffman.o    \
      crctable.o   \
      randtable.o  \
      compress.o   \
      decompress.o \
      bzlib.o

libbz2.a: $(OBJS)
        rm -f libbz2.a
        $(AR) cq libbz2.a $(OBJS)
        
blocksort.o: blocksort.c
        @cat words0
        $(CC) $(CFLAGS) -c blocksort.c
huffman.o: huffman.c
        $(CC) $(CFLAGS) -c huffman.c
crctable.o: crctable.c
        $(CC) $(CFLAGS) -c crctable.c
randtable.o: randtable.c
        $(CC) $(CFLAGS) -c randtable.c
compress.o: compress.c
        $(CC) $(CFLAGS) -c compress.c
decompress.o: decompress.c
        $(CC) $(CFLAGS) -c decompress.c
bzlib.o: bzlib.c
        $(CC) $(CFLAGS) -c bzlib.c
Copy the code

C, crctable.c, randtable.c, compress. C, decompress.c, and bzlib.c. You also need two header files, bzlib.h and bzlib_private.h. That’s a total of nine files in the newly created zip2-1.0.6 directory above. Then place the bspatch.c extracted from bsdiff into SRC /main/ CPP.

The directory structure should now look like this:

. ├ ─ ─ the SRC │ ├ ─ ─ the main │ ├ ─ ─ CPP │ ├ ─ ─ bzip2-1.0.6 | ├ ─ ─ bspatch. C │ ├ ─ ─ nativelib. CPP │ └ ─ ─ CMakeLists. TXT ├...Copy the code

Nativelib. CPP is automatically generated when a new module is created. You can change the file name to another file, such as bspatch_merge.cpp.

Write CMakeLists. TXT

Then write the cmakelists. TXT rule to add both the bzip2 source files and the bspatch source files:

cmake_minimum_required(VERSION 3.10.2)
project("bspatch")

file(GLOB bzip_sources ${CMAKE_SOURCE_DIR}/bzip2-1.0.6/*.c)

add_library(
    bspatch
    SHARED

    bspatch.c
    bspatch_merge.cpp
    ${bzip_sources}
)


find_library(
    log-lib
    log
)


target_link_libraries(
    bspatch
    ${log-lib}
)
Copy the code

In bspatch.c, the entry method is also the main function, since bspatch.c is ultimately compiled into an executable under Linux. On Android, we end up compiling it as a shared library so, so it’s best to rename main to avoid collisions when adding other libraries later. Change this to patch_main. Also, change the header file referenced in bspatch.c to #include

instead of #include “bzip2-1.0.6/bzlib.h”

Write the code

Then rename the NativeLib class to PatchUtils and define it as a singleton class:

object PatchUtils {

	init {
        // The name here must be the same as defined in add_library in cmakelists.txt
        System.loadLibrary("bspatch")}/** * Note that this method is a time-consuming operation and should not be added to the main thread. [oldFile] the merged file should be a specific file path * [oldFile] the path of the oldFile, should be a specific file path * [patch] the subcontract file, Should be a specific file path * * return true on success, false */ otherwise
    external fun bsPatch(newFile: String, oldFile: String, patch: String): Boolean
}
Copy the code

In this case, the bsPatch method should report red error, and you can directly generate jNI method according to the prompts. When selecting the location of the generated file, remember to select bsPatch. Or, instead of generating it, write it by hand in bspatch.c, making sure that the package name and class name in the method are the same.

#include <jni.h>
#include <string>

extern "C" {
extern int patch_main(int argc, char *argv[]);
}

extern "C"
JNIEXPORT jboolean JNICALL
Java_com_study_bspatch_PatchUtils_bsPatch(JNIEnv *env, jobject thiz, jstring new_file, jstring old_file, jstring patch_file) {
    const char *newFile = env->GetStringUTFChars(new_file, nullptr);
    const char *oldFile = env->GetStringUTFChars(old_file, nullptr);
    const char *patchFile = env->GetStringUTFChars(patch_file, nullptr);

    char *argv[] = {"".const_cast<char *>(oldFile), const_cast<char *>(newFile),
                    const_cast<char *>(patchFile)};
    int res = patch_main(4, argv);

    env->ReleaseStringUTFChars(old_file, oldFile);
    env->ReleaseStringUTFChars(new_file, newFile);
    env->ReleaseStringUTFChars(patch_file, patchFile);

    return res == 0;
}
Copy the code

The patch_main method in bspatch.c is first introduced via the extern keyword and then called. In the executable file, we use the./bspatch old.apk new.apk patch command to generate the new file. In the corresponding method, there are actually 4 parameters, because the first parameter is the function itself, which needs to be noted here.

At this point, you’re done with Android, just call the patchutils.bspatch method. The current installation apk can through the context. ApplicationInfo. SourceDir to obtain.

The detailed code was uploaded to the Github repository.

conclusion

Using Bsdiff to do incremental updates to the installation package is not difficult, or even very simple, since we actually just call the main method in Bspatch to synthesize in Android. Similarly, compiling Bsdiff for Linux is as simple as slightly modifying the Makefile.

Bsdiff can effectively reduce the volume of the downloaded installation package during the update, because we only need to download the corresponding patch subpackage instead of downloading the complete installation package file, which is also our ultimate goal.

However, it is very troublesome to use in practice, because after each update, corresponding patch subcontracting needs to be generated with all previous apK versions, and then the corresponding patch download address is returned according to the version parameters transmitted when obtaining update information.

This is just one channel package. In fact, each app store we upload is a different channel package, and each app store has about a dozen channels. That is to say, each upgrade will generate at least a dozen patch subcontracts, and this is only with one old version OF APK, but in reality, we have a lot of old versions, which means that the number of files subcontracted by patch will be very large…

Of course, you can write script files to manage….