background
I haven’t seen you for a long time. I have been busy with the development of this black technology, so I don’t have time to write blog.
In the current project, Tencent’s AndResGuard was used for an in-depth optimization of the size of resource files. AndResGuard also confuses file names, arSC files, and R files to reduce the overall resource file size.
However, it was not a perfect solution, so we decided to do a secondary development on top of it.
AndResGuard principle
Let me briefly explain how AndResGuard (ARG) works.
First of all, we need to compile our APP project, and then generate APK file after all the compilation process is finished. Then ARG will decompress the APK file and make a copy of it, and then mix and rename arSC and other resource files from the copy, and finally repackage the copy into APK. Then re-sign the APK.
Only after we understand the complete process of the ARG can we do secondary development and secondary optimization. First of all, of course, we set goals, what are we going to do, and then what can we do?
TODO
What are we going to do?
-
Is it possible to put confounded processes into apK compilation processes to take full advantage of compile-time multithreading?
-
Is it possible to adjust the confused rules twice to improve the compression ratio?
-
Is there a way to save compilation speed and improve plug-in efficiency?
ACTION
Before development, we must first sort out the scheme and analyze the competing products to find out if there are any competing products that can help us.
During our investigation, Meituan, Tencent and Toutiao all have corresponding resource file confusion schemes. Tencent’s IS ARG, and ARG is the most used. Meituan seems to have no open source project to follow up. Toutiao’s AabResGuard is mainly responsible for the compression of Toutiao’s App Bundle, but also for common resource obfuscation. My friend said that the compression of app bundle of sea project mainly depends on this.
We made a reasonable change in the execution order of ARG by referring to the way that AabResGuard modified the execution order of tasks.
How do I change the execution order of compilation tasks
In the process of code analysis of Aab, we actually found some very magical and subtle points, which have a great inspiration for our subsequent optimization.
private fun createAabResGuardTask(project: Project, scope: VariantScope) {
val variantName = scope.variantData.name.capitalize()
val bundleTaskName = "bundle$variantName"
if (project.tasks.findByName(bundleTaskName) == null) {
return
}
val aabResGuardTaskName = "aabresguard$variantName"
val aabResGuardTask: AabResGuardTask
aabResGuardTask = if (project.tasks.findByName(aabResGuardTaskName) == null) {
project.tasks.create(aabResGuardTaskName, AabResGuardTask::class.java)
} else {
project.tasks.getByName(aabResGuardTaskName) as AabResGuardTask
}
aabResGuardTask.setVariantScope(scope)
val bundleTask: Task = project.tasks.getByName(bundleTaskName)
val bundlePackageTask: Task = project.tasks.getByName("package${variantName}Bundle")
bundleTask.dependsOn(aabResGuardTask)
aabResGuardTask.dependsOn(bundlePackageTask)
// AGP-4.0.0-alpha07: use FinalizeBundleTask to sign bundle file
// FinalizeBundleTask is executed after PackageBundleTask
val finalizeBundleTaskName = "sign${variantName}Bundle"
if(project.tasks.findByName(finalizeBundleTaskName) ! =null) {
aabResGuardTask.dependsOn(project.tasks.getByName(finalizeBundleTaskName))
}
}
Copy the codeThis part of the code is where Aab’s plugin tampered with the order of dependencies for task execution when constructing an obfuscation task.
A variantName represents a variant of a build, which can be a multi-channel build or a variant of a Debug release.
A normal Android app Bundle is executed in the order of package${variantName}Bundle immediately after Bundle $variantName.
The aAB plugin inserts a custom obfuscated task, aabResGuardTaskName, so that when a package${variantName}Bundle is executed, Bundle $variantName -aabresGuardTaskName -package${variantName} bundle
Gradle tasks are ordered by a directed acyclic graph (DAG), so when tasks are dependent on each other, Gradle executes them according to the order of the DAG. Basically if there is any dependsOn you can simply refer to them as DAG.
Observe the process of compiling a project
Sometimes students will say, “What compilation process do you ask during the interview?” They will not use it in real development. But sometimes there’s nothing wrong with having multiple skills.
A Task compiled by Apk was printed using a piece of code logic that took time to print tasks before.
159ms :libres:generateDebugRFile
186ms :libres:compileDebugJavaWithJavac
181ms :app:processFlavor2Flavor1DebugManifest
121ms :app:mergeFlavor2Flavor1DebugResources
999ms :app:processFlavor2Flavor1DebugResources
1025ms :app:compileFlavor2Flavor1DebugKotlin
1163ms :app:resguardFlavor2Flavor1Debug
1183ms :app:mergeFlavor2Flavor1DebugNativeLibs
296ms :app:compileFlavor2Flavor1DebugJavaWithJavac
451ms :app:transformClassesWithDexBuilderForFlavor2Flavor1Debug
99ms :app:mergeProjectDexFlavor2Flavor1Debug
124ms :app:mergeFlavor2Flavor1DebugJavaResource
295ms :app:packageFlavor2Flavor1Debug
Copy the codeWhen we start compiling an Apk, the task stack from top to bottom is similar to the one above. I added the Plavor variant to the demo, but it does not affect the task. The sneak resguardFlavor2Flavor1Debug this task is the task of our resources to confuse, basic implementation rules and bytes of aab scheme of similar. And then we insert node selection is after processFlavor2Flavor1DebugResources, to carry out our confusion before mergeFlavor2Flavor1DebugJavaResource tasks at the same time.
Why choose this node?
When we compile an APK, we generate many input and output files in the Build /intermediates folder. This is a trick I found while developing transform. Then I search under this folder and observe which task node is completed by compiling our resource file.
We can see the first aapt compiled about a process, in the end I found an interesting directory processed_res, namely the processFlavor2Flavor1DebugResources said above this task. There will be an out file directory under this folder, which will contain a.ap_ file, based on a keen sense of development, I found the truth is only one (Shi N ji tsu wa I tsu mo Hi to tsu), I used Jadx to decompile this file, It found that it stored all the resource files, ARSC files.
I also did a bold experiment, if I put the confused AP_ here, and then overwrite the file with the same name. Will the apK compiled later be a confused APK?
The result of the experiment was the same as I predicted, and the apK compiled at last was a confused APK.
I want to leave some small regrets here. I wanted to touch the source code of the Task for the whole compilation process, but I tried to look at this part of the source code, but it was too difficult and the debug cost was too high, so I did not understand it carefully.
The first mission is complete
After going through the above process, we just need to re-develop the ARG code and optimize it according to the corresponding task task, so that our first task will be completed.
private fun runGradleTask(absPath: String, outputFile: File, minSDKVersion: Int): File? {
val packageName = applicationId
valwhiteListFullName = ArrayList<String>() configuration? .let {val sevenzip = project.extensions.findByName("sevenzip") as ExecutorExtension
configuration.whiteList.forEach { res ->
if (res.startsWith("R")) {
whiteListFullName.add("$packageName.$res")}else {
whiteListFullName.add(res)
}
}
val builder = InputParam.Builder()
.setMappingFile(configuration.mappingFile)
.setWhiteList(whiteListFullName)
.setUse7zip(configuration.use7zip)
.setMetaName(configuration.metaName)
.setFixedResName(configuration.fixedResName)
.setKeepRoot(configuration.keepRoot)
.setMergeDuplicatedRes(configuration.mergeDuplicatedRes)
.setCompressFilePattern(configuration.compressFilePattern)
.setZipAlign(getZipAlignPath())
.setSevenZipPath(sevenzip.path)
.setOutBuilder(useFolder(outputFile))
.setApkPath(absPath)
.setUseSign(configuration.useSign)
.setDigestAlg(configuration.digestalg)
.setMinSDKVersion(minSDKVersion)
if(configuration.finalApkBackupPath ! =null && configuration.finalApkBackupPath.isNotEmpty()) {
builder.setFinalApkBackupPath(configuration.finalApkBackupPath)
} else {
builder.setFinalApkBackupPath(absPath)
}
builder.setSignatureType(InputParam.SignatureType.SchemaV1)
val inputParam = builder.create()
return Main.gradleRun(inputParam)
}
return null
}
Copy the codeGradle’s multi-threading process can take full advantage of the fact that processRes’ task and Transform are running in parallel.
Data contrast
Figure 1 shows the decompression speed and execution order after we changed, and Figure 2 shows the speed of using the native ARG. It can be found that although we only changed the execution of the next task, we also got great optimization in speed. Part of this is because the ARG unpacked and repackaged the entire APK project, and we just manipulated the fake APK project generated by the resource file. And since it’s a concurrent task, it’s actually a little faster.
Parallelism is accomplished through multiple threads
Is this it? Is there a way to speed this up even further?
Can we consider executing the task directly in the thread, so that the next task can continue to execute, as long as the task is executed before the compilation is completed, can we optimize the time of this part of the resource confusion? Just do it, directly to the code.
open class ResProguardTask : DefaultTask() {
private var executor: ExecutorService = Executors.newSingleThreadExecutor()
private var future: Future<out Any>? = null
@TaskAction
fun execute(a) {
future = executor.submit {
// Resource files are mixed up}}fun await(a){ future? .get()}}Copy the codeExecutorService (ExecutorService) : ExecutorService (ExecutorService) : ExecutorService (ExecutorService) : ExecutorService (ExecutorService) : ExecutorService (ExecutorService) : ExecutorService (ExecutorService)
In accordance with the rigor of program monkey, in fact, if we assume that our future takes a minute and a half, and the total compilation time is 1 minute, there will be problems when we merge and package, which will lead to the failure of this resource confusion. Is there a way to wait for our Future to complete execution before the last Task executes? Have you noticed the await operation I wrote below? Because of the nature of the Future, get will have a value only after all methods have been executed, otherwise it is a while(true) loop. So how do we wait before the final packaged Task?
val bundlePackageTask: Task = project.tasks.getByName("package${variantName}")
bundlePackageTask.doFirst {
val resProguardTask = project.tasks.getByName(resGuardTaskName)
as ResProguardTask
resProguardTask.await()
}
Copy the codeIn this case, we can perform any operation before and after the Task by taking full advantage of the doFirst and doLast methods provided by the Task. Here we make a wait, waiting for all future executions that our resource file confuses to complete before allowing the packageTask to execute.
Someone poisoned the code
In the plugins actual line phase, we ran into a very strange problem where resource file obfuscation failed. In the end, it was discovered during the actual debugging that because shrink was enabled for the project, during the R8 phase the project regenerated an AP_ file that was used for the final APK synthesis package.
val manager = scope.transformManager
val field = manager.javaClass.getDeclaredField("transforms").apply {
isAccessible = true
}
val list = field.get(manager) as List<Transform>
list.forEach {
if (it is ShrinkResourcesTransform) {
}
}
}
Copy the codeI finally reflected the list of transforms held by the scope, then pulled the Transform ShrinkResourcesTransform out, finally got the Task transformed by the transform, and then preceded the Task with the await operation. This ensures that resource file obfuscations are completed before ShrinkResourcesTransform is executed.
teasing
Groovy really sucks, because there are no compile-time alarms, so you don’t know if your code is writing correctly or incorrectly. And before writing a Gradle plug-in, I used to upload a local AAR, so it is particularly disgusting, resulting in my recent writing of a new plug-in using Kotlin, really fragrant.
The last
(゜ ゜ つ ロ cheers ~ – bilibili.
Our group recently open source routing project BRouter, welcome everyone to praise ah.