background
The content of this article is a small broken station ah, do not reprint without authorization
Recently, many companies are facing the same problem as us, and cooperate with the CaC to carry out the privacy permission rectification. This involves disabling calls to sensitive apis, such as iMEI, AndroiDID, IP, MacAddress, etc., until the user agrees to privacy permissions.
A previous article described how to search for calls with sensitive permissions in Python by decompilating the APK artifacts, and then notifying the caller to correct them.
How to efficiently check where sensitive permissions are used in APK and where system methods are called
But there is a problem with the above boss approach, because the project will continue to iterate, and it needs to be reviewed every once in a while before reminding the business side of changes, which is too passive.
Because the network library and the basic components of buried dots depend on unique identifiers, which are not allowed to be invoked before the privacy permission is granted, the initialization task is confused. Meanwhile, these basic repositories should also be provided to other apps of B station. Part of it is for privacy governance, and part of it is to sort out our initialization tasks.
In fact, the solution is relatively simple. We will first abstract a privacy middleware. When no privacy permission is granted, all API calls will return null values.
Then you need to replace the business API calls with privacy middleware on the line.
The Demo project
pipeline
The Jenkins official document describes Pipeline as follows: Jenkins Pipeline (or simply “Pipeline”) is a suite of plugins which supports implementing and integrating continuous delivery pipelinesinto Jenkins. Pipeline is a set of plugins officially provided by Jenkins that can be used to implement and integrate continuous delivery in Jenkins.
Pipeline is a process that defines the steps to complete a CI/CD process. Instead of manually completing the CI/CD process, the process is defined by the user.
The pipeline on GitLab corresponds to.gitlab-ci.yaml.
This is all the steps that are performed after the current bilibili branch code is pushed to the remote end, before all the steps are passed to allow the code to be entered.
GithubActions
Github is currently offering a very simple CI/CD access solution, interested players can try it.
The results can be found at github.com/Leifzhang/A…
Static checking
If you’re interested in learning about basic use of Lint, see my previous article Android Custom Lint Development before we talk about Android Lint
Because the repository at Site B is basically a mono-repo of source code, all the source code is in one place, so it is convenient for static code review.
At the same time, because all the code and entries have to complete the static scanned pipline first, we can ensure that all subsequent code and entries are canonical, so we can effectively and consistently avoid this problem.
The number of API changes we’re dealing with is large, and the text for each tip is different, so it would be very difficult to develop one by one, so we need to provide a simpler and more extensible way to make these simple Lints configurable.
This part is not our original technology, but based on the previous technology stack of Meituan and Mihuyu, we have carried out iteration and transformation. He also shared it on His Github account.
Github reference link AndroidLint
Json format
First, let’s take a look at this simple JSON definition, because we’re going to do dynamic JSON matching based on this JSON.
Since constructors and method calls are actually written in two different ways, we’ve defined two differences here
{ "methods": [ { "name_regex": "android.net.wifi.WifiManager.getSSID", "message": This is privacy "govemment govemment API. Please use the PrivacyUtil getWifiName replace oh", "excludes" : [" com. XXXXXX. Privacy. PrivacyImp "]}, {" name_regex ": "Android.net.wifi.WifiManager.getBSSID", "message" : "govemment govemment this is privacy API. Please use the PrivacyUtil getWifiName replace oh", "excludes" : [ "com.xxxxx.privacy.PrivacyImp" ] }, { "name_regex": "Settings.Secure.getString", "message": "Gie this is the private API, please use PrivacyUtil. GetAndroidId instead ", "excludes": [ "com.bilibili.privacy.PrivacyImp", "com.bilibili.adcommon.util.LocationUtil" ] }, { "name_regex": "Android. Telephony. TelephonyManager. GetImei", "message" : "govemment govemment this API is privacy. Please use the PrivacyUtil getDeviceId replace oh", "excludes" : [ "com.xxxx.privacy.PrivacyImp" ] }, { "name_regex": "android.telephony.TelephonyManager.getDeviceId", "message": This is privacy "govemment govemment API. Please use the PrivacyUtil getDeviceId replace oh", "excludes" : [" com. XXXX. Privacy. PrivacyImp "]}, {" name_regex ": "android.telephony.TelephonyManager.getDeviceId", "message": This is privacy "govemment govemment API. Please use the PrivacyUtil getDeviceId replace oh", "excludes" : [" com. XXXX. Privacy. PrivacyImp "]}, {" name_regex ": "Java.net.getHostAddress ", "message": "gie this is a private API please use privacyutil. getIpAddress instead ", "excludes": [ "com.xxxxx.privacy.PrivacyImp" ] }, { "name_regex": "android.content.pm.PackageManager.getInstalledApplications", "message": "Govemment govemment this API is privacy. Please use the PrivacyUtil getPackageList replace oh", "excludes" : [ "com.Xxxxxx.privacy.PrivacyImp" ] }, { "name_regex": "android.content.pm.PackageManager.getInstalledPackages", "message": This is privacy "govemment govemment API. Please use the PrivacyUtil getAppList replace oh", "excludes" : [" com. XXXXX. Privacy. PrivacyImp "]}], "constructions" : [ { "name_regex": "", "message": "" } ] }Copy the codeThe above is our current list of privacy-related JSON, excluding the code related to our middleware. We hope to rectify all privacy-related APIS to the middleware at one time.
Because the appeal is simple this time, we only define the method and the constructor arrays. Name_regex indicates rule matching, message indicates prompt copywriting, and excludes indicates whitelist list. Because our appeal is to uniformly call the middleware that we define, all of which are on our whitelist.
Dynamically configurable Lint
The hard part here is getting the Lint code to read the JSON file of our configuration.
class DynamicLint : Detector(), Detector.UastScanner {
lateinit var globalConfig: DynamicConfigEntity
override fun beforeCheckRootProject(context: Context) {
super.beforeCheckRootProject(context)
globalConfig = GsonUtils.inflate(context.project.dir)
}
}
Copy the codeDetector provides a beforeCheckRootProject method. So, this method is going to pass in the directory information and so on, and we’re going to get our configurable JSON file information from this Context.
There’s a small detail here, because our project is in the compose Building mode, and the Context is normally only passed in the Module path, so we’re going to do a simple recursive lookup.
private fun findCodeQuality(projectDir: File): File? {
if(projectDir.parent ! =null) {
val parent = projectDir.parentFile
valfile = parent.listFiles()? .firstOrNull { it.name ==".codequality" && it.isDirectory
}
returnfile ? : findCodeQuality(parent) }return null
}
Copy the codeA simple recursive call addressing. I will be one of the few spicy chicken algorithm problem, ha ha ha.
NameRegex * inClassName = nameRegex * inClassName = nameRegex * inClassName = class * exclude = class */
private fun match(
nameRegex: String? , qualifiedName:String? , inClassName:String? = null,
exclude: List<String> = emptyList(),
excludeRegex: String? = null) :Boolean{ qualifiedName ? :return false
/ / out
if(inClassName ! =null && inClassName.isNotEmpty()) {
if (exclude.contains(inClassName)) return false
if(excludeRegex ! =null &&
excludeRegex.isNotEmpty() &&
Pattern.compile(excludeRegex).matcher(inClassName).find()
) {
return false}}if(nameRegex ! =null && nameRegex.isNotEmpty() &&
Pattern.compile(nameRegex).matcher(qualifiedName).find()
) {// In the match nameRegex
return true
}
return false}}Copy the codeThe code check matches through the above code, this part of the logic is relatively simple, you are interested to see the line.
How to validate
While we are constrainting code in a project via Lint, the compiled.class is not recognized by this UastScanner.
It is possible to do class Lint scans using the ClassScanner, but the logic is relatively complex, and I have actually written it for ASM.
And if we put this requirement up for testing, there is no way for the testing students to test this requirement. So we need another way to provide some hook capability at runtime, either to generate a file record or to crash directly when these private apis are called.
Based on Epic dynamic hook
Epic’s hook mechanism is based on THE ELFT file format of ART, so it can hook all method calls in the code. Although it is a bit passive, it can avoid the privacy permission call problems caused by extreme cases such as reflection, and the call situation in the third-party SDK.
First, we can use the dynamic JSON file defined in the previous project, and then copy it directly into the Assets folder of debug through the soft link.
Soft link is a common Linux command that creates a link between a file and another location. The specific usage is as follows: ln -s Source file Destination file.
fun hookManager(context: Context) {
val steam = context.resources.assets.open("dynamic.json")
val configEntity = GsonUtils.inflate(steam)
configEntity.methods.forEach {
start(it)
}
}
fun start(entity: DynamicEntity) {
if (entity.name_regex.isNotEmpty()) {
val methodName = entity.name_regex.substring(entity.name_regex.lastIndexOf(".") + 1)
val className = entity.name_regex.substring(0, entity.name_regex.lastIndexOf("."))
val lintClass = Class.forName(className)
DexposedBridge.hookAllMethods(lintClass, methodName, object : XC_MethodHook() {
override fun beforeHookedMethod(param: MethodHookParam?) {
super.beforeHookedMethod(param)
Log.i("EpicHook", "EpicHook")
}
})
}
}
Copy the codeAfter in the debug package case, through the deserialization json, also good to generate the corresponding hooks file configuration, called after DexposedBridge. HookAllMethods method.
Tips: Because the dynamic hook framework is extremely unstable, please do not release this feature online, and it is better to include version control logic, because it will crash in Android 10 version.
Remember that the debug tool must not be brought online, because generally the features designed for debug are risky operations, so this part of the variation must be added.
Privacy calls in third party libraries
Although we have the ability of dynamic Hook, because the dynamic Hook must wait until the method is called, the exception will be executed. For some pages with deep calling logic, the situation may not be covered.
A better solution would be to use ASM to replace third-party privacy code and subcontract it to our middleware.
In this way, you can achieve multiple insurance, and you can deal with a great degree of institutional scrutiny.
Use Transform + Asm to locate sensitive permissions
This part is also relatively simple, I have written a small demo to verify this part of the fix, and the following indicates an attempt to fix devicEID.
Byte Override Fun modifyClass(srcClass: ByteArray?) : ByteArray {val classNode = classNode (opcodes.asm5) val classReader = classReader (srcClass) //1 Accept (classNode, 0) //2 Iterator<MethodNode> = classNode.methods.iterator() while (iterator.hasNext()) { val method = iterator.next() method.instructions? .iterator()? .foreach {if (it is MethodInsnNode && it.isprivacy ()) {// Replace operator it.opcode = opcodes.invokestatic it.owner = "com/wallstreetcn/sample/utils/PrivacyUtils" it.name = "getImei" method.instructions.remove(it.previous) } } } val ClassWriter = classWriter (ClassWriter.COMPUTE_MAXS) //3 Convert the classNode into a byte array classNode.accept(classWriter) return Classwriter.tobytearray ()} private fun methodinsnNode.isprivacy (): Boolean { if (owner == "android/telephony/TelephonyManager" && name == "getDeviceId") { return true } return false }Copy the codeThis part can locate the problem of more detailed API calls of the third-party library, and help us to promote the adjustment of the third-party library.
Also with the old method, through the Tree API Asm, after the judgment method of the current stack frame is “android/telephony/TelephonyManager getDeviceId method”, if have to be modified, replaced with our definition of a static method.
Here is a little tips, because must be obtained before the android/telephony/TelephonyManager and completed the pressure stack method, so we have to put on a method call.
conclusion
Because we have this static checking capability configurable this time, all we need to do is change the scan rules to meet this requirement. Greatly expanded our ability to respond passively to the review, but also better for our current large warehouse mode of affirmation.
This time we share the main purpose is to make a contribution to China’s harmonious mobile ecology, you and I have the responsibility to purify the network environment, the importance of user privacy for today’s society is self-evident. Because all incoming code is statically reviewed and manually reviewed, you can be assured that all subsequent incoming and incoming code has completed this part of the review capability. Hope the article can be helpful to you.