There are three reasons for writing this article. First of all, there are classmates in the ByteX communication group who give feedback that the source code is not easy to read and looks more laborious, so I hope to help you learn ByteX source code through my own understanding and sorting. Secondly, some students in reading my blog AOP sharp tool ASM foundation after feedback, do not know how to find ASM in the work of the use of scenarios, so I hope to use ByteX plug-ins can give you some application scenarios inspired. Finally, as a summary of personal learning ASM.
1. ByteX is introduced
ByteX is a bytecode plug-in platform based on the Gradle Transform API and ASM. The platform-based design can avoid the linear increase in compilation and construction time and code coupling bloat caused by the independent development of a plug-in for each Feature business. At present, ByteX integrates several bytecode plug-ins, and each plug-in is completely independent. It can not only exist independently from ByteX host, but also automatically integrate into host and other plug-ins into a single Transform. The code between plug-ins and between hosts is completely decoupled, which makes ByteX very extensible in code and makes the development of new plug-ins easier and more efficient.
ByteX warehouse address: https://github.com/bytedance/ByteX
2. Project structure
By abstracting Plugin and Transform, the common code is stripped into the common library and made available to all plug-ins for reuse, so that each plug-in only needs to focus on the business logic processing of bytecode staking. At the same time, plug-ins are automatically and seamlessly integrated into a Transform to improve the efficiency of construction. The IO to the class file is time-consuming during the Transform process, and consolidating all the plugins into a single Transform can avoid the linear increase in the additional time cost of packaging. Let’s change the time from 1+1=2 to 1+1<2 or approximately equal to 1. Platform the code between each plug-in from each other to avoid code coupling, easy to expand.
2.1 Plugins application layer
In the ByteX project, a number of plug-ins are provided.
- access-inline-plugin:
accessMethods inline optimization, reduceapkNumber of methods and package size - shrink-r-plugin:
RThin files and check for useless resources - closeable-check-plugin: file stream
closecheck - Const-inline-plugin: constant inline
- Field-assignment-opt-plugin: Optimizes redundant assignment instructions
- getter-setter-inline-plugin ：
getter和setterMethods the inline - method-call-opt-plugin: Cleanly removes some method calls, such as
Log.d - Coverage-plugin: Online code coverage
- Refer-check-plugin: Checks for calls to nonexistent methods and references to nonexistent fields
- Serialization-check-plugin: serialization check
- SourceFileKiller: delete
SourceFileAnd line number properties - ButterKnifeChecker: to test across
moduleuseButterKnifeProblems that may result - RFileKnifeRepair:
R.javaToo large compiled messagecode too largeThe problem of
Of course, we can also develop custom plug-ins based on the common library. See the Development WIKI: Developer API here.
2.2 the Common layer
The Common module is the core of the framework and encapsulates the common code portion for use by all plug-ins. The structure can be roughly divided into the following:
- General plug-in
BasePluginThe classes involved are:AbsMainProcessPlugin,AbsPlugin,MainProcessHandler和IPlugin - An abstraction of plug-in configuration items, involving the following classes:
BaseExtension - general
TransformThe classes involved are:CommonTransform,SimpleTransform,ProxyTransform - Gm’s
ClassVisitor, the classes involved are:BaseClassVisitor,ClassVisitorChain - Compile life cycle listening:
ByteXBuilderListener,TransformFlowListener,PluginStatus - Class diagram generation: involves
flow和graphTwo packages - Class parser and validation:
ClassFileAnalyzer和ClassFileTransformer,AsmVerifier,AsmVerifyClassVisitor
Many students feedback that the common library is more complex, through the above module split, can better assist in reading the source code of the Common layer.
2.3 Underlying Dependencies
In the common layer, it relies on ASM repository, TransformEngine package, and GradleToolKit.
3. Core logic of Common module
The Common module is the core of the framework and contains common code, which is the focus of our study.
3.1 Core Process
Regardless of the variation, the general core flow of a Plugin developed based on the Transform mechanism starts with the plugin.apply () method. See the figure below for the ByteX core process.
3.2 AbsPlugin. Apply method
Absplugin.apply () is a key step in customizing plug-ins, which typically includes the definition of plug-in configuration items, callback listening, and registration of Transform handling.
public final void apply(@NotNull Project project) {
// Call onByteXPluginApply to listen
GlobalByteXBuildListener.INSTANCE.onByteXPluginApply(project, this);
this.project = project;
this.android = project.getExtensions().getByType(AppExtension.class);
// Read the Project configuration item
ProjectOptions.INSTANCE.init(project);
// Whitelist configuration read
GlobalWhiteListManager.INSTANCE.init(project);
Class<E> extensionClass = getExtensionClass();
if(extensionClass ! =null) {
Instantiator instantiator = ((DefaultGradle) project.getGradle()).getServices().get(Instantiator.class);
extension = createExtension(instantiator, extensionClass);
// Add the ByteX configuration
project.getExtensions().add(extension.getName(), extension);
}
// Call back the onApply method
onApply(project);
// Read Proxy Proxy name
String hookTransformName = hookTransformName();
if(hookTransformName ! =null) {
// code ProxyTransform processing
TransformHook.inject(project, android, this);
} else {
// Whether it is an independent plug-in
if(! alone()) {// Not a standalone plug-in
try {
ByteXExtension byteX = project.getExtensions().getByType(ByteXExtension.class);
byteX.registerPlugin(this);
isRunningAlone = false;
} catch (UnknownDomainObjectException e) {
android.registerTransform(getTransform());
isRunningAlone = true; }}else {
// independent plug-in
android.registerTransform(getTransform());
isRunningAlone = true; }}/ / callback onByteXPluginApplied
GlobalByteXBuildListener.INSTANCE.onByteXPluginApplied(this);
}
Copy the codeImplement custom plug-in by inheriting CommonPlugin. After plug-in registration, execute to absplugin.apply (), and complete plug-in callback listening, configuration definition, Transform registration and other operations in this method.
3.3 Implementation of Transform mechanism
The handling of the Transform is a key part of implementing a Plugin. Encapsulate the processing logic of Transform into the CommonTransform class in ByteX.
@Override
public final synchronized void transform(TransformInvocation transformInvocation) throws TransformException, InterruptedException, IOException {
try {
/ / the plugin compiled start callback onByteXPluginTransformStart Transform
GlobalByteXBuildListener.INSTANCE.onByteXPluginTransformStart(this, transformInvocation);
// Transform core logic encapsulation
transformInternal(transformInvocation);
/ / plug-in end callback onByteXPluginTransformFinished compile the Transform
GlobalByteXBuildListener.INSTANCE.onByteXPluginTransformFinished(this.null);
} catch (Exception e) {
GlobalByteXBuildListener.INSTANCE.onByteXPluginTransformFinished(this, e);
throwe; }}private void transformInternal(TransformInvocation transformInvocation) throws TransformException, InterruptedException, IOException {
super.transform(transformInvocation);
// 1. If it is not incremental compilation and OutputProvider is not empty, clear the output folder path
if(! transformInvocation.isIncremental() && transformInvocation.getOutputProvider() ! =null) {
transformInvocation.getOutputProvider().deleteAll();
}
// 2. Read the TransformContext instance
TransformContext transformContext = getTransformContext(transformInvocation);
// 3. Initialize Log and HtmlReporter Log output
init(transformContext);
// 4. Filter the plugins whose enable status is false and sort them
List<IPlugin> plugins = getPlugins().stream().filter(p -> p.enable(transformContext)).collect(Collectors.toList());
// 5. For plug-ins that do not enable incremental compilation, disable incremental variables
if(plugins.stream().anyMatch(iPlugin -> ! iPlugin.transformConfiguration().isIncremental())) { transformContext.requestNotIncremental(); } Timer timer =new Timer();
Create the TransformFlowerManager object
final ITransformPipeline manager = new TransformFlowerManager(transformContext);
try {
if(! plugins.isEmpty()) {// 7. Iterate over the startExecute that executes Plugin
plugins.forEach(iPlugin -> iPlugin.startExecute(transformContext));
// 8. Loop over each Plugin and register the Plugin's Transform callback, binding it to the TransformFlowerManager
plugins.forEach(plugin -> manager.bind(manager1 -> plugin.registerTransformFlow(manager1.getCommonFlow(), transformContext)));
Trigger the onPreTransform callback
manager.onPreTransform();
Trigger the runTransform callback
manager.runTransform();
Trigger the onPostTransform callback
manager.onPostTransform();
} else {// If the plug-in collection is empty, the skip callback is triggered directly
manager.skipTransform();
}
Trigger afterTransform callback
afterTransform(transformInvocation);
} catch (Throwable throwable) {
LevelLog.sDefaultLogger.e(throwable.getClass().getName(), throwable);
throw throwable;
} finally {
// 13. Iterate through afterExecute of Plugin
for (IPlugin plugin : plugins) {
try {
plugin.afterExecute();
} catch (Throwable throwable) {
LevelLog.sDefaultLogger.e("do afterExecute", throwable); }}// 14. Release TransformContext
transformContext.release();
this.configurations = null;
timer.record("Total cost time = [%s ms]");
// 15. Log Output
if (BooleanProperty.ENABLE_HTML_LOG.value()) {
HtmlReporter.getInstance().createHtmlReporter(getName());
HtmlReporter.getInstance().reset();
}
LevelLog.sDefaultLogger = newLevelLog(); }}Copy the codeIn ByteX, the core implementation of transform() is separated into the transformInternal() method for implementation, which can be roughly divided into the above 15 steps. For details, please refer to the comments in the code. Here we focus on steps 6-10, registering the Plugin with the Transform callback and binding it to the TransformFlowerManager.
The key classes involved in this process are:
CommonTransform: user-defined plug-inTransformThe implementation parent class ofTransformFlowerManager: Used for managementTransformProcess management, includingTransformFlowRelationships andTransformprocessTransformEngine:TransformThe process processing utility class, encapsulatedTransformFor exampletraverseAndroidJarMainTransformFlow:TransformTreatment product carrierAbsMainProcessPlugin: Abstract class for custom plug-ins
Step 6: Create the ITransformPipeline object
In this step, create an instance TransformFlowerManager object, Manager, The Class TransformFlowerManager consists of MainTransformFlow, TransformEngine and TransformFlow.
private final MainTransformFlow commonFlow;
private final TransformEngine engine;
private TransformFlow first;
public TransformFlowerManager(TransformContext context) {
this.engine = new TransformEngine(context);
this.commonFlow = new MainTransformFlow(engine);
}
Copy the codeStep 7: Call plugin.startexecute
In this step, go back to the startExecute method of each Plugin and see that the implementation in AbsPlugin triggers onByteXPluginStart.
// the source corresponds to the AbsPlugin class
public void startExecute(TransformContext transformContext) {
GlobalByteXBuildListener.INSTANCE.onByteXPluginStart(this);
}
Copy the codeStep 8: Plugin. RegisterTransformFlow
Let’s adjust the Lambda notation for step 8 to make it easier to understand.
plugins.forEach(plugin -> {
manager.bind(new FlowBinder() {
@Override
public TransformFlow bind(TransformFlowerManager manager) {
returnplugin.registerTransformFlow(manager.getCommonFlow(), transformContext); }}); });Copy the codeIn this step, each Plugin is iterated over and the corresponding Transformflows of each Plugin are concatenated by calling the Bind method of the TransformFlowerManager. Each Plugin generates a corresponding TransformFlow object by calling the registerTransformFlow method.
// The source corresponds to the AbsMainProcessPlugin class
public final TransformFlow registerTransformFlow(@Nonnull MainTransformFlow mainFlow, @Nonnull TransformContext transformContext) {
if (transformFlow == null) {
transformFlow = provideTransformFlow(mainFlow, transformContext);// mainFlow is returned
if (transformFlow == null) {
throw new RuntimeException("TransformFlow can not be null.");
}
if(! transformFlow.isLifecycleAware()) { transformFlow =newLifecycleAwareTransformFlow(transformFlow); }}return transformFlow;
}
Copy the codeThrough the source code as you can see the returned TransformFlow object is MainTransformFlow or LifecycleAwareTransformFlow. Here provideTransformFlow within a method implementation is called MainTransformFlow. AppendHandler () method, pay attention to join here is that the current Plugin object, Because the AbsMainProcessPlugin implements the MainProcessHandler interface.
Step 9: onPreTransform()
TransformFlowerManager. OnPreTransform () is the implementation of the traversal of each TransformFlow callback method to prepare.
// The source corresponds to the TransformFlowerManager class
public void onPreTransform(a) throws IOException, InterruptedException {
for(TransformFlow flow : first) { flow.prepare(); }}Copy the codeCan know from the step 8, TransformFlow object is MainTransformFlow or LifecycleAwareTransformFlow, here we MainTransformFlow source code, for example.
// The source corresponds to mainTransformflow.java
@Override
public void prepare(a) throws IOException, InterruptedException {
try {
/ / callback startPrepare
listenerManager.startPrepare(this);
prepareInternal();
/ / callback finishPrepare
listenerManager.finishPrepare(this.null);
} catch (Exception e) {
listenerManager.finishPrepare(this, e);
throwe; }}private void prepareInternal(a) throws IOException, InterruptedException {
super.prepare();
markRunningState(TransformContext.State.INITIALIZING);
timer.startRecord("INIT");
// Execute the init task for each Handler. Note that the Handler collection here is the Plugin object itself that was added when registerTransformFlow was created
Schedulers.COMPUTATION().submitAndAwait(handlers, handler -> handler.init(transformEngine));
timer.stopRecord("INIT"."Process init cost time = [%s ms]");
markRunningState(TransformContext.State.INITIALIZED);
if(! isOnePassEnough()) {// Whether to traverse only once
markRunningState(TransformContext.State.INCREMENTALTRAVERSING);
if (context.isIncremental()) {// Whether to incrementally compile
try {
GlobalMainProcessHandlerListener.INSTANCE.startTraverseIncremental(handlers);
timer.startRecord("TRAVERSE_INCREMENTAL");
// Only increments are traversed
traverseArtifactOnly(getProcessors(Process.TRAVERSE_INCREMENTAL, new ClassFileAnalyzer(context, Process.TRAVERSE_INCREMENTAL, null.new ArrayList<>(handlers))));
timer.stopRecord("TRAVERSE_INCREMENTAL"."Process project all .class files cost time = [%s ms]");
GlobalMainProcessHandlerListener.INSTANCE.finishTraverseIncremental(handlers, null);
} catch (Exception e) {
GlobalMainProcessHandlerListener.INSTANCE.finishTraverseIncremental(handlers, e);
throw e;
}
}
markRunningState(TransformContext.State.BEFORETRAVERSE);
// Perform the beforeTraverse task for each HandlerSchedulers.COMPUTATION().submitAndAwait(handlers, plugin -> plugin.beforeTraverse(transformEngine)); }}Copy the codeThe onPreTransform ultimately executes the prepareInternal() method of the MainTransformFlow class. During each TransformFlow traversal, the business processing is done in the traverseArtifactOnly() method.
traverseArtifactOnly(getProcessors(Process.TRAVERSE_INCREMENTAL, new ClassFileAnalyzer(context, Process.TRAVERSE_INCREMENTAL, null.new ArrayList<>(handlers))));
Copy the codeThere are two important parameters involved:
ClassFileAnalyzerClass: used to complete the class file parsing, a key step, parse the file and form a class diagram;getProcessors()Method: According toProcessState build resolution process.
private FileProcessor[] getProcessors(Process process, FileHandler fileHandler) {
// Add custom FileProcessor during traverse and Transform for more flexibility
List<FileProcessor> processors = handlers.stream()
.flatMap((Function<MainProcessHandler, Stream<FileProcessor>>) handler -> handler.process(process).stream())
.collect(Collectors.toList());
switch (process) {
case TRAVERSE_INCREMENTAL:
processors.add(0.newFilterFileProcessor(fileData -> fileData.getStatus() ! = Status.NOTCHANGED)); processors.add(new IncrementalFileProcessor(new ArrayList<>(handlers), ClassFileProcessor.newInstance(fileHandler)));
break;
case TRAVERSE:
case TRAVERSE_ANDROID:
case TRANSFORM:
processors.add(ClassFileProcessor.newInstance(fileHandler));
processors.add(0.newFilterFileProcessor(fileData -> fileData.getStatus() ! = Status.NOTCHANGED && fileData.getStatus() ! = Status.REMOVED));break;
default:
throw new RuntimeException("Unknow Process:" + process);
}
return processors.toArray(new FileProcessor[0]);
}
Copy the codeFirst add a custom FileProcessor, then add the corresponding processor based on the Process state.
TRAVERSE_INCREMENTAL: Indicates the incremental traversal statusFilterFileProcessor: Filters out unwanted items according to specified conditionsFileData, the filtering condition is that the file status is notStatus.NOTCHANGED;IncrementalFileProcessor: Incremental file processor,It contains oneClassFileProcessor(FileHandler)Parameter for file parsing.
TRAVERSE,TRAVERSE_ANDROID,TRANSFORMStatus:ClassFileProcessor: class file processor,It contains oneClassFileProcessor(FileHandler)Parameter for file parsing.FilterFileProcessor: Filters out unwanted items according to specified conditionsFileData, the filtering condition is that the file status is notStatus.NOTCHANGED和Status.REMOVED.
The implementation of ClassFileProcessor, IncrementalFileProcessor, and FilterFileProcessor will not be expanded for space reasons, but will focus on the processing of FileHandler.
// The source corresponds to fileHandler.java
public void handle(FileData fileData) {
try {
List<MainProcessHandler> pluginList = handlers;
if (fileData.getStatus() == Status.REMOVED) {// Remove state file processing
if(process ! = Process.TRAVERSE_INCREMENTAL) {throw new IllegalStateException("REMOVED State is only valid in TRAVERSE_INCREMENTAL process");
}
for (MainProcessHandler handler : pluginList) {
handler.traverseIncremental(fileData, (ClassVisitorChain) null);
}
return;
}
byte[] raw = fileData.getBytes();
String relativePath = fileData.getRelativePath();
ClassReader cr = new ClassReader(raw); // Build a ClassReader object and parse the file
int flag = getFlag(handlers);
ClassVisitorChain chain = getClassVisitorChain(relativePath); // Create a ClassVisitorChain object, which is essentially a ClassWriter
if (this.mGraphBuilder ! =null) {/ / if the builder class diagrams is empty, create GenerateGraphClassVisitor class reading class files, and build a class diagram
//do generate class diagram
chain.connect(new GenerateGraphClassVisitor(process == TRAVERSE_ANDROID, mGraphBuilder));
}
pluginList.forEach(plugin -> {// According to different state, callback Plugin corresponding processing procedure
switch (process) {
case TRAVERSE_INCREMENTAL:
plugin.traverseIncremental(fileData, chain);
break;
case TRAVERSE:
plugin.traverse(relativePath, chain);
break;
case TRAVERSE_ANDROID:
plugin.traverseAndroidJar(relativePath, chain); / / traverse Android. The jar
break;
default:
throw new RuntimeException("Unsupported Process"); }}); ClassNode cn =new SafeClassNode();
chain.append(cn);
chain.accept(cr, flag);
pluginList.forEach(plugin -> { // do a command check
switch (process) {
case TRAVERSE_INCREMENTAL:
plugin.traverseIncremental(fileData, cn);
break;
case TRAVERSE:
plugin.traverse(relativePath, cn);
break;
case TRAVERSE_ANDROID:
plugin.traverseAndroidJar(relativePath, cn);
break;
default:
throw new RuntimeException("Unsupported Process"); }}); }catch (ByteXException e) {
throw new RuntimeException(String.format("Failed to resolve class %s[%s]", fileData.getRelativePath(), Utils.getAllFileCachePath(context, fileData.getRelativePath())), e);
} catch (Exception e) {
e.printStackTrace();
LevelLog.sDefaultLogger.e(String.format("Failed to read class %s", fileData.getRelativePath()), e);
if(! GlobalWhiteListManager.INSTANCE.shouldIgnore(fileData.getRelativePath())) {throw new RuntimeException(String.format("Failed to resolve class %s[%s]", fileData.getRelativePath(), Utils.getAllFileCachePath(context, fileData.getRelativePath())), e); }}}Copy the codeThe core of handle method is to read class files through ASM and form GraphBuilder.
Step 10: runTransform()
In onPreTransform() of Step 9, a traversal was completed. The runTransform() is traversed again.
/ / the source code corresponding TransformFlowerManager. Java
@Override
public void runTransform(a) throws IOException, InterruptedException {
for (TransformFlow flow : first) {
flow.run(); Run the transformflow. run method
Graph graph = flow.getClassGraph(); // Read the Graph object
if (graph instanceof EditableGraph) {
// Clear the class diagram. We won't use it anymore((EditableGraph) graph).clear(); }}}Copy the codeCan know from the step 8, TransformFlow object is MainTransformFlow or LifecycleAwareTransformFlow, here we MainTransformFlow source code, for example.
// The source corresponds to mainTransformflow.java
@Override
public void run(a) throws IOException, InterruptedException {
try {
listenerManager.startRunning(this, context.isIncremental());
runTransform(); // Call the runTransform method
listenerManager.finishRunning(this.null);
} catch (Exception e) {
listenerManager.finishRunning(this, e);
throw e;
} finally{ markRunningState(TransformContext.State.STATELESS); }}private void runTransform(a) throws IOException, InterruptedException {
markRunningState(TransformContext.State.RUNNING);
if (handlers.isEmpty()) return;
timer.startRecord("PRE_PROCESS");
Schedulers.COMPUTATION().submitAndAwait(handlers, plugin -> plugin.startRunning(transformEngine));
if(! isOnePassEnough()) { timer.startRecord("LOADCACHE");
GraphBuilder graphBuilder = new CachedGraphBuilder(getGraphCache(), context.isIncremental(), context.shouldSaveCache());
if(context.isIncremental() && ! graphBuilder.isCacheValid()) {throw new IllegalStateException("Transform is running as incrementally, but failed to load cache for the transform!");
}
timer.stopRecord("LOADCACHE"."Process loading cache cost time = [%s ms]");
markRunningState(TransformContext.State.TRANSFORMING);
try {
GlobalMainProcessHandlerListener.INSTANCE.startTraverse(handlers);
timer.startRecord("PROJECT_CLASS");
// The class file is traversed
traverseArtifactOnly(getProcessors(Process.TRAVERSE, new ClassFileAnalyzer(context, Process.TRAVERSE, graphBuilder, new ArrayList<>(handlers))));
timer.stopRecord("PROJECT_CLASS"."Process project all .class files cost time = [%s ms]");
GlobalMainProcessHandlerListener.INSTANCE.finishTraverse(handlers, null);
} catch (Exception e) {
GlobalMainProcessHandlerListener.INSTANCE.finishTraverse(handlers, e);
throw e;
}
try {
GlobalMainProcessHandlerListener.INSTANCE.startTraverseAndroidJar(handlers);
timer.startRecord("ANDROID");
markRunningState(TransformContext.State.TRAVERSINGANDROID);
/ / traverse Android. The jar
traverseAndroidJarOnly(getProcessors(Process.TRAVERSE_ANDROID, new ClassFileAnalyzer(context, Process.TRAVERSE_ANDROID, graphBuilder, new ArrayList<>(handlers))));
GlobalMainProcessHandlerListener.INSTANCE.finishTraverseAndroidJar(handlers, null);
} catch (Exception e) {
GlobalMainProcessHandlerListener.INSTANCE.finishTraverseAndroidJar(handlers, e);
throw e;
}
timer.stopRecord("ANDROID"."Process android jar cost time = [%s ms]");
timer.startRecord("SAVECACHE");
mClassGraph = graphBuilder.build(); // Class diagram construction
timer.stopRecord("SAVECACHE"."Process saving cache cost time = [%s ms]");
}
GlobalMainProcessHandlerListener.INSTANCE.startTransform(handlers);
timer.stopRecord("PRE_PROCESS"."Collect info cost time = [%s ms]");
timer.startRecord("PROCESS");
// Execute the Transform task
transform(getProcessors(Process.TRANSFORM, new ClassFileTransformer(new ArrayList<>(handlers), needPreVerify(), needVerify())));
timer.stopRecord("PROCESS"."Transform cost time = [%s ms]");
}
Copy the codeCompared to onPreTransform(), runTransform() is also traversed by traverseArtifactOnly(), with android.jar handling added. Finally, transform(getProcessors(process. transform, new ClassFileTransformer() is called to execute the transform task.
At this point, the logic for the core of ByteX’s Common layer comparison is covered. This part of the code for listening and state design is more complex, need to understand the details of each flow and class inheritance relationship, clear, basic logic out.
4. To summarize
The flexible use of many technologies in ByteX bytecode plug-ins, such as Lambda expressions, generics, the Stream API, and the ForkJoin framework, improves code simplicity but also makes it harder to read. Keep it up ollie!
reference
- ByteX ：https://github.com/bytedance/ByteX