In daily development, we might be involved in repetitive, template code work, such as fromJson/toJson methods for data models. This kind of generation rules are clear, can be templates of the code, if every time to write is very bad for fishing, after all, “stroke for a while, always stroke straight”, a good fishing tool for workers is crucial. The source_gen tool is available in Dart to help automate this with scripting. In this installment, we will briefly discuss the use of this tool and analyze its construction principles in detail. (If you’ve worked with source_gen, skip to part3.)

What is source_gen — code handling script

In my opinion, source_gen is essentially a code handling script on the Dart side (similar to Java-APT) that provides an entry point to all the code files in the current repository. This allows us to implement specific operations such as code generation, statistics, etc., through identifiers such as annotations (most common), class names, etc.

In terms of the code compilation process, it is the editing phase of the code, so you can see the generated code directly.

Let’s look at a concrete example.

What can source_gen do with json parsing

The most typical I think is json parsing: jSON_serializable

This is a nice json parsing generation library that automatically generates the corresponding fromJson/toJson methods for us by annotating the model.

import 'package:json_annotation/json_annotation.dart';

part 'example.g.dart';

/// The annotation identifies this as a Model object
@JsonSerializable(a)class Person {
  final String name;
  factory Person.fromJson(Map<String.dynamic> json) => _$PersonFromJson(json);
  Map<String.dynamic> toJson() => _$PersonToJson(this);
}
Copy the code

After executing the script, a new file is automatically generated that contains the following generation code:

part of 'example.dart';

Person _$PersonFromJson(Map<String.dynamic> json) => Person(
      name: json['firstName'] as String,);Map<String.dynamic> _$PersonToJson(Person instance) => <String.dynamic> {'name': instance.firstName,
    };
Copy the code

The advantage of this parsing versus generating the entire model from JSON data is that:

  • Later field increase/delete is convenient, you can directly change the model file, and then execute the script to generate parsing method, avoid modification to regenerate the whole model.
  • Models can be reused, and some existing models can be written to other models as fields.

Of course, as we mentioned earlier, it just provides an entry point to walk through the project code file, so what you can do is entirely up to your implementation.

How to use source_gen — develop scripts and run them

Much has been written about source_gen development, but for the following process analysis, let’s take a quick look at the key steps with an annotation handler.

1. Create a new Package, create your annotations, and your annotation handler

For example, create a TestMetadata annotation:

class TestMetadata {
  const TestMetadata();
}
Copy the code

Then create an annotation processor Generator that inherits from the Generator for Annotation.

Implement generateForAnnotatedElement method. This method will automatically filter out all the elements in the project with this annotation, and you can take them and perform any logic you need. (We can get all the classes in the project, but we just inherit from GeneratorForAnnotation which will automatically filter the classes with annotations for us)

class TestGenerator extends GeneratorForAnnotation<TestMetadata> { @override generateForAnnotatedElement( Element Element, ConstantReader annotation, BuildStep BuildStep) {/// generate the following code return "class Tessss{}"; }}Copy the code

2. Configure in the newly created Packagebuild.yamlfile

builders:
  testBuilder:
    /// The file in which your annotation program is located
    import: "package:flutter_annotation/test.dart"
    /// Annotate the corresponding constructor of the program
    builder_factories: ["testBuilder"]
    /// New file suffix generated
    build_extensions: {".dart": [".g.part"]}
    auto_apply: root_package
    build_to: source
Copy the code

Then introduce the package in the project you want to execute, add annotations and execute

flutter packages pub run build_runner build 
Copy the code

You can see the class file testModel.g.art that is automatically generated according to the script

// GENERATED CODE - DO NOT MODIFY BY HAND
/ / * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
// TestGenerator
/ / * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

class Tessss {}

Copy the code

The more detailed process for Flutter annotation processing and code generation and explains how to generate code from annotations in Dart.

Let’s look at the source_gen execution flow through jSON_serialIZABLE.

4. Execution principle of source_gen

After we introduce jSON_serialIZABLE in the project, execute

flutter packages pub run build_runner build 
Copy the code

Go to the jSON_serializable process and execute it separately if we have other scripts. That is, source_gen searches for scripts that depend on the entire project and passes in files for the current project.

The whole process is as follows:

The key process consists of three steps, each corresponding to the instruction

  • 1, Flutter packages pub
  • 2, run build_runner build
  • 3, source_gen actually executes to jSON_serialIZABLE file

Here we go step by step:

1. Flutter_tools parsing instructionflutter packages pub

Since the script starts the flutter packages pub run build_runner build command, the entry must start from the flutter command.

Dart will execute the flutter script corresponding to $FLUTTER_ROOT/bin/cache/flutter_tools.snapshot based on the SDK location we configured in the environment variable. Dart this program is compiled by the Flutter_tools project and is equivalent to flutter_tools.dart in the Tools /bin directory, where we can execute the program directly.

So the equivalent argument from this entry becomes Packages pub run build_runner build

The executable. Main (args) as follows

In the command mode, a variety of commands are configured. According to the different parameters we pass, different commands are finally selected for execution.

Packages pub eventually find PackagesPassthroughCommand executive orders, after the processing parameters are only run build_runner build.

This method internally starts a new process, passes in the run build_runner build parameter, executes the bin/pub program in the DARt-SDK directory, and communicates with the two processes through sockets.

Summary: the first section of the flutter packages pub will eventually find the corresponding command execution for us in flutter_tools. Start a new process and pass the remaining parameters for execution.

2,run build_runner build

Dart command is also used to execute the pub.dart.snapshot program in bin/pub.

This program source code indart-sdkIn simple terms, it encapsulates the interactive instructions of the PUB repository.dart run build_runner buildWill findbuild_runnerMiddle, and finally you’re left withbuildParameters.

The main method eventually leads to generateAndRun, which has three main steps

  • A,findBuildScriptOptionsFind all libraries that contain build.yaml in your project dependencies
  • B. Generated in the projectentrypoint/build.dartImport file, import all scripts
  • C. Create the ISOLATEbuild.dart

A, findBuildScriptOptions() finds and generates script information

The findBuildScriptOptions method first calls findBuildScriptOptions to find all of the dependencies in the project, sorting them by dependency, and finally finding all libraries in the dependencies that contain build.yaml files.

B. Generate entry configuration files in the project according to script information

Concatenate the information found in the previous step into a string and format it

String in the project. Dart_tools/build/entrypoint/build. The dart position is written to the file.

This serves as the entry file for all dependent scripts in your project,

C. Create the ISOLATEbuild.dart

After the entry files are generated, a new ISOLATE is created in Build_Runner to execute the entry procedures above. The two programs communicate with each other through ReceivePort.

Summary: Dart run build_runner build will find build_Runner execution, scan all scripts under the project, generate a build_. dart entry program, and create isolate execution script.

3. Execute the script jSON_serialIZABLE

When it comes to execution, it is clear that it must eventually be called to the script. This call relationship is more complicated. Below is the whole sequence diagram.

The call is quite deep, I think it is ok to understand the last part of source_gen.

Generators call generate(libraryReader, buildStep), where generators are all the scripts we configured earlier.

LibraryReader is a collection of current project code that contains all code information. It is also based on analyzer that transforms dart code into an AST(Abstract Syntax Tree) so that all class information can be accessed for our custom operations such as code generation/statistics.

According to the result of Gen. generate, if it is not empty, write it to the file.

Summary: All currently dependent scripts are configured in build.dart, starting with the main method and executing through each script. Check whether a new file is generated based on the result.

The whole process is almost finished. There are still some details to be explored, such as command mode of Flutter_tools, execution principle of DART-SDK, DART VIRTUAL machine and ISOLATE, etc. If you are interested, we can have a detailed discussion in several sessions.

Of course, the core flow from a code generation point of view is still in this diagram, so it should make sense when you understand it.

5. Problems in source_gen code generation

Finally, a word about the problem I had with source_gen, which is actually where this article came from.

  • 1. Debugging is difficult

As can be seen from the above process, the whole script execution link is very long, involving cross-process and isolate communication, which makes it very difficult to debug the script program. The only way to debug the script program is to add logs.

  • 2. Long execution time

Another example is the introduction of multiple scripts in a project, all of which are executed each time a command is executed. The more scripts, the longer the execution time. At present, there is no support to specify a package to run, and it is difficult to debug, so there is a certain development cost.

Of course, after understanding the whole construction principle, these problems should be solved in the direction of the general, please look forward to the next article ~

Six, summarized

There are issues with source_gen, but in general, it’s a good way to quickly develop basic scripts like JSON/routing. And understanding basic compilation of code by working with the AST can be followed by learning some dynamic /AOP practices. There are also some details, such as the command mode of Flutter_tools, the execution principle of DART-SDK, DART VIRTUAL machine and ISOLATE, etc. If you are interested, we can open several sessions for detailed discussion.

If you have any questions, please contact me through the official account. If the article inspires you, I hope to get your thumbs up, attention and collection, which is the biggest motivation for me to continue writing. Thanks~

The most detailed guide to the advancement and optimization of Flutter has been collected in Advance of Flutter or RunFlutter.

Past highlights:

Advance optimization of Flutter

The Flutter core rendering mechanism

Flutter routing design and source code analysis

Next period:

Flutter_tools/source_gen optimization/Dart VM theme