Thoughts and practice on dynamic thermal renewal of Flutter
Thoughts and Practice on Flutter dynamic thermal renewal (II) —-Dart code conversion AST
Thoughts and Practice on Flutter dynamic thermal renewal (iii) —- analysis of AST Runtime
Thoughts and Practice on Flutter dynamic thermal renewal (IV) —- analysis of AST widgets
Thinking and practice of Flutter dynamic thermal update (v) —- call AST dynamic code
In our previous article, “Thoughts and Practice on Flutter Dynamic Thermal Updating”, we discussed a feasible solution for Flutter dynamic thermal updating. In this article, we will discuss the first phase of the solution: how to convert Dart code into AST description files.
1. Introduction of AST
AST is also mentioned in the last article, but there is not too much explanation, so in this article, I will do a simple popular science for this noun (if you already know it, you can skip ^ ^).
The full name of AST is Abstract Syntax Tree. The Chinese name is Abstract Syntax Tree, which represents the Syntax in our source code in Tree structure. The AST does not depend on the programming language, any programming language can be converted to the AST structure, about the AST in-depth parts, such as the logical steps generated by the AST, etc. (I also do not understand this part, involving the knowledge of the compilation principle, head = =). If you want to understand what AST is, Javascript is the most appropriate language. There is an online tool for converting Javascript to AST, AST Explorer. If you are interested, you can experiment by yourself. For example, var c = a + 10 to AST would look like this:
I don’t have enough space to capture a portion of it, but you can get an idea of what the AST looks like. It’s usually organized in JSON data, which makes it easy to read and parse.
As can be seen from the AST data structure in the above screenshot, the AST structure contains many node objects. Each node object contains various attributes and other node objects. Usually, these node objects are defined in a standard way, as shown in the above example:
VariableDeclaratorGenerally defined as a declared variableBinaryExpressionGenerally defined as an operational expressionIdentifierCommonly defined as an identifier name, this identifier includes variable names, method names, class names, and so on.NumericLiteralGenerally defined as a value, including integer, floating point, and so on
There are many other node objects that have specific meanings, usually describing a syntax in the source code. For more official definitions, look for AST node objects here
2. Dart official tool Analyzer
Now that you have some knowledge and understanding of the AST, it’s time to think about how to convert the Dart code to the AST. Fortunately, the Dart official has been kind enough to provide a toolkit analyzer that provides methods to generate AST objects from a Dart source. Of course, in addition to generating AST objects, the toolkit can also do some code analysis to find syntax errors or potential risk warnings. Dartfmt is a code formatting tool, Dart Doc is a code document generation tool, and Dart Analysis Server is a code syntax preanalysis service.
3. Use the analyzer
Now let’s get down to business, which is how to generate JSON-structured AST data from a Dart code using tool methods provided by Analyzer. Let’s first look at the methods provided by Analyzer:
/// Return the result of parsing the file at the given [path].
///
/// If a [resourceProvider] is given, it will be used to access the file system.
///
/// [featureSet] determines what set of features will be assumed by the parser.
/// This parameter is required because the analyzer does not yet have a
/// performant way of computing the correct feature set for a single file to be
/// parsed. Callers that needthe feature set to be strictly correct must
/// create an [AnalysisContextCollection], query it to get an [AnalysisContext],
/// query it to get an [AnalysisSession], and then call `getParsedUnit`.
///
/// Callers that don't need the feature set to be strictly correct can pass in
/// `FeatureSet.fromEnableFlags([])` to enable the default set of features; this
/// is much more performant than using an analysis session, because it doesn't
/// require the analyzer to process the SDK.
///
/// If [throwIfDiagnostics] is `true` (the default), then if any diagnostics are
/// produced because of syntactic errors in the [content] an `ArgumentError`
/// will be thrown. If the parameter is `false`, then the caller can check the
/// result to see whether there are any errors.
ParseStringResult parseFile(
{@required String path,
ResourceProvider resourceProvider,
@required FeatureSet featureSet,
bool throwIfDiagnostics = true{...}) }Copy the codeParseStringResult: ParseStringResult: ParseStringResult: ParseStringResult: ParseStringResult: ParseStringResult: ParseStringResult
/// The result of parsing of a single file. The errors returned include only
/// those discovered during scanning and parsing.
///
/// Similar to [ParsedUnitResult], but does not allow access to an analysis
/// session.
///
/// Clients may not extend, implement or mix-in this class.
abstract class ParseStringResult {
/// The content of the file that was scanned and parsed.
String get content;
/// The analysis errors that were computed during analysis.
List<AnalysisError> get errors;
/// Information about lines in the content.
LineInfo get lineInfo;
/// The parsed, unresolved compilation unit for the [content].
CompilationUnit get unit;
}
Copy the codeIn this class structure we need to pay attention to unit, which is of type CompilationUnit, an “unprocessed CompilationUnit”. Let’s not take this literally, but let’s look at the definition of CompilationUnit:
/// A compilation unit.
///
/// While the grammar restricts the order of the directives and declarations
/// within a compilation unit, this class does not enforce those restrictions.
/// In particular, the children of a compilation unit will be visited in lexical
/// order even if lexical order does not conform to the restrictions of the
/// grammar.
///
/// compilationUnit ::=
/// directives declarations
///
/// directives ::=
/// [ScriptTag]? [LibraryDirective]? namespaceDirective* [PartDirective]*
/// | [PartOfDirective]
///
/// namespaceDirective ::=
/// [ImportDirective]
/// | [ExportDirective]
///
/// declarations ::=
/// [CompilationUnitMember]*
///
/// Clients may not extend, implement or mix-in this class.
abstract class CompilationUnit implements AstNode {
/// Set the first token included in this node's source range to the given
/// [token].
set beginToken(Token token);
/// Return the declarations contained in this compilation unit.
NodeList<CompilationUnitMember> get declarations;
/// Return the element associated with this compilation unit, or `null` if the
/// AST structure has not been resolved.
CompilationUnitElement get declaredElement;
/// Return the directives contained in this compilation unit.
NodeList<Directive> get directives;
/// Set the element associated with this compilation unit to the given
/// [element].
set element(CompilationUnitElement element);
/// Set the last token included in this node's source range to the given
/// [token].
set endToken(Token token);
/// The set of features available to this compilation unit, or `null` if
/// unknown.
///
/// Determined by some combination of the .packages file, the enclosing
/// package's SDK version constraint, and/or the presence of a `@dart`
/// directive in a comment at the top of the file.
///
/// Might be `null` if, for example, this [CompilationUnit] has been
/// resynthesized from a summary.
FeatureSet get featureSet;
/// Return the line information for this compilation unit.
LineInfo get lineInfo;
/// Set the line information for this compilation unit to the given [info].
set lineInfo(LineInfo info);
/// Return the script tag at the beginning of the compilation unit, or `null`
/// if there is no script tag in this compilation unit.
ScriptTag get scriptTag;
/// Set the script tag at the beginning of the compilation unit to the given
/// [scriptTag].
set scriptTag(ScriptTag scriptTag);
/// Return a list containing all of the directives and declarations in this
/// compilation unit, sorted in lexical order.
List<AstNode> get sortedDirectivesAndDeclarations;
}
Copy the code(To tell you the truth, this class is a bit of a puzzle to look at the comments, compiler principles of knowledge is small, headache = =)
We can see that this class implements the AstNode interface, so we can guess that this class is used to store AST data. The unit argument in ParseStringResult should be the root of the AST syntax tree. We should be able to construct the JSON-structured AST data we want. The next step is how to traverse. After looking up some information, we know that the CompilationUnit is designed in visitor mode, so we’ll traverse the entire tree as a visitor. A related class is also provided in the Analyzer toolkit, AstVisitor
, defined as follows:
/// An object that can be used to visit an AST structure.
///
/// Clients may not extend, implement or mix-in this class. There are classes
/// that implement this interface that provide useful default behaviors in
/// `package:analyzer/dart/ast/visitor.dart`. A couple of the most useful
/// include
/// * SimpleAstVisitor which implements every visit method by doing nothing,
/// * RecursiveAstVisitor which will cause every node in a structure to be
/// visited, and
/// * ThrowingAstVisitor which implements every visit method by throwing an
/// exception.
abstract class AstVisitor<R> {... }Copy the codeThe comments make it clear that objects are used to access the AST structure, and several implementation classes are provided:
SimpleAstVisitorGeneralizingAstVisitorUnifyingAstVisitorRecursiveAstVisitorThrowingAstVisitor- .
We usually use the first three classes. If we need to customize the node data we visit, we can inherit the SimpleAstVisitor or GeneralizingAstVisitor class and override the method we want to customize the processing. The GeneralizingAstVisitor or UnifyingAstVisitor generally helps us analyze what the AST structure generated by source code looks like. If we didn’t use the above implementation class, we could just implement AstVisitor, but there are 121 methods in AstVisitor. If we implement the class ourselves, that means we have to reload all 121 methods, even though there are many methods. So if you don’t have any special requirements, just inherit the SimpleAstVisitor or the GeneralizingAstVisitor and that’s pretty much what you need.
Now that we know how to traverse the AST structured data, we are ready to implement our own Visitor:
class MyAstVisitor extends SimpleAstVisitor<Map> {
/// iterate over the node
Map _safelyVisitNode(AstNode node) {
if(node ! =null) {
return node.accept(this);
}
return null;
}
/// Iterate over the node list
List<Map> _safelyVisitNodeList(NodeList<AstNode> nodes) {
List<Map> maps = [];
if(nodes ! =null) {
int size = nodes.length;
for (int i = 0; i < size; i++) {
var node = nodes[i];
if(node ! =null) {
var res = node.accept(this);
if(res ! =null) { maps.add(res); }}}}returnmaps; }}Copy the codeIn our own Visitor, we first defined two private methods for traversing a single node and a list of nodes. Next we begin to iterate from the root of the AST tree and construct Map data:
class MyAstVisitor extends SimpleAstVisitor<Map> {
/// iterate over the node
Map _safelyVisitNode(AstNode node) {
if(node ! =null) {
return node.accept(this);
}
return null;
}
/// Iterate over the node list
List<Map> _safelyVisitNodeList(NodeList<AstNode> nodes) {
List<Map> maps = [];
if(nodes ! =null) {
int size = nodes.length;
for (int i = 0; i < size; i++) {
var node = nodes[i];
if(node ! =null) {
var res = node.accept(this);
if(res ! =null) { maps.add(res); }}}}return maps;
}
// Construct the root node
Map _buildAstRoot(List<Map> body) {
if (body.isNotEmpty) {
return {
"type": "Program"."body": body,
};
} else {
return null; }}@override
Map visitCompilationUnit(CompilationUnit node) {
return_buildAstRoot(_safelyVisitNodeList(node.declarations)); }}Copy the codeThe contents below the root are parsed one by one from our source code. Let’s start with a simple example. For example, we want to generate the following AST syntax tree:
//demo_blog_code2.dart
int incTen(int a) {
int b = a + 10;
return b;
}
Copy the codeLet’s start by writing a command-lines program that uses the GeneralizingAstVisitor class mentioned above to print out what the access paths for each node in the AST syntax tree generated by this source code look like:
import 'dart:io';
import 'package:analyzer/dart/analysis/features.dart';
import 'package:analyzer/dart/analysis/utilities.dart';
import 'package:analyzer/dart/ast/visitor.dart';
import 'package:analyzer/dart/ast/ast.dart';
import 'package:args/args.dart';
void main(List<String> arguments) {
exitCode = 0; // presume success
finalparser = ArgParser().. addFlag("file", negatable: false, abbr: 'f');
var argResults = parser.parse(arguments);
final paths = argResults.rest;
if (paths.isEmpty) {
stdout.writeln('No file found');
} else {
generate(paths[0]); }}class DemoAstVisitor extends GeneralizingAstVisitor<Map> {
@override
Map visitNode(AstNode node) {
// Outputs the AST Node contents
stdout.writeln("${node.runtimeType}<---->${node.toSource()}");
return super.visitNode(node); }}/ / generated AST
Future generate(String path) async {
if (path.isEmpty) {
stdout.writeln("No file found");
} else {
await _handleError(path);
if (exitCode == 2) {
try {
var parseResult =
parseFile(path: path, featureSet: FeatureSet.fromEnableFlags([]));
var compilationUnit = parseResult.unit;
/ / traverse the AST
compilationUnit.accept(DemoAstVisitor());
} catch (e) {
stdout.writeln('Parse file error: ${e.toString()}');
}
}
}
}
Future _handleError(String path) async {
if (await FileSystemEntity.isDirectory(path)) {
stderr.writeln('error: $path is a directory');
} else {
exitCode = 2; }}Copy the codeThen we run the code above:
dart main.dart -f dsldemos/demo_blog_code2.dart
Dart is the test source code file. The output is as follows:
From this output, we can roughly determine what overloaded methods need to be handled in the Visitor. Finally, our Visitor implements:
class MyAstVisitor extends SimpleAstVisitor<Map> {
/// iterate over the node
Map _safelyVisitNode(AstNode node) {
if(node ! =null) {
return node.accept(this);
}
return null;
}
/// Iterate over the node list
List<Map> _safelyVisitNodeList(NodeList<AstNode> nodes) {
List<Map> maps = [];
if(nodes ! =null) {
int size = nodes.length;
for (int i = 0; i < size; i++) {
var node = nodes[i];
if(node ! =null) {
var res = node.accept(this);
if(res ! =null) { maps.add(res); }}}}return maps;
}
// Construct the root node
Map _buildAstRoot(List<Map> body) {
if (body.isNotEmpty) {
return {
"type": "Program"."body": body,
};
} else {
return null; }}// Construct the block Bloc structure
Map _buildBloc(List body) => {"type": "BlockStatement"."body": body};
// Construct the operation expression structure
Map _buildBinaryExpression(Map left, Map right, String lexeme) => {
"type": "BinaryExpression"."operator": lexeme,
"left": left,
"right": right
};
// Construct the variable declaration
Map _buildVariableDeclaration(Map id, Map init) => {
"type": "VariableDeclarator"."id": id,
"init": init,
};
// Construct the variable declaration
Map _buildVariableDeclarationList(
Map typeAnnotation, List<Map> declarations) =>
{
"type": "VariableDeclarationList"."typeAnnotation": typeAnnotation,
"declarations": declarations,
};
// Construct the identifier definition
Map _buildIdentifier(String name) => {"type": "Identifier"."name": name};
// Construct a numeric definition
Map _buildNumericLiteral(num value) =>
{"type": "NumericLiteral"."value": value};
// constructor declaration
Map _buildFunctionDeclaration(Map id, Map expression) => {
"type": "FunctionDeclaration"."id": id,
"expression": expression,
};
// Constructor expression
Map _buildFunctionExpression(Map params, Map typeParameters, Map body) => {
"type": "FunctionExpression"."parameters": params,
"typeParameters": typeParameters,
"body": body,
};
// Constructor arguments
Map _buildFormalParameterList(List<Map> parameterList) =>
{"type": "FormalParameterList"."parameterList": parameterList};
// Constructor arguments
Map _buildSimpleFormalParameter(Map type, String name) =>
{"type": "SimpleFormalParameter"."paramType": type, "name": name};
// Constructor parameter type
Map _buildTypeName(String name) => {
"type": "TypeName"."name": name,
};
// Construct returns the data definition
Map _buildReturnStatement(Map argument) => {
"type": "ReturnStatement"."argument": argument,
};
@override
Map visitCompilationUnit(CompilationUnit node) {
return _buildAstRoot(_safelyVisitNodeList(node.declarations));
}
@override
Map visitBlock(Block node) {
return _buildBloc(_safelyVisitNodeList(node.statements));
}
@override
Map visitBlockFunctionBody(BlockFunctionBody node) {
return _safelyVisitNode(node.block);
}
@override
Map visitVariableDeclaration(VariableDeclaration node) {
return _buildVariableDeclaration(
_safelyVisitNode(node.name), _safelyVisitNode(node.initializer));
}
@override
Map visitVariableDeclarationStatement(VariableDeclarationStatement node) {
return _safelyVisitNode(node.variables);
}
@override
Map visitVariableDeclarationList(VariableDeclarationList node) {
return _buildVariableDeclarationList(
_safelyVisitNode(node.type), _safelyVisitNodeList(node.variables));
}
@override
Map visitSimpleIdentifier(SimpleIdentifier node) {
return _buildIdentifier(node.name);
}
@override
Map visitBinaryExpression(BinaryExpression node) {
return _buildBinaryExpression(_safelyVisitNode(node.leftOperand),
_safelyVisitNode(node.rightOperand), node.operator.lexeme);
}
@override
Map visitIntegerLiteral(IntegerLiteral node) {
return _buildNumericLiteral(node.value);
}
@override
Map visitFunctionDeclaration(FunctionDeclaration node) {
return _buildFunctionDeclaration(
_safelyVisitNode(node.name), _safelyVisitNode(node.functionExpression));
}
@override
Map visitFunctionDeclarationStatement(FunctionDeclarationStatement node) {
return _safelyVisitNode(node.functionDeclaration);
}
@override
Map visitFunctionExpression(FunctionExpression node) {
return _buildFunctionExpression(_safelyVisitNode(node.parameters),
_safelyVisitNode(node.typeParameters), _safelyVisitNode(node.body));
}
@override
Map visitSimpleFormalParameter(SimpleFormalParameter node) {
return _buildSimpleFormalParameter(
_safelyVisitNode(node.type), node.identifier.name);
}
@override
Map visitFormalParameterList(FormalParameterList node) {
return _buildFormalParameterList(_safelyVisitNodeList(node.parameters));
}
@override
Map visitTypeName(TypeName node) {
return _buildTypeName(node.name.name);
}
@override
Map visitReturnStatement(ReturnStatement node) {
return_buildReturnStatement(_safelyVisitNode(node.expression)); }}Copy the codeModify our command-lines program using the Visitor implemented above:
.try {
var parseResult =
parseFile(path: path, featureSet: FeatureSet.fromEnableFlags([]));
var compilationUnit = parseResult.unit;
/ / traverse the AST
var astData = compilationUnit.accept(MyAstVisitor());
stdout.writeln(jsonEncode(astData));
} catch (e) {
stdout.writeln('Parse file error: ${e.toString()}'); }...Copy the codeThe AST data generated after re-execution (partial screenshots) :
This is the data structure of the AST syntax tree generated from the above test code. Of course, the AST tree has been simplified and some redundant nodes have been removed, mainly for the convenience of our next stage of work, parsing the AST, to prepare and reduce the pressure of our parsing.
Ok, this article shows you how to convert Dart code into an AST using a simple example. In the next article, we’ll explore how to parse the generated AST syntax tree to achieve the same performance as our source code.