Mrale. Ph /blog/2017/0…

Original author: Mrale. Ph /

Release time: January 8, 2017

Dart: Mirrors is probably the most misunderstood, misunderstood, and overlooked component of the DART core library. It has been part of the Dart language since the beginning, but is still surrounded by a fog of uncertainty, marked as state. Even though the API hasn’t changed in a long time.

Dart: Mirrors receives little attention because it is a library capable of metaprogramming, so the average DART user rarely has to deal with it directly — instead, it is something framework/library authors must deal with.

In fact, my process for writing this article didn’t start with Dart: Mirrors, but with something completely different.

JSON deserialization

In December 2016, I was hanging out on Dart’s Slack channel (now defunct, dart-Lang Gitter instead) when I saw a user, we’ll call him Maximilian, asking how to parse JSON in Dart. If you come from JavaScript, this problem may surprise you: Parsing JSON in Dart is not the same as doing json.parse (…) in JavaScript. Is it as easy?

Yes, it’s easy: you can simply use json.decode (…) from the built-in Dart: Convert library. But there is a problem. JavaScript json. parse can return you an object, and since JavaScript objects are shapeless property clouds, code like this is perfectly fine.

let userData = JSON.parse(str);
console.log(`Got user ${userData.name} from ${userData.city}`);
Copy the code

However, each Dart object has a fixed class that completely determines its shape. Dart’s Json. Decode can give you a Map, but it can’t give you an object of a custom type.

Map userData = JSON.decode(str);
console.log("Got user ${userData['name']} from ${userData['city']}");  // OK

class UserData {
  String name;
  String city;
}

UserData userData = JSON.decode(str);  // NOT OK
console.log("Got user ${userData.name} from ${userData.city}");
Copy the code

A common solution to this problem is to write the Marshaling helper, which creates your objects from Maps.

class UserData {
  String name;
  String city;

  UserData(this.name, this.city);

  UserDate.fromJson(Map m)
      : this(m['name'], m['city']);
}

UserData userData = new UserDate.fromJson(JSON.decode(str));
console.log("Got user ${userData.name} from ${userData.city}");
Copy the code

But writing this kind of template code is a way that few people like to spend their day. This is why pub contains many packages that can be automated, and…… Applause… Some of them use Dart: Mirrors.

What is Dart: Mirrors?

If you’re not a language worker, chances are you’ve never heard of the term mirroring in relation to programming languages. It’s a one-word game: mirroring apis allow programs to reflect on themselves. Historically, it originated with SELF, like many other great virtual machine technologies. If you want to learn more about mirroring and its role in other systems, check out Gilad Bracha’s article and follow the links.

Mirror images exist to answer reflexive questions, for example. What is the type of the given object?” “, “What fields/methods does it have? “, “What is the type of the field? “, “What are the types of parameters for a given method? And performing reflective actions such as “Get the value of this field from this object!” “And” Call this method on this object! .

In JavaScript, objects are inherently reflective: to dynamically get a property by name, you can just do obj[key] and call the name of a method with a dynamic argument list, obj[name].apply(obj, args).

Dart, on the other hand, encapsulates these capabilities in the Dart: Mirrors library.

import 'dart:mirrors' as mirrors;

setField(obj, name, value) {
  // Get an InstanceMirror that allows to access obj state
  final mirror = mirrors.reflect(obj);
  // Set the field through the mirror.
  mirror.setField(new Symbol(name), value);
}
Copy the code

This API may seem a bit verbose at first glance, but things like New Symbol(name) actually exist for a reason: They remind you that the names given to classes, methods, and fields don’t last forever — they can be changed, tampered with by compilers in an attempt to reduce the size of the generated code. Keep that in mind and we’ll come back to that later.

Back to JSON deserialization

Deserialization libraries based on Dart: Mirrors are usually very easy to use: all you need to do is add an annotation to your class. For example, what the data model in Maximilian’s Slack question looks like under the Dartson annotations of Dart: Mirrors.

I renamed all the fields and classes to make them anonymous. The structure of the data model is preserved. ]

import 'package:dartson/dartson.dart';

@Entity(a)class Data {
  String something0;
  String something1;
  List<Data1> sublist0;
  List<Data2> sublist1;
  List<Data3> sublist2;
}

@Entity(a)class Data1 {
  String something0;
  String something1;
  String something2;
  String something3;
  String something4;
  String something5;
  String something6;
}

@Entity(a)class Data2 {
  String something0;
  String something1;
}

@Entity(a)class Data3 {
  String something0;
  String something1;
}

final DSON = new Dartson.JSON();

// Deserialize Data object from a JSON encoded string.
Data d = DSON.decode(string, new Data());
Copy the code

One might ask: If things are so simple and smooth, what is the problem?

Dartson, it turns out, is really, really slow. When Maximilian holds up a 9Mb data file that contains the data he needs, he sees the following.

$ dart decode-benchmark.dart --with-dartson
loaded data.json: 9389696 bytes
DSON.decode(...) took: 2654ms
$ d8 decode-benchmark.js
loaded data.json: 9389696 bytes
JSON.parse(...) took: 118ms
Copy the code

These numbers don’t look very good. Let’s do some more measurements.

$ dart decode-benchmark.dart --with-json
loaded data.json: 9389696 bytes
JSON.decode(...) took: 236ms
Copy the code

So just decoding JSON strings into unstructured Maps and Lists forest is 10 times faster than using Dartson? This is pretty frustrating, and Dart: Mirrors is the obvious culprit here! At least that’s the consensus on the subject on Slack channels.

Before we dig any deeper, I’d like to make a comment here. V8’s JSON parser is written in C++, and Dart’s is actually written in Dart, which is a streaming parser. When you realize this, Dart’s Json.decode (…) And the V8 JSON. Parse (…). That two-fold difference doesn’t look so bad.

Another interesting thing is to try to use other languages as benchmarks to deserialize JSON with reflection: Maximilian tried the Encoding/JSON package for Go.

import (
  "io/ioutil"
  "encoding/json"
  "fmt"
  "time"
)

type Data struct {
  Something0 string `json:"something0"`
  Something1 string `json:"something1"`
  Sublist0 []Data1 `json:"sublist0"`
  Sublist1 []Data2 `json:"sublist1"`
  Sublist2 []Data3 `json:"sublist2"`
}

var data Data
err := json.Unmarshal(data, &data)
Copy the code

$ go run decode-benchmark.go
loaded data.json: 9389696 bytes
json.Unmarshal(...) took: 279ms
Copy the code

So, in Go, it takes roughly the same amount of time to deserialize JSON into a structure as it does to deserialize JSON into an unstructured Maps in Dart, although encoding/JSON does use reflection.

Below is a graph of how long it takes Maximilian to parse the same 9MB input file (running 30 times in the same process).

Dartson is not in the picture as it is at least 10 times slower than anything else……

Delve into Dartson’s performance

As always, one of the easiest ways to study the performance of Dart code is to use Observatory.

$dart --observe decode-benchmark.dart --dartson Observatory listening on http://127.0.0.1:8181/ loaded data.json: 9389696 bytes ...Copy the code

Looking at the CPU profile page in the observatory, I found disturbing images.

This picture tells us that Dartson spent a lot of time interpolating strings. If you look at the source code, Dartson made a lot of these notes.

void _fillObject(InstanceMirror objMirror, Map filler) {
  // ...
  _log.fine("Filled object completly: ${filler}");
}
Copy the code

And like this

Object _convertValue(TypeMirror valueType, Object value, String key) {
  // ...
  _log.finer('Convert "${key}": $value to ${symbolName}');
  // ...
}
Copy the code

This obviously wastes a lot of time, because string interpolation is done even when logging is disabled. Therefore, the first step to achieving higher performance deserialization is to remove all of these logs completely.

$ dart decode-benchmark.dart --with-dartson
loaded data.json: 9389696 bytes
DSON.decode(...) took: 1542ms
Copy the code

Voila! We just improved JSON deserialization by 42% using Dartson by changing something that had nothing to do with mirroring or JSON. If we look at the data again, we’ll see that things are finally starting to get interesting.

Here, we seem to be repeatedly asking the mirror to compute the metadata associated with certain declarations. Let’s look at dartson._fillobject.

void _fillObject(InstanceMirror objMirror, Map filler) {
  ClassMirror classMirror = objMirror.type;

  classMirror.declarations.forEach((sym, decl) {
    // Look at the declaration (e.g. a field) and make a decision
    // how to deserialize it from the given [filler] based on
    // the metadata associated with it and field's type.
  });
}
Copy the code

Whenever I’m faced with an optimization problem, the first question I always ask myself is “Does this code repeat something? Can it be cached?” . For dartson._fillobject, the answers are Yes and Yes. The code above iterates through all the declarations in a given class and makes very much the same decision again and again. These decisions can be cached because classes in Dart don’t change dynamically — they don’t get new fields, and fields don’t change their declared types. Let’s refactor the code.

// For each class this cache contains a list of deserialization actions:
// "take a value from JSON map, convert it and store it in the given field".
// Actions are stored as triplets linearly in the list:
//
// [jsonName_0, fieldName_0, convertion_0,
// jsonName_1, fieldName_1, convertion_1,
/ /... ]
//
final Map<ClassMirror, List> fillActionsCache = <ClassMirror, List> {};/// Puts the data of the [filler] into the object in [objMirror]
/// Throws [IncorrectTypeTransform] if json data types doesn't match.
void _fillObject(InstanceMirror objMirror, Map filler) {
  ClassMirror classMirror = objMirror.type;

  var actions = fillActionsCache[classMirror];
  if (actions == null) {
    // We did not hit the cache and we need to compute a list of actions.
  }

  // Iterate all actions and execute those that have a matching field
  // in the [filler].
  for (var i = 0; i < actions.length; i += 3) {
    final jsonName = actions[i];
    final value = filler[jsonName];
    if(value ! =null) {
      final fieldName = actions[i + 1];
      final convert = actions[i + 2]; objMirror.setField(fieldName, convert(jsonName, value)); }}}Copy the code

Note that we cache the source jsonName and the target fieldName (as a Symbol), respectively, because the fieldName does not necessarily match the source JSON Property name, since dartson supports @property (name:…). The annotation is renamed. The other thing is that we cache type conversions as a closure to avoid repeatedly looking up the type of the field and figuring out which conversions to apply.

When we first go into _fillObject with a class, we need to calculate the list of deserialization actions for that class. In essence, this is the same code that performs deserialization, but instead of populating an actual object, we now populate a list of actions to perform.

actions = [];
classMirror.declarations.forEach((Symbol fieldName, decl) {
  // We are only interested in public non-constant fields and setters.
  if(! decl.isPrivate && ((declisVariableMirror && ! decl.isFinal && ! decl.isConst) || (declis MethodMirror && decl.isSetter))) {
    String jsonName = _getName(fieldName);
    TypeMirror valueType;

    if (decl is MethodMirror) { // Setter.
      // Setters are called `name=`. Remove trailing `=`.
      jsonName = jsonName.substring(0, jsonName.length - 1);
      valueType = decl.parameters[0].type;
    } else {  // Field.
      valueType = decl.type;
    }

    // Check if the property was renamed via @Property(name: ...)
    final Property prop = _getProperty(decl);
    if(prop? .name ! =null) {
      jsonName = prop.name;
    }

    // Populate actions.actions.add(jsonName); actions.add(fieldName); actions.add(_valueConverter(valueType)); }});// No more actions will be added so convert actions list to non-growable array
// to reduce amount of indirections.
fillActionsCache[classMirror] = actions = actions.toList(growable: false);
Copy the code

In addition to this code, we have to rewrite dartson._convertValue. In the original Dartson, this function took TypeMirror and a value to convert, and returned the converted value.

Object _convertValue(TypeMirror valueType, Object value, String key) {
  if (valueType isClassMirror && ! valueType.isOriginalDeclaration && valueType.hasReflectedType && ! _hasOnlySimpleTypeArguments(valueType)) { ClassMirror classMirror = valueType;// handle generic lists
    if (classMirror.originalDeclaration.qualifiedName == _QN_LIST) {
      return _convertGenericList(classMirror, value);
    } else if (classMirror.originalDeclaration.qualifiedName == _QN_MAP) {
      // handle generic maps
      return_convertGenericMap(classMirror, value); }}// else if (...) {

  // ... various types handled here ...

  return value;
}
Copy the code

We follow the same idea as applied to _fillObject, and instead of performing the transformation, we calculate how to perform the transformation.

// Function that converts a [value] read from the JSON property to expected
// type.
typedef Object ValueConverter(String jsonName, Object value);

// Compute [ValueConverter] based on the property of the field where the
// the value taken from JSON map will be stored.
ValueConverter _valueConverter(TypeMirror valueType) {
  if (valueType isClassMirror && ! valueType.isOriginalDeclaration && valueType.hasReflectedType && ! _hasOnlySimpleTypeArguments(valueType)) { ClassMirror classMirror = valueType;// handle generic lists
    if (classMirror.originalDeclaration.qualifiedName == _QN_LIST) {
      return (jsonName, value) => _convertGenericList(classMirror, value); / / ⚠
    } else if (classMirror.originalDeclaration.qualifiedName == _QN_MAP) {
      // handle generic maps
      return (jsonName, value) => _convertGenericMap(classMirror, value); / / ⚠}}// else if (...) {

  // ... various types handled here ...

  // Identity convertion.
  return (jsonName, value) => value; / / ⚠
}
Copy the code

With this optimization, Dartson reached new performance heights.

$ dart decode-benchmark.dart --with-dartson
loaded data.json: 9389696 bytes
DSON.decode(...) took: 488ms
Copy the code

That’s 82% faster than the original performance — so we’re finally starting to get close to the numbers we saw from Go.

Can we push it further? B: Sure. By reading the code and analyzing it, we found some obvious places where we could avoid repeating reflection overhead by applying the previous optimization pattern (using reflection to calculate what to do, not what to do). For example, _convertGenericList contains the following code.

List _convertGenericList(ClassMirror listMirror, List fillerList) {
  ClassMirror itemMirror = listMirror.typeArguments[0];
  InstanceMirror resultList = _initiateClass(listMirror);
  fillerList.forEach((item) {
    (resultList.reflectee as List)
        .add(_convertValue(itemMirror, item, "@LIST_ITEM"));
  });
  return resultList.reflectee;
}
Copy the code

There are several problems with this code.

Adding items to the list one by one leads to growth and relocation — even if the list is known in advance.
Repeatedly use reflection to figure out how to transform each individual item.

Code can be easily rewritten using our _valueConverter helper.

List _convertGenericList(ClassMirror listMirror, List fillerList) {
  ClassMirror itemMirror = listMirror.typeArguments[0];
  InstanceMirror resultList = _initiateClass(listMirror);
  final List result = resultList.reflectee as List;

  // Compute how to convert list items based on itemMirror *once*
  // outside of the loop.
  final convert = _valueConverter(itemMirror);

  // Presize the list.
  result.length = fillerList.length;

  for (var i = 0; i < fillerList.length; i++) {
    result[i] = convert("@LIST_ITEM", fillerList[i]);
  }

  return result;
}
Copy the code

Another small optimization that can be done in dartson code is to cache class initiators and stay away.

// code from _valueConverter handling conversion from Map
// to object.
return (key, value) {
  var obj = _initiateClass(valueType);
  // ...
  // fill obj from value
  // ...
  return obj.reflectee;
};
Copy the code

final init = _classInitiator(valueType);
return (key, value) {
  var obj = init();
  // ...
  // fill obj from value
  // ...
  return obj.reflectee;
};
Copy the code

Again, this is exactly the same pattern we used before, so I won’t bother you with implementation details.

$ dart decode-benchmark.dart --with-dartson
loaded data.json: 9389696 bytes
DSON.decode(...) took: 304ms
Copy the code

We have now made Dartson 89% faster than its original performance! Furthermore, it turns out that the warm-up performance of this code is actually comparable to that of Go.

Can we further improve performance? Of course you can! If we look at our deserialization chain: from String to Map to the actual object forest, the first step becomes redundant. We seem to be allocating a Map just to fill it with an actual object. Can we parse a string directly into an object?

If we take a closer look at the source of the Dart VM’s JSON parser, we see that it is split into two parts: the actual parser is responsible for browsing the input (a single string or sequence of blocks arriving asynchronously) and sending events such as beginObject, beginArray, and so on to the listener that builds the object. Unfortunately, both the _ChunkedJsonParser and listener interface _JsonListener are hidden in DART :convert and cannot be used by user code.

At least until we apply this little patch.

diff --git a/runtime/lib/convert_patch.dart b/runtime/lib/convert_patch.dart
index 9606568e0e.. 0ceae279f5 100644
--- a/runtime/lib/convert_patch.dart
+++ b/runtime/lib/convert_patch.dart
@ @ - 21, 6 + 21, 15 @ @ import "dart:_internal" show POWERS_OF_TEN;
   return listener.result;
 }

+parseJsonWithListener(String json, JsonListener listener) {
+ var parser = new _JsonStringParser(listener);
+ parser.chunk = json;
+ parser.chunkEnd = json.length;
+ parser.parse(0);
+ parser.close();
+ return listener.result;
+}
+
 @patch class Utf8Decoder {
   @patch
   Converter<List<int>, T> fuse<T>(
@ @ + 76-67, 7, 7 @ @ class _JsonUtf8Decoder extends Converter<List<int>, Object> {
 /**
  * Listener for parsing events from [_ChunkedJsonParser].
  */
-abstract class _JsonListener {
+abstract class JsonListener {
   void handleString(String value) {}
   void handleNumber(num value) {}
   void handleBool(bool value) {}
Copy the code

With JsonListener and parseJsonWithListener exposed from DART: Convert, we can set up our own JsonListener, which does not create any intermediate mappings but instead fills in the actual object as the parser browses through the string.

I’ve already made a quick prototype of such a parser, and while I’m not going to show you every line of code, I’ll highlight some of my design decisions. The full code can be found in this GIST.

Convert Type to secondary data structures once in advance

/// Deserialization descriptor built from a [Type] or a [mirrors.TypeMirror].
/// Describes how to instantiate an object and how to fill it with properties.
class TypeDesc {
  /// Tag describing what kind of object this is: expected to be
  /// either [tagArray] or [tagObject]. Other tags ([tagString], [tagNumber],
  /// [tagBoolean]) are not used for [TypeDesc]s.
  final int tag;

  /// Constructor closure for this object.
  final Constructor ctor;

  /// Either map from property names to property descriptors for objects or
  /// element [TypeDesc] for lists.
  final /* Map<String, Property> | TypeDesc */ properties;
}

/// Deserialization descriptor built from a [mirrors.VariableMirror].
/// Describes what kind of value is expected for this property and how
/// how to store it in the object.
class Property {
  /// Either [TypeDesc] if the property is a [List] or an object or
  /// [tagString], [tagBool], [tagNumber] if the property has primitive type.
  final /* int | TypeDesc */ desc;

  /// Setter callback.
  final Setter assign;

  Property(this.desc, this.assign);
}
Copy the code

As our work on Dartson showed, it pays to use a mirror only once to set up secondary data structures that describe the deserialization action in some form.

In my mirror-based deserializer, I follow the same route, but do it eagerly rather than lazily, and cache the results.

Dynamically ADAPTS the order of attributes in JSON

An interesting observation from looking at JSON is that the attributes of objects of the same type usually arrive in the same order.

This means we can take a page out of V8’s game manual and dynamically adapt this order to avoid dictionary queries in Typedesc.properties for each new property.

The way I do this in my prototype is very simple — I just record the properties in the order of the first object given TypeDesc, and then try to follow the trail of the record. This approach worked well for our sample JSON, but might be too naive for full or optional properties in the real world.

class TypeDesc {
  /// Mode determining whether this [TypeDesc] is trying to adapt to
  /// a particular order of properties in the incoming JSON:
  /// the expectation here is that if JSON contains several serialized objects
  /// of the same type they will all have the same order of properties inside.
  int mode = modeAdapt;

  /// A sequence of triplets (property name, hydrate callback, assign callback)
  /// recorded while trying to adapt to the property order in the incoming JSON.
  /// If [mode] is set to [modeFollow] then [HydratingListener] will attempt
  /// to follow the trail.
  List<dynamic> propertyTrail = [];
}

class HydratingListener extends JsonListener {
  @override
  void propertyName() {
    if (desc.mode == TypeDesc.modeNone || desc.mode == TypeDesc.modeAdapt) {
      // This is either the first time we encountered an object with such
      // [TypeDesc], which means we are currently recording the [propertyTrail]
      // or we have already failed to follow the [propertyTrail] and have fallen
      // back to simple dictionary based property lookups.
      final p = desc.properties[value];
      if (p == null) {
        throw "Unexpected property ${name}, only expect: ${desc.properties.keys
            .join(', ')}";
      }

      if (desc.mode == TypeDesc.modeAdapt) {
        desc.propertyTrail.add(value);
        desc.propertyTrail.add(p.desc);
        desc.propertyTrail.add(p.assign);
      }
      prop = p;
      expect(p.desc);
    } else {
      // We are trying to follow the trail.
      final name = desc.propertyTrail[prop++];
      if(name ! = value) {// We failed to follow the trail. Fall back to the simple dictionary
        // based lookup.
        desc.mode = TypeDesc.modeNone;
        desc.propertyTrail = null;
        return propertyName();
      }

      // We are still on the trail.
      finalpropDesc = desc.propertyTrail[prop++]; expect(propDesc); }}}Copy the code

Dynamically generated setter closures

Although we mostly avoid using mirrors during parsing, we are still forced to use them to set fields when populating objects.

final Setter setField = (InstanceMirror obj, dynamic value) {
  obj.setField(fieldNameSym, value);
};
Copy the code

If we look at the source code for the Dart VM, we’ll see that it uses dynamic code generation to speed up instancemiror.setfield: It generates and caches small closures of the form (x, v) => x.$fieldName = v, and uses them to assign fields instead of going through a common runtime path.

There is an obvious optimization opportunity here: Instead of defining setField as a closure that calls Instancemiror.setfield, which in turn looks up another closure and calls it to set the field, define setField as the closure that sets the field directly.

There are two ways to achieve this.

Massage the optimization compiler for the Dart VM and teach it how to professionalize the instancemiror.setfield call site when the first parameter is a constant or pseudo-constant, i.e., an immutable capture variable. It’s a very complicated path.
Make a small patch to the source code for the Dart VM to expose dynamic code evaluation capabilities.

diff --git a/runtime/lib/mirrors_impl.dart b/runtime/lib/mirrors_impl.dart
index ec4ac55147.. a61baa3e3c 100644
--- a/runtime/lib/mirrors_impl.dart
+++ b/runtime/lib/mirrors_impl.dart
@ @ + 393-393, 6, 8 @ @ abstract class _LocalObjectMirror extends _LocalMirror implements ObjectMirror {
   }
 }

+$evaluate(String expression) => _LocalInstanceMirror._eval(expression, null);
+
 class _LocalInstanceMirror extends _LocalObjectMirror
     implements InstanceMirror {
Copy the code

And use these capabilities to rewrite our setField

final setField = mirrors.$evaluate('''(obj, value) {
  obj.reflectee.${fieldNameStr}= value; } ' ' ');
Copy the code

The results of

Our prototypes labeled μDecode and μDecode* with mirrors written in Dart performed well against the competition, coming close to the hand-tuned C++ parser used by V8.

The only difference between μDecode and μDecode* is that μDecode* uses mirrors.$evaluate to generate a field setter closure, while μDecode insists on using the common DART: Mirrors API.

You may notice that the plot also mentions a new player, Source_gen, which we haven’t encountered before.

The mirror problem

I want to be able to demonstrate that mirroring works well in the JIT compilation environment provided by the Dart VM. Unfortunately, this picture wouldn’t be complete if I didn’t show how mirrors behave in an AOT-compiled environment.

Currently, Dart has the goal of being compiled ahead of time.

Dart 2JS is an AOT compiler for the browser environment that compiles Dart to JavaScript.
Dart_boostrap is a virtual-based AOT compiler for native code.

Mirroring is really bad — THE AOT configuration of the VM currently does not support Dart :mirrors at all, and Dart 2JS only partially supports dart:mirrors, and will issue angry warnings when you compile applications that import Dart :mirrors.

AOT compilers have been put off by Dart: Mirrors because of its unlimited reflectivity, which allows you to explore, modify, and dynamically invoke anything you can touch. AOT compilers typically operate under a closed-world assumption, allowing them to determine with varying degrees of precision which instances of classes are likely to be assigned, which methods are likely to be called (and of course not), and even what is the value of arriving at a particular field. Dart: Mirrors’ reflective capabilities significantly complicate and degrade the quality of global data flow and control flow analysis. Therefore, the AOT compiler has a choice.

Refuse to implement DART :mirrors – this is what the VM chooses to do.
Make the analysis conservative when using Dart :mirrors — which is what Dart 2JS has chosen to do.
Implement more sophisticated control and data flow analysis to cope with some reflective capabilities.
Fix the DART: Mirrors API to make it more declarative to help AOT compilation.

We’ll come back to the last option in a moment, but let’s take a look at what the current dart2js choice gives us. This is what we would see if we compiled our benchmark into JavaScript with dart2js and ran it in a relatively new V8.

[Note: Dartson comes with a converter that eliminates the use of DART :mirrors, but I purposely don’t use it here because I want to measure the overhead of dart:mirrors]

It is clear that we are at least 4 times penalized for using mirrors compared to the baseline: Dart2JS output is using Json.parse, so it makes perfect sense to use V8 JSON as a baseline when trying to determine the overhead our abstraction is adding to the native JSON parser.

In addition, importing DART :mirrors can completely disable tree-shaker, so the output contains all the code for all the imported libraries — most of which will not be used by the application itself, but Dart 2JS must assume conservatively that it is possible for them to be reached through mirrors.

Web developers clearly want JavaScript output to be smaller and faster, so libraries like Source_gen came into being.

There were pub-transformers before Source_gen, but since the Dart ecosystem is moving away from them, I’m not going to go into detail here. The main difference between converters and Source_gen is that source_gen generates files on disk that you should carry around and even commit to your REPO, while converters run in memory as part of the PUB pipeline, making them invisible to tools that are not integrated with PUB. Transformers were very powerful, but they were also cumbersome, which led to their demise. In general, you don’t want to do expensive global conversions, especially if you don’t want to do them in memory. What you really want is modular transformations with cacheable intermediate results to speed up incremental builds. ]

Instead of using reflection to abstract out template code, write a tool that generates templates for you. For example, if you write in a file called decode. Dart.

import 'package:source_gen/generators/json_serializable.dart';
part 'decode.g.dart';


@JsonSerializable(a)class Data extends Object with _$DataSerializerMixin {
  final String something0;
  final List<Data1> sublist0;
  final List<Data2> sublist1;
  final String something1;
  final List<Data3> sublist2;

  Data(this.something0, this.sublist0, this.sublist1, this.something1, this.sublist2);

  factory Data.fromJson(json) => _$DataFromJson(json);
}
Copy the code

Then you run source_gen and you get a decode. G.art file that contains all the serialization/deserialization templates.

// GENERATED CODE - DO NOT MODIFY BY HAND

part of mirrors.src.source_gen.decode;

/ / * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
// Generator: JsonSerializableGenerator
// Target: class Data
/ / * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

Data _$DataFromJson(Map json) => new Data(
    json['something0'] as String,
    (json['sublist0'] as List)
        ?.map((v0) => v0 == null ? null : new Data1.fromJson(v0))
        ?.toList(),
    (json['sublist1'] as List)
        ?.map((v0) => v0 == null ? null : new Data2.fromJson(v0))
        ?.toList(),
    json['something1'] as String,
    (json['sublist2'] as List)
        ?.map((v0) => v0 == null ? null : new Data3.fromJson(v0))
        ?.toList());

abstract class _$DataSerializerMixin {
  String get something0;
  List get sublist0;
  List get sublist1;
  String get something1;
  List get sublist2;
  Map<String.dynamic> toJson() => <String.dynamic> {'something0': something0,
        'sublist0': sublist0,
        'sublist1': sublist1,
        'something1': something1,
        'sublist2': sublist2
      };
}
Copy the code

This looks bad (and still doesn’t get rid of templates entirely), but it works. The resulting code is fully visible to the AOT compiler and does not use mirroring — so the compiler works just as well as if you were writing the same template by hand — resulting in smaller and faster JavaScript output.

For comparison, the JavaScript output of the source_gen-based benchmark is 12 times smaller and 2 times faster than the Dartson-mirror-based output.

However, the source_gen based solution.

It is still about 2 times slower than the local baseline.
It doesn’t look right.

Why is it slower?

The main reason it’s slow is that we’re still assigning intermediate results: Instead of converting strings directly to objects, we first call Dart: convert.json. decode to produce a forest of Map and List objects, which we then convert into actual “typed” objects.

If we look at the Dart :convert implementation provided by Dart 2JS, we see that things get even more complicated because the JS object returned by native Json.parse is not compatible with Dart’s Map and we need to make it compatible with the Map interface.

If we don’t use Reviver to parse JSON, we simply wrap the JS objects returned from local Json.parse to_JsonMapClass, which implements Map<String, dynamic>. This wrapper acts like a proper Dart Map when the user first passes through itMap.operator[]When accessing any nested objects, be sure to lazily encapsulate or transform them — preventing “raw “JS objects from escaping into the Dart world that doesn’t know what to do with them. Nested objects (at least those that are not arrays) are transformed in the same lazy way, wrapping them as _JsonMap.
If we had parsed JSON with Reviver, we would have eagerly converted json.parse’s output to Dart Maps and Lists.

All this indirect and lazy data converting between three worlds (JavaScript flicker Map Flicker class instances) comes at a cost, and to be honest, it’s not clear that there’s a really easy way to improve it. By passing an empty Reviver in json. decode, you can eliminate the overhead of lazy marshalling, which makes new data.fromjson (json. decode(STR, reviver: _empty) is 10% faster than new data.fromjson (json.decode (STR)), but multiple copies are still a problem.

Thanks to Brian Slesinsky for pointing out a technique for using air Reviver.

One thing is clear –dart2js will probably provide first-class support for JSON in the form of special annotations like @jsonserializable. It can then support JSON serialization/deserialization with minimal replication (for example, by lazily tinkering with prototypes of objects returned by native Json.parse).

How to make it more beautiful?

Although dart2js can achieve first-class support for JSON, it is only one of many serialization formats in the world, so it seems that a better solution is needed. Source_gen does the job, but it’s not pretty. Dart: Mirrors are beautiful, but have an impact in AOT environments. What can we do?

There are at least two possible solutions.

Replace Dart :mirrors with more declarations

The problem with DART: Mirrors is that it is imperative, and its unlimited ability makes it difficult to do static analysis. If the Dart language provides a reflection library, you are required to declare it statically.

A set of reflectors that you’re going to use.
A set of classes where you apply these abilities.

This may sound restrictive — but if you recall our JSON deserialization example, you’ll see.

We only need to access the class annotated with @Entity().
We need to be able to call the default constructors of these classes.
We just need to iterate over their fields, know their types, and know how to assign them.

Many other applications of mirroring have similar limitations: there is an annotation they are looking for, and they only perform a limited subset of actions on the annotated class.

This observation led to the development of the Reflectable package as an experiment. The central idea of this package is to declaratively specify reflection capabilities.

Unfortunately, development of reflectable stalled and it never became part of the core, which meant it had to rely on the dreaded converter to generate small but fast output on dart2js. Hopefully at some point we will reevaluate the benefits of Reflectable and restore it.

Compile-time metaprogramming/macro system

You may have noticed that source_gen and Reflectable both approach the problem from a similar perspective: they generate code based on what you write. So Dart really seems to benefit from a good macro system.

This seems like something from the land of PL dreams, as no real mainstream language has a good macro system. However, I think many developers are beginning to realize that syntax-level abstraction is a very useful and powerful tool — and language designers are no longer afraid to provide overly powerful tools. Here are some examples of modern macro systems in Scala, Rust, and Julia.

With Dart, a macro system seems impossible for a long time because there are a whole bunch of parsers running (VM has one written in C++, Dart 2js has one written in Dart, and dart_analyzer has a third one originally written in Java and now using Dart). However, with recent attempts to unify all Dart tools into a common front end and generated kernel written in Dart, it is finally possible to try to introduce a macro system that will be uniformly supported by all tools, from the IDE to the VM and the Dart 2JS compiler.

It is still a dream, but one that seems within reach.

Translation via www.DeepL.com/Translator (free version)

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

[Dart] Fear of mirrors

JSON deserialization

What is Dart: Mirrors?

Back to JSON deserialization

Delve into Dartson’s performance

Convert Type to secondary data structures once in advance

Dynamically ADAPTS the order of attributes in JSON

Dynamically generated setter closures

The results of

The mirror problem

Why is it slower?

How to make it more beautiful?

Replace Dart :mirrors with more declarations

Compile-time metaprogramming/macro system

[Dart] Fear of mirrors

JSON deserialization

What is Dart: Mirrors?

Back to JSON deserialization

Delve into Dartson’s performance

Convert Type to secondary data structures once in advance

Dynamically ADAPTS the order of attributes in JSON

Dynamically generated setter closures

The results of

The mirror problem

Why is it slower?

How to make it more beautiful?

Replace Dart :mirrors with more declarations

Compile-time metaprogramming/macro system

Related Posts

My 2021- From online education to New energy vehicles

Designers, behind every successful “man” mysterious “woman” | designer diary

After reading the resume of the graduate student of Huake University, I found several problems