A, sequence

At the beginning of the article, I would like to talk about some of my own experiences.

Client and server to deal with, first to determine the protocol, including the selection of data protocol and convention fields. When you think of messaging protocols, you might think of XML, JSON, maybe protobuf, Protostuff, Thrift, Msgpack, Avro…

I remember when I was just out of school as an intern, I actually wrote an interface to request service data using XML protocol. Then, of course, JSON took over and gradually replaced XML as the primary data protocol on both the client and server sides. At first, we used JSONObject/JSONArray to parse the packets directly. Later, parsing frameworks such as Gson/FastJson/JackJson emerged, and a review meeting was opened to select which one to introduce. Later, Kotlin became popular, and kotlin’s own kotlinx.serialization can also do data analysis.

Moreover, serialization, whether from frameworks like Gson or Kotlinx. Serialization, is more than just message encapsulation and parsing: Because it can do json string and object conversion, i.e., serialization and deserialization, it can replace Serializable to store objects. It can be said that json is quite dominant in both message transmission and object storage.

However, as a text protocol, its performance still has some limitations. Even if it is optimized well, it still has a gap with some well-implemented binary protocol frameworks. Of course, JSON is sufficient for small amounts of data.

But there will always be cases where a better binary protocol is needed. We have encountered such a situation in a business: the amount of data in this business is relatively large, data will be triggered to upload at a certain time, before which will be accumulated. At first, we packaged all the data together into JSON strings. Later, when we found out that there were OOM cases, we changed to sharding and uploading. Although the OOM issue was solved, the data volume forced us to look for a better performance solution. At that point Protobuf, Protostuff, Thrift, Avro, and so on came into view and the technical lead decided to use Protobuf. Protobuf also lives up to expectations, with much improved performance after replacing JSON. Of course only protobuf is used in this scenario instead of other services

But protobuf is a real hassle to use, you need to write a.proto file, download and compile the software, generate the Java file, copy the file into the project, and introduce a pretty big SDK into the project… Kotlinx. serialization actually provides a Protobuf implementation, but the performance is not very useful.

After the change of work, the new project has no message data particularly large business, JSON protocol is basically enough. But the idea of a good serialization solution persisted, and eventually, I decided to implement one myself. After searching all kinds of data and spending many days, I finally realized a serialization scheme that is both efficient and easy to use.

Make for a long time, is the mule is the horse, must pull out to walk. The project name Packable is a reference to Parcelable, the Android serialization solution.

Second, usage,

2.1 General Usage

To serialize/deserialize an object, implement the above interface and then call the encoding/decoding method. Use the following example:

static class Data implements Packable {
    String msg;
    Item[] items;

    @Override
    public void encode(PackEncoder encoder) {
        encoder.putString(0, msg)
                .putPackableArray(1, items);
    }

    public static final PackCreator<Data> CREATOR = decoder -> {
        Data data = new Data();
        data.msg = decoder.getString(0);
        data.items = decoder.getPackableArray(1, Item.CREATOR);
        return data;
    };
}

static class Item implements Packable {
    int a;
    long b;

    Item(int a, long b) {
        this.a = a;
        this.b = b;
    }

    @Override
    public void encode(PackEncoder encoder) {
        encoder.putInt(0, a);
        encoder.putLong(1, b);
    }

    static final PackArrayCreator<Item> CREATOR = new PackArrayCreator<Item>() {
        @Override
        public Item[] newArray(int size) {
            return new Item[size];
        }

        @Override
        public Item decode(PackDecoder decoder) {
            return new Item(
                    decoder.getInt(0),
                    decoder.getLong(1)); }}; }static void test(a) {
    Data data = new Data();
    / / the serialization
    byte[] bytes = PackEncoder.marshal(data);
    // Deserialize
    Data data_2 = PackDecoder.unmarshal(bytes, Data.CREATOR);
}
Copy the code

  • 1. Declare implements Packable interface; 2. Implement encode() method to encode each field (PackEncoder provides various types of API); 3. Call the PackEncoder. Marshal () method, pass in the object, and get the byte array.

  • 1. Create a static object that is an instance of PackCreator. 2, implement decode() method, decode each field, assign value to the object; Call packdecoder.unmarshal (), passing in the byte array and the PackCreator instance, and get the object.

If you want to deserialize an array of objects, you need to create an instance of PackArrayCreator. PackArrayCreator inherits from PackCreator and adds a newArray method. It simply returns an array of objects of the corresponding type.

Those of you who have used Parcelable should be familiar with this notation. The difference is that Packable put/get requires index. The index is added because the field can be read correctly when it is added or subtracted. Parcelable, however, writes value directly in sequence. The read and write fields need to be exactly the same. Therefore, data exchange for memory is possible, but persistence is not recommended.

2.2 Direct Coding

The above example is just one of the examples, the specific use of the process, can be used flexibly. PackCreator doesn’t have to be created in the class that you want to deserialize. It can be created anywhere else, and it can be named whatever you want. 2. If only serialization (sender) is required, then only Packable can be implemented, PackCreator is not required, and vice versa. 3. If there is no class definition, or it is not convenient to rewrite the class, you can also directly encode/decode.

static void test2(a) {
    String msg = "message";
    int a = 100;
    int b = 200;

    PackEncoder encoder = new PackEncoder();
    encoder.putString(0, msg)
                .putInt(1, a)
                .putInt(2, b);
    byte[] bytes = encoder.getBytes();

    PackDecoder decoder = PackDecoder.newInstance(bytes);
    String dMsg = decoder.getString(0);
    int dA = decoder.getInt(1);
    int dB = decoder.getInt(2);
    decoder.recycle();
}
Copy the code

2.3 Custom coding

Consider the following class:

class Info  {
    public long id;
    public String name;
    public Rectangle rect;
}
Copy the code

Rectangle is a class in the JDK), and has four fields:

class Rectangle {
  int x, y, width, height;
}
Copy the code

Of course, there are a number of ways to do this (having Rectangle implementation Packable isn’t one of them, since you can’t modify the JDK). Packable provides an efficient (execution) method:

public static class Info implements Packable {
    public long id;
    public String name;
    public Rectangle rect;

    @Override
    public void encode(PackEncoder encoder) {
        encoder.putLong(0, id)
                .putString(1, name);
        // Return PackEncoder's buffer
        EncodeBuffer buf = encoder.putCustom(2.16);     // 4 int, 16 bytes
        buf.writeInt(rect.x);
        buf.writeInt(rect.y);
        buf.writeInt(rect.width);
        buf.writeInt(rect.height);
    }

    public static final PackCreator<Info> CREATOR = decoder -> {
        Info info = new Info();
        info.id = decoder.getLong(0);
        info.name = decoder.getString(1);
        DecodeBuffer buf = decoder.getCustom(2);
        if(buf ! =null) {
            info.rect = new Rectangle(
                    buf.readInt(),
                    buf.readInt(),
                    buf.readInt(),
                    buf.readInt());
        }
        return info;
    };
}
Copy the code

In general, it’s not uncommon for large objects to nest small objects with fixed fields. Using this method, you can reduce the recursion level, and reduce the index parsing, can improve a lot of efficiency,

2.4 Type Support

This is the overall serialization/deserialization usage of Packable. PackEncoder/PackDecoder, what types are supported? PackEncoder is used as an example. Some interfaces are as follows:

Third, performance testing

In addition to protobuf, I also selected gson for comparison.

In terms of space, the serialized data size is as follows:

Data size (byte)
packable 2537191 (57%)
protobuf 2614001 (59%)
gson 4407901 (100%)

In terms of time consuming, two sets of data were tested on PC and mobile phones:

  1. Macbook Pro
Serialization time (ms) Deserialization time (ms)
packable 9 8
protobuf 19 11
gson 67 46
  1. The glory of 20 s
Serialization time (ms) Deserialization time (ms)
packable 32 21
protobuf 81 38
gson 190 128

Four,

The design and implementation of the Packable reference Parcelable and Protobuf, but different. Compared to Protobuf, Packable is easier to use and performs better. Compared to Parcelable, Packable supports version compatibility, cross-platform support, data persistence and network transmission. Speaking of cross-platform, Packable currently implements Java, C++, C#, Objective-C, and GO.

The Java platform, currently published to the Maven repository, can be introduced straight out of the box.

dependencies {
    implementation 'the IO. Making. Billywei001: packable: 1.0.1'
}
Copy the code

Source address: github.com/BillyWei001…