introduce

ProtoBuf is a tool developed by the Google team to efficiently store and read structured data. What is structured data, as it is literally put, is data with some structure. For example, a phone book has many records of data, each containing name, ID, mail, phone number, and so on. This structure repeats.

The similar

XML and JSON can also be used to store such structured data, but data represented using ProtoBuf is more efficient and can be compressed into smaller pieces.

The principle of

ProtoBuf is a special.proto that will be language-independent through the ProtoBuf compiler The suffixed data structure files are compiled into class files specific to each programming language (Java,C/C++,Python), and the API can be called through the support libraries for each programming language provided by Google, lib. (For how to write the PROTO structure, please refer to the documentation)

ProtoBuf compiler installed

Mac : brew install protobuf

For example

1. Create a proto file message.proto

syntax = "proto3"; message Person { int32 id = 1; string name = 2; repeated Phone phone = 4; enum PhoneType { MOBILE = 0; HOME = 1; WORK = 2; } message Phone { string number = 1; PhoneType type = 2; }}Copy the code

2. Create a Java project and place the proto file in the SRC /main/proto folder

3. Compile the proto file to the Java version using the command line CD to the SRC /main directory

Protoc –java_out=./ Java./proto/*.proto

You’ll notice that your SRC /main/ Java class already generates the corresponding Java class

ProtoBuf support libraries that rely on Java versions. Here is just one example of using Gradle dependencies

Implementation 'com. Google. Protobuf: protobuf - Java: 3.9.1'Copy the code

5. Convert Java objects to ProtoBuf data

Message.Person.Phone.Builder phoneBuilder = Message.Person.Phone.newBuilder(); Message.Person.Phone phone1 = phoneBuilder .setNumber("100860") .setType(Message.Person.PhoneType.HOME) .build(); Message.Person.Phone phone2 = phoneBuilder .setNumber("100100") .setType(Message.Person.PhoneType.MOBILE) .build(); Message.Person.Builder personBuilder = Message.Person.newBuilder(); personBuilder.setId(1994); personBuilder.setName("XIAOLEI"); personBuilder.addPhone(phone1); personBuilder.addPhone(phone2); Message.Person person = personBuilder.build(); long old = System.currentTimeMillis(); byte[] buff = person.toByteArray(); System.out.println("ProtoBuf time: "+ (system.currentTimemillis () -old)); System.out.println(Arrays.toString(buff)); System.out.println("ProtoBuf data length :" + buff.length);Copy the code

6. Convert ProtoBuf data back to Java objects

System.out.println("- start decode -"); old = System.currentTimeMillis(); Message.Person personOut = Message.Person.parseFrom(buff); System.out.println("ProtoBuf decoding time: "+ (system.currentTimemillis () -old)); System.out.printf("Id:%d, Name:%s\n", personOut.getId(), personOut.getName()); List<Message.Person.Phone> phoneList = personOut.getPhoneList(); For (message.person.phone Phone: phoneList) {system.out.printf (" Phone number :%s (%s)\n", phone.getNumber(), phone.getType()); }Copy the code

To compare

To take advantage of ProtoBuf, I wrote Java classes with the same structure and converted Java objects into JSON data to compare with ProtoBuf. JSON compilation library using The GSON library provided by Google, part of the JSON code is not posted, directly show the results

Comparison results

Run 1 time

Length of data :106 - Start decode - JSON decode 1 time, duration :1 ms 【 ProtoBuf start encode 】 ProtoBuf encode 1 time, duration: 32ms ProtoBuf data length :34 n/a start decoding n/a ProtoBuf decodes once, which takes 3msCopy the code

Run ten times

Length of data :106 - start decoding - JSON decoding 10 times, duration: 4ms 【 ProtoBuf start encoding 】 ProtoBuf encoding 10 times, duration: 29ms ProtoBuf data length :34 n/a start decoding n/a ProtoBuf decodes 10 times, duration: 3msCopy the code

Run 100 times

Length of data :106 - start decode - JSON decode 100 times, duration: 8ms 【 ProtoBuf start encode 】 100 times for ProtoBuf, duration: 31ms ProtoBuf data length :34 n/a start decoding n/a ProtoBuf decodes 100 times, duration: 4msCopy the code

Run 1000 times

Length of data :106 - Start decode - JSON decode 1000 times, duration: 21ms 【 ProtoBuf start encode 】 1000 times for ProtoBuf, duration: 37ms ProtoBuf data length :34 n/a start decoding n/a ProtoBuf decodes 1000 times, duration: 8msCopy the code

Run 10,000 times

Length of JSON data :106 - Start decoding - 10000 times of JSON decoding, time: 126ms 49ms ProtoBuf data length :34 n/a start decoding n/a ProtoBuf decodes 10000 times, duration: 23msCopy the code

Run 100,000 times

Length of JSON data :106 - Start decoding - JSON decoding 100000 times, duration: 248ms 180ms [ProtoBuf start encoding] 100000 times for ProtoBuf encoding: 51ms length of ProtoBuf data :34 n/a start decoding n/a 100000 times for ProtoBuf encoding: 58msCopy the code

conclusion

Codec performance the above chestnut is just a simple sampling, in fact according to my experiment found

  • If the number of times is less than 1,000, ProtoBuf’s encoding and decoding performance is comparable to JSON, and tends to be worse than JSON.
  • If the number of times is more than 2,000, the encoding and decoding performance of ProtoBuf is much higher than that of JSON.
  • The codec performance of a ProtoBuf is significantly higher than that of JSON when the number of times exceeds 100,000.

Memory usage for a ProtoBuf is 34, whereas for JSON it is 106, so the memory usage for a ProtoBuf is only 1/3 of that for JSON.

At the end

There’s a lot to be optimized for this experiment, and even this cursory test shows the advantages of ProtoBuf.

Compatible with

The new field

  • Add the nickname field in the proto file

  • Generating a Java file

  • With the old Proto byte array data, converted to objects

    Id:1994, Name:XIAOLEI MOBILE number :100860 (HOME) MOBILE number :100100 (MOBILE) getNickname=

As a result, the conversion is successful.

Delete the field

  • Delete the name field in the proto file

  • Generating a Java file

  • With the old Proto byte array data, converted to objects

    Id:1994, Name: NULL MOBILE phone Number :100860 (HOME) MOBILE phone number :100100 (MOBILE)

As a result, the conversion is successful.

Source: my.oschina.net/xiaolei123/blog/3085607