Convert Protobuf to Json

preface

Recent work started using Google’s Protobuf to build the REST API, and as of now, there are no special benefits to be felt other than that the interface has been rigorously defined for Protobuf features. – Protobuf is smaller and faster than Json serialization, but given current demand, it’s unlikely there will be any need for this capability. Since it’s a brand new technology, I’m more than happy to learn.

In the code architecture of MVC, Protbuf is the technology used in the Controller layer. In order to divide each layer so that the implementation of the Service layer does not depend on Protobuf, Protobuf’s entity class, let’s call it ProtoBean, needs to be converted to poJOs. In the implementation process, there is a Protobuf to Json implementation, because of this article. ProtoBean to POJO and I’ll do that in another article, maybe several articles, because it’s a little bit more complicated.

This article has been sitting around for a long time, and I’ve been looking forward to seeing two implementations of the JsonFormat. I want to finish reading it before I write it, but I’d better write it out first. I’m tired of dragging.

In order for the reader to read the text smoothly, the links are given at the end of the article, not in the middle.

The following Protobuf file is used for this test:

syntax = "proto3";

import "google/protobuf/any.proto";

option java_package = "io.gitlab.donespeak.javatool.toolprotobuf.proto";
package data.proto;

message OnlyInt32 {
    int32 int_val = 1;
}

message BaseData {
    double double_val = 1;
    float float_val = 2;
    int32 int32_val = 3;
    int64 int64_val = 4;
    uint32 uint32_val = 5;
    uint64 uint64_val = 6;
    sint32 sint32_val = 7;
    sint64 sint64_val = 8;
    fixed32 fixed32_val = 9;
    fixed64 fixed64_val = 10;
    sfixed32 sfixed32_val = 11;
    sfixed64 sfixed64_val = 12;
    bool bool_val = 13;
    string string_val = 14;
    bytes bytes_val = 15;

    repeated string re_str_val = 17;
    map<string, BaseData> map_val = 18;
}

message DataWithAny {
    double double_val = 1;
    float float_val = 2;
    int32 int32_val = 3;
    int64 int64_val = 4;
    bool bool_val = 13;
    string string_val = 14;
    bytes bytes_val = 15;

    repeated string re_str_val = 17;
    map<string, BaseData> map_val = 18;

    google.protobuf.Any anyVal = 102;
}
Copy the code

Optional tools

ProtoBean can be transformed to Json tool has two, one is com. Google. Protobuf/protobuf – Java – util, The other is com.googlecode.protobuf-java-format/protobuf-java-format. The performance and effectiveness of the two are yet to be compared. Here are using com. Google. Protobuf/protobuf – Java – util, the reason is that protobuf – Java – the format of JsonFormat will Map format {” key “:” “, “value” : “”}, and the JsonFormat in protobuf-java-util can be serialized to the desired key-value structure.

<! -- https://mvnrepository.com/artifact/com.google.protobuf/protobuf-java-util -->
<dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java-util</artifactId>
    <version>3.7.1</version>
</dependency>

<! -- https://mvnrepository.com/artifact/com.googlecode.protobuf-java-format/protobuf-java-format -->
<dependency>
    <groupId>com.googlecode.protobuf-java-format</groupId>
    <artifactId>protobuf-java-format</artifactId>
    <version>1.4</version>
</dependency>
Copy the code

Code implementation

import com.google.gson.Gson;
import com.google.protobuf.Message;
import com.google.protobuf.util.JsonFormat;

import java.io.IOException;

/** * * <ul> * <li> The implementation cannot handle Message</li> * <li>enum type data is converted to enum string name </li> * <li> Bytes is converted to Base64 string </li> * </ul> *@author Yang Guanrong
 * @date2019/08/20 17:11 * /
public class ProtoJsonUtils {

    public static String toJson(Message sourceMessage)
            throws IOException {
        String json = JsonFormat.printer().print(sourceMessage);
        return json;
    }

    public static Message toProtoBean(Message.Builder targetBuilder, String json) throws IOException {
        JsonFormat.parser().merge(json, targetBuilder);
        returntargetBuilder.build(); }}Copy the code

For general data types, such as int, double, float, long, string can be carried out in accordance with the ideal way of transformation. In protobuf, the enum field is converted to string based on the enum name. For bytes fields, this is converted to a string of type UTF8.

Any, as well as Oneof

Any and Oneof are special types in Protobuf. If you try to convert a Oneof field to JSON, you can convert it normally. The field name is the name of the assigned Oneof field.

For Any, it’s a little more special. If you convert directly, you will get an exception like the following: The type specified by the typeUrl cannot be found.

com.google.protobuf.InvalidProtocolBufferException: Cannot find type for url: type.googleapis.com/data.proto.BaseData at com.google.protobuf.util.JsonFormat$PrinterImpl.printAny(JsonFormat.java:807)  at com.google.protobuf.util.JsonFormat$PrinterImpl.access$900(JsonFormat.java:639) at com.google.protobuf.util.JsonFormat$PrinterImpl$1.print(JsonFormat.java:709) at com.google.protobuf.util.JsonFormat$PrinterImpl.print(JsonFormat.java:688) at com.google.protobuf.util.JsonFormat$PrinterImpl.printSingleFieldValue(JsonFormat.java:1183) at com.google.protobuf.util.JsonFormat$PrinterImpl.printSingleFieldValue(JsonFormat.java:1048) at com.google.protobuf.util.JsonFormat$PrinterImpl.printField(JsonFormat.java:972) at com.google.protobuf.util.JsonFormat$PrinterImpl.print(JsonFormat.java:950) at com.google.protobuf.util.JsonFormat$PrinterImpl.print(JsonFormat.java:691) at com.google.protobuf.util.JsonFormat$Printer.appendTo(JsonFormat.java:332) at com.google.protobuf.util.JsonFormat$Printer.print(JsonFormat.java:342) at io.gitlab.donespeak.javatool.toolprotobuf.ProtoJsonUtil.toJson(ProtoJsonUtil.java:12) at io.gitlab.donespeak.javatool.toolprotobuf.ProtoJsonUtilTest.toJson2(ProtoJsonUtilTest.java:72) ...Copy the code

To solve this problem, we need to manually add the type corresponding to the typeUrl, which I took from Tomer Rothschild’s article Protocol Buffers, Part 3 — JSON Format. It took a long time before I found it. In fact, it says prominently above the print method that this method will throw an exception because there is no type of any.

/**
* Converts a protobuf message to JSON format. Throws exceptions if there
* are unknown Any types in the message.
*/
public String print(MessageOrBuilder message) throws InvalidProtocolBufferException {... }Copy the code

A TypeRegistry is used to resolve Any messages in the JSON conversion. You must provide a TypeRegistry containing all message types used in Any message fields, Distinctive Or the JSON conversion will fail because data in Any message fields is undistinctive. You don’t need to supply a TypeRegistry if you don’t use Any message fields.

Class JsonFormat.TypeRegistry @JavaDoc

The above implementation cannot handle Any data. You need to add your own TypeRegirstry to do the conversion.

@Test
public void toJson(a) throws IOException {
    // You can add more than one TypeRegistry DescriptorJsonFormat.TypeRegistry typeRegistry = JsonFormat.TypeRegistry.newBuilder() .add(DataTypeProto.BaseData.getDescriptor())  .build();// The usingTypeRegistry method will rebuild a Printer
    JsonFormat.Printer printer = JsonFormat.printer()
        .usingTypeRegistry(typeRegistry);

    String json = printer.print(DataTypeProto.DataWithAny.newBuilder()
        .setAnyVal(
            Any.pack(
                DataTypeProto.BaseData.newBuilder().setInt32Val(1235).build()))
        .build());

    System.out.println(json);
}
Copy the code

From the above implementation, it’s easy to think that for a field of type Any, all the associated Message types must be registered before it can be converted to Json properly. Similarly, when we use jsonFormat.parser ().merge(json, targetBuilder); , you must first add the associated Message to the Printer, which will inevitably lead to a lot of duplication throughout the code.

To solve this problem, I try to take all the Message Descriptor values from the Any field directly, and then create Printer, so I can get a general conversion method. In the end it failed. I thought I’d get stuck in a repeated or map paradigm, but it turns out that’s not a problem, at least not when converting protoBean to JSON. The problem is that the design of Any itself does not fulfill this requirement.

We can extract some of the code as follows:

public  final class Any 
    extends GeneratedMessageV3 implements AnyOrBuilder {

    // typeUrl_ will be a java.lang.string value
    private volatile Object typeUrl_;
    private ByteString value_;
    
    private static String getTypeUrl(String typeUrlPrefix, Descriptors.Descriptor descriptor) {
        return typeUrlPrefix.endsWith("/")? typeUrlPrefix + descriptor.getFullName() : typeUrlPrefix +"/" + descriptor.getFullName();
    }

    public static <T extends com.google.protobuf.Message> Any pack(T message) {
        return Any.newBuilder()
            .setTypeUrl(getTypeUrl("type.googleapis.com",
                                message.getDescriptorForType()))
            .setValue(message.toByteString())
            .build();
    }

    public static <T extends Message> Any pack(T message, String typeUrlPrefix) {
        return Any.newBuilder()
            .setTypeUrl(getTypeUrl(typeUrlPrefix,
                                message.getDescriptorForType()))
            .setValue(message.toByteString())
            .build();
    }

    public <T extends Message> boolean is(Class<T> clazz) {
        T defaultInstance = com.google.protobuf.Internal.getDefaultInstance(clazz);
            return getTypeNameFromTypeUrl(getTypeUrl()).equals(
                defaultInstance.getDescriptorForType().getFullName());
    }

    private volatile Message cachedUnpackValue;

    @java.lang.SuppressWarnings("unchecked")
    public <T extends Message> T unpack(Class<T> clazz) throws InvalidProtocolBufferException {
        if(! is(clazz)) {throw new InvalidProtocolBufferException("Type of the Any message does not match the given class.");
        }
        if(cachedUnpackValue ! =null) {
            return (T) cachedUnpackValue;
        }
        T defaultInstance = com.google.protobuf.Internal.getDefaultInstance(clazz);
        T result = (T) defaultInstance.getParserForType().parseFrom(getValue());
        cachedUnpackValue = result;
        returnresult; }... }Copy the code

From the code above, we can easily see that a field of type Any stores a Message of type Any and has nothing to do with the original Message value. Once saved as Any, Any will save it to value_ of ByteString and build a typeUrl_, so from an Any object, we have no way of knowing what type the Message object that was originally used to construct the Any object was (typeUrl_ just gives a description, You can’t use reflection to get the original class type. In the unpack method, the implementation uses the class to build an example object and the parseFrom method to restore the original value. At this point I’m particularly curious, why can’t the Any class hold the original value class type? You can also define value as a Message object, which is much easier to handle and does not affect serialization. There is still a lot to learn about being able to penetrate the designer’s intentions.

At the end of the day, there was no way to write a generic way to directly convert Message to JSON, as the idea was. It’s not that smart, so manually register all the messages you can.

package io.gitlab.donespeak.javatool.toolprotobuf;

import com.google.protobuf.Descriptors;
import com.google.protobuf.Message;
import com.google.protobuf.util.JsonFormat;

import java.io.IOException;
import java.util.List;

public class ProtoJsonUtilV1 {

    private final JsonFormat.Printer printer;
    private final JsonFormat.Parser parser;

    public ProtoJsonUtilV1(a) {
        printer = JsonFormat.printer();
        parser = JsonFormat.parser();
    }

    public ProtoJsonUtilV1(List<Descriptors.Descriptor> anyFieldDescriptor) {
        JsonFormat.TypeRegistry typeRegistry = JsonFormat.TypeRegistry.newBuilder().add(anyFieldDescriptor).build();
        printer = JsonFormat.printer().usingTypeRegistry(typeRegistry);
        parser = JsonFormat.parser().usingTypeRegistry(typeRegistry);
    }

    public String toJson(Message sourceMessage) throws IOException {
        String json = printer.print(sourceMessage);
        return json;
    }

    public Message toProto(Message.Builder targetBuilder, String json) throws IOException {
        parser.merge(json, targetBuilder);
        returntargetBuilder.build(); }}Copy the code

This is implemented through Gson

In the process of searching for information, I also found a transformation method completed by Gson. Converting Protocol Buffers Data to Json and Back with Gson Type Adapters by Alexander Moses. One is that protbuf plugins are still available, for example Idea is easy to find, VSCode is easy to find, and Eclipse can use Protobuf-DT (this DT will be a bit of a problem, I’ll talk about it later). The article is very clear, but my main purpose here is to change his implementation to be more generic.

This implementation is still the same JsonFormat as above, so there is no support for Any conversion. If you want to support Any, you can follow the code above, but I won’t change much here.

package io.gitlab.donespeak.javatool.toolprotobuf;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.JsonParser;
import com.google.gson.TypeAdapter;
import com.google.gson.stream.JsonReader;
import com.google.gson.stream.JsonWriter;
import com.google.protobuf.Message;
import com.google.protobuf.util.JsonFormat;
import io.gitlab.donespeak.javatool.toolprotobuf.proto.DataTypeProto;

import java.io.IOException;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;

/ * * *@author Yang Guanrong
 * @date2019/08/31 17:23 * /
public class ProtoGsonUtil {

    public static String toJson(Message message) {
        return getGson(message.getClass()).toJson(message);
    }

    public static <T extends Message> Message toProto(Class<T> klass, String json) {
        return getGson(klass).fromJson(json, klass);
    }

    /** * If this method is to be set to public, then you need to determine whether gson is an immutable object, otherwise it should not be open **@param messageClass
     * @param <E>
     * @return* /
    private static <E extends Message> Gson getGson(Class<E> messageClass) {
        GsonBuilder gsonBuilder = new GsonBuilder();
        Gson gson = gsonBuilder.registerTypeAdapter(DataTypeProto.OnlyInt32.class, new MessageAdapter(messageClass)).create();

        return gson;
    }

    private static class MessageAdapter<E extends Message> extends TypeAdapter<E> {

        private Class<E> messageClass;

        public MessageAdapter(Class<E> messageClass) {
            this.messageClass = messageClass;
        }

        @Override
        public void write(JsonWriter jsonWriter, E value) throws IOException {
            jsonWriter.jsonValue(JsonFormat.printer().print(value));
        }

        @Override
        public E read(JsonReader jsonReader) throws IOException {
            try {
                // You must use the 
      
        paradigm. You can't use Message directly. Otherwise, you won't find the newBuilder method
      
                Method method = messageClass.getMethod("newBuilder");
                // Call static methods
                E.Builder builder = (E.Builder)method.invoke(null);

                JsonParser jsonParser = new JsonParser();
                JsonFormat.parser().merge(jsonParser.parse(jsonReader).toString(), builder);
                return (E)builder.build();
            } catch (NoSuchMethodException | IllegalAccessException | InvocationTargetException e) {
                e.printStackTrace();
                throw newProtoJsonConversionException(e); }}}public static void main(String[] args) {
        DataTypeProto.OnlyInt32 data = DataTypeProto.OnlyInt32.newBuilder()
            .setIntVal(100) .build(); String json = toJson(data); System.out.println(json); System.out.println(toProto(DataTypeProto.OnlyInt32.class, json)); }}Copy the code

reference

Com. Google. Protobuf/protobuf – ja…
Com. Googlecode. Protobuf – Java – format/protobuf – ja…
Protocol Buffers, Part 3 — JSON Format
Converting Protocol Buffers data to Json and back with Gson Type Adapters
Any source @ dead simple
Any Official document @office

preface

Optional tools

Code implementation

Any, as well as Oneof

This is implemented through Gson

reference

Related Posts

JDK9 has a new round of String optimization

Eureka functionality and usability interpretation

Learn Go (2) Development tools and project directory Settings