From the official account: New World grocery store

There are several implementations of Protobuf 3 missing values and default values in different projects. Today the author cold rice new fried, combined with the realization of the project and personal experience in total the following six schemes.

Add identification field

It is well known that the default value for numeric types in Go is 0 (just to take the numeric type as an example), which can cause some ambiguity in some scenarios.

In order to
is_showFor example, if the field is not present, the data in DB is not updated. If the field is present and the value is
0The data in the update DB is not visible, if the field is present and the value is
1This indicates that the data in the update DB is visible.

In the above scenario, the real problem to solve is how to distinguish between default values and missing fields. To add an identity field is to add an additional field to achieve the purpose of differentiation.

For example, add a has_show_field to indicate if is_show is a valid value. Is_show is valid if has_show_field is true, otherwise is_show is unset.

Although this scheme is straightforward, each time you set the value of is_show, you also need to set the value of has_show_field, which is very troublesome, so I do not recommend it.

Field meaning and default value distinction

Field meaning and default value distinction means that the default value of the corresponding type is not used as a valid value for the field. If is_show is 1, it means show, if is_show is 2, it means no show, and if is_show is not set, it means no show.

The only problem with this solution is that it is slightly out of line with the default habits of developers.

Use oneof

OneOf is meant to be like the C UNION data type, but many big names have found that it can identify missing fields.

message Status {
  oneof show {
    int32 is_show = 1;
  }
}
message Test {
    int32 bar = 1;
    Status st = 2;
}

After the above proto file generates the corresponding GO file, test. St is the pointer type of Status, so the default value and missing field can be distinguished through this scheme. However, I think this scheme is very unfriendly when it comes to JSON serialization. Here is my example:

// oneof to json
ot1 := oneof.Test{
  Bar: 1,
  St: &oneof.Status{
    Show: &oneof.Status_IsShow{
      IsShow: 1,
    },
  },
}
bts, err := json.Marshal(ot1)
fmt.Println(string(bts), err)
// json to oneof failed
jsonStr := `{"bar":1,"st":{"Show":{"is_show":1}}}`
var ot2 oneof.Test
fmt.Println(json.Unmarshal([]byte(jsonStr), &ot2))

The above output is as follows:

{"bar":1,"st":{"Show":{"is_show":1}}} <nil>
json: cannot unmarshal object into Go struct field Status.st.Show of type oneof.isStatus_Show

As you can see from the above output, OneOf’s JSON.Marshal output will be an extra layer, and JSON.Unmarshal will fail, so use OneOf with caution.

Using the Wrapper type

This should be the official Google solution, let’s take a look at the following example:

import "google/protobuf/wrappers.proto";
message Status {
    google.protobuf.Int32Value is_show = 1;
}
message Test {
    int32 bar = 1;
    Status st = 2;
}

Using this scheme need to introduce Google/protobuf/wrappers. Proto. After this scheme generates the corresponding GO file, test. St is also the pointer type of Status. Also, let’s look at the JSON serialization effect:

wra1 := wrapper.Test{ Bar: 1, St: &wrapper.Status{ IsShow: wrapperspb.Int32(1), }, } bts, err = json.Marshal(wra1) fmt.Println(string(bts), Err) jsonStr ` = {" bar ": 1," st ": {" is_show" : {" value ": 1}}} ` / / can be normal json var wra2 wrapper. The Test fmt.Println(json.Unmarshal([]byte(jsonStr), &wra2))

The above output is as follows:

{"bar":1,"st":{"is_show":{"value":1}}} <nil>
<nil>

JSON deserialization from wrapper is fine compared to OneOf, but the output from JSON.Marshal is an extra layer. In addition, after the author’s local test, this scheme cannot be used with Gogoproto.

Allow PROTO3 to be usedoptionalThe label

It is estimated that the previous several schemes are still not perfect in practice. So on May 16, 2020, Protoc v3.12.0 was released. The compiler allows the fields of Proto3 to be optional as well.

Here’s an example:

message Status {
  optional int32 is_show = 1;
}
message Test {
    int32 bar = 1;
    Status st = 2;
}

This solution requires the new version of Protoc and must be turned on with –experimental_allow_proto3_optional. Protoc upgrade tutorial see https://github.com/protocolbu… . Let’s look at the JSON serialization effect of this scenario

var isShow int32 = 1
p3o1 := p3optional.Test{
  Bar: 1,
  St:  &p3optional.Status{IsShow: &isShow},
}
bts, err = json.Marshal(p3o1)
fmt.Println(string(bts), err)
var p3o2 p3optional.Test
jsonStr = `{"bar":1,"st":{"is_show":1}}`
fmt.Println(json.Unmarshal([]byte(jsonStr), &p3o2))

The above output is as follows:

{"bar":1,"st":{"is_show":1}} <nil>
<nil>

According to the above results, this solution is more expected than the JSON serialization of the OneOf and Wrapper schemes. Again, I have tested this solution locally and found that it does not work with GoGoProto.

Proto2 and Proto3 are used in combination

As a loyal gogoproto user, I want to be able to distinguish between default and missing values while still using gogoproto’s features. The result is the combination of proto2 and proto3.

// proto2
message Status {
    optional int32 is_show = 2;
}
// proto3
message Test {
    int32 bar = 1 [(gogoproto.moretags) = 'form:"more_bar"', (gogoproto.jsontag) = 'custom_tag'];
    p3p2.Status st = 2;
}

Message is defined to distinguish missing fields from default values in a file with the syntax proto2, and proto3 imports proto2’s message for this purpose.

The field modified with optional generates the pointer type in Go, so it is easy to tell the missing value from the default value. Here’s a look at the JSON serialization effect of this scenario:

// p3p2 to json
p3p21 := p3p2.Test{
  Bar: 1,
  St:  &p3p2.Status{IsShow: &isShow},
}
bts, err = json.Marshal(p3p21)
fmt.Println(string(bts), err)
var p3p22 p3p2.Test
jsonStr = `{"custom_tag":1,"st":{"is_show":1}}`
fmt.Println(json.Unmarshal([]byte(jsonStr), &p3p22))

The above output is as follows:

{"custom_tag":1,"st":{"is_show":1}} <nil>
<nil>

According to the above results, this scheme can not only use various tags of gogoproto, the result is also the same as using optional effect directly in proto3. I’ve already used this in my own projects, but there’s still a word of caution: “When writing this article, I went to GitHub to look at the release log of GogoProto. The latest version of GogoProto was released on October 14, 2019. I boldly predict that GogoProto will not be updated again, so please use this solution as appropriate.”

Finally, I sincerely hope that this article can be of some help to you readers.

Note:

  1. The Go version used by the author in this paper is: GO1.15.2
  2. The version of PROTOC used by the author is 3.14.0
  3. Complete example used in the article: https://github.com/Isites/go-…