Dubbo3 provides Triple(Dubbo3), Dubbo2 protocols, which are native to the Dubbo framework. In addition, Dubbo3 also integrates a number of third-party protocols into Dubbo’s programming and service governance architecture, including gRPC, Thrift, JsonRPC, Hessian2, REST, etc. The following focuses on Triple and Dubbo2 protocols.

Next generation RPC protocol – Triple

Triple protocol is the main protocol introduced by Dubbo3. Triple means the third generation. Through the evolution of Dubbo1.0/ Dubbo2.0 protocols and the wave of technology standardization brought by cloud native, Triple, a new Dubbo3 protocol, emerged at the historic moment.

Introduction to RPC Protocol

Protocol is the core of RPC, which regulates the content and format of data transmission in the network. In addition to the required request and response data, it usually contains additional control data, such as the serialization method of a single request, timeout time, compression method, and authentication information.

The content of the agreement includes three parts:

  • Data exchange format: defines the byte stream content of RPC’s request and response objects in network transmission, also known as serialization mode;
  • Protocol structure: define the list of fields, the semantics of each field and the arrangement of different fields;
  • Protocols specify how data is transferred across networks by defining rules, formats, and semantics. A successful RPC requires both ends of communication to be able to read and write network byte streams and convert objects according to protocol. If the two ends cannot agree on the protocol to be used, there will be a chicken fight, which cannot meet the needs of remote communication.

The design of RPC protocol needs to consider the following:

  • Versatility: unified binary format, cross-language, cross-platform, multi-transport layer protocol support
  • Extensibility: Protocols add fields, upgrade, support user extension, and attach business metadata
  • Performance: As fast As it can be
  • Penetration: can be identified and forwarded by various terminal devices: Gateway, proxy server and other common and high performance usually cannot be achieved at the same time, the protocol designer must make certain choices

HTTP / 1.1

Remote invocation solutions built on top of HTTP are more versatile than proprietary RPC protocols built directly on the TCP transport layer, such as WebServices or REST architectures, using HTTP + JSON is a de facto standard solution.

Choosing to build on top of HTTP has two big advantages:

  • The semantics and extensibility of HTTP meet the requirement of RPC call.
  • Universal, HTTP protocol is almost supported by all devices on the network, has a good protocol penetration.

But there are obvious problems:

  • In a typical request-response model, a link can have only one waiting Request at a time. It’s going to produce HOL.
  • Human Readable Headers, which uses a header transmission format that is more general and human-readable, but has poor performance
  • There is no direct Server Push support, requiring alternate modes such as Polling long-polling

gRPC

The advantages and disadvantages of building RPC protocols on top of HTTP and TCP were mentioned above. Compared with Dubbo building RPC protocols on top of TCP transport layer, Google chose to define gRPC directly on top of HTTP/2 protocol. The advantages of gRPC are inherited by HTTP2 and Protobuf.

  • The HTTP2-based protocol is simple enough, low cost for users to learn, and naturally has server push/multiplexing/flow control capabilities
  • Based on Protobuf multi-language cross-platform binary compatibility ability, providing powerful unified cross-language ability
  • The ecosystem based on the protocol itself is relatively rich, K8s/ETCD and other components of the natural support protocol, the cloud native de facto protocol standard

But there are some problems

  • The support for service governance is relatively basic, preferring the basic RPC function. The protocol layer lacks the necessary unified definition, which is not easy for users to use directly.
  • Protobuf serialization, which is strongly bound, requires a high learning cost and transformation cost, and the migration cost is not negligible for existing users of a partial language

Thinking about Triple selection

In the end, we chose grPC-compatible and built the new protocol with HTTP2 as the transport layer, namely Triple. The rise of containerized applications and microservices has led to the development of techniques for content optimization for load. Traditional communication protocols (RESTFUL or other HTTP – based custom protocols) used by clients fail to meet application performance, maintainability, scalability, and security requirements. A cross-language, modular agreement will gradually become the new application development protocol standard. Since gRPC protocol became a CNCF project in 2017, more and more infrastructure and businesses, including K8s and ETCD, have started to use gRPC ecology. As a cloud native microservization framework, Dubbo’s new protocol is also perfectly compatible with gRPC. In addition, Triple will also enhance and supplement some imperfect parts in gRPC protocol. So, does Triple protocol solve the problems we mentioned above?

  • Performance: Triple protocol adopts the policy of separating metadata and payload. In this way, intermediate devices such as gateways can avoid parsing and deserializing the payload, thus reducing the response time.
  • In terms of routing support, metadata allows users to add custom headers, so that users can easily divide clusters or route routes based on the headers. In this way, when publishing data, cutting stream gray scale or DISASTER recovery (Dr) has higher flexibility.
  • In terms of security, it supports bidirectional TLS authentication (mTLS) and other encrypted transmission capabilities.
  • In terms of ease of use, In addition to supporting Protobuf serialization recommended by native gRPC, Triple supports other serialization such as Hessian/JSON in a general way, which makes it more convenient for users to upgrade to Triple protocol. For the original Dubbo service, modifying or adding Triple protocol only needs to add a line of protocol configuration in the code block declaring the service, and the transformation cost is almost zero.

The status quo

1, fully compatible with gRPC, client/server can get through with the original gRPC client

2. At present, it has been verified by large-scale production practice and reached the production level

Features and Advantages

1. Cross-language communication ability. Traditional multi-language multi-SDK mode and Mesh cross-language mode both require a more general and easily extensible data transmission format.

Support Streaming and Bidirectional in addition to Request/Response.

3. Easy to expand, high penetration, including but not limited to Tracing/Monitoring and other support, should also be recognized by devices of all layers, gateway facilities can identify data packets, friendly deployment of Service Mesh, reduce the difficulty of users to understand.

4, a variety of serialization support, smooth upgrade.

5, Support Java user no perception upgrade, do not need to define tedious IDL file, only need to simply modify the protocol name can easily upgrade to Triple protocol.

Introduction to Triple Protocol

It is further extended based on gRPC protocol

  • Service-Version → “tri-service-version” {Dubbo service version}
  • Service-Group → “tri-service-group” {Dubbo service group}
  • Tracing-ID → “tri-trace-traceid” {tracing id}
  • Tracing- rpc-id → “tri-trace-rpcid” {_span ID _}
  • Cluster-Info → “tri-unit-info” {cluster infomation}

GRPC path specifies Service name and method name. GRPC path specifies Service name and method name. Compared with Dubbo protocol, version and group information are missing. Tracing-ID and Tracing- rpc-id represent Tracing ID and SPAN ID respectively. Cluster-info represents Cluster information and can be used to build flexible routing-related service governance capabilities, such as Cluster division.

Triple Streaming

Triple protocol compared with the traditional unARY method, more than the Streaming RPC capability provided by the present

  • Streaming for what scenarios?

In some application scenarios such as large file transfer and live broadcast, the consumer or provider needs to transmit a large amount of data with the peer end. Because the amount of data in these cases is very large, it is impossible to transmit data in a SINGLE RPC packet. So for these packets we need for packet fragmentation, transmitted through RPC calls for many times, if we have split the RPC packets for parallel transmission, then to end after related packets is disorderly, the received data needs to be sorted, relevant logic will be very complicated. However, if we carry out serial transmission of the split RPC packets, the delay of the corresponding network transmission RTT and data processing will be very large.

In order to solve the above problems and to transmit a large amount of data between consumer and provider in a pipelining way, the Streaming RPC model arises at the right moment.

Through Triple protocol Streaming RPC mode, multiple user-mode long connections (Stream) will be established between consumer and provider. Multiple streams can exist on the same TCP connection at the same time, and each Stream is identified by StreamId. Data packets on a Stream are read and written in sequence.

conclusion

The most important trend in the API space is the rise of standardized technologies. Triple protocol is the main protocol introduced by Dubbo3. It adopts hierarchical design, and its data exchange format is developed based on the Protocol Buffers (Protobuf) Protocol, which has excellent serialization/deserialization efficiency. Of course, it also supports a variety of serialization methods and many development languages. In the transport layer protocol, Triple chooses HTTP/2, and its transmission efficiency has been greatly improved compared with HTTP/1.1. In addition, HTTP/2, as a mature open standard, has rich security, flow control and other capabilities, and has good interoperability. Triple can not only be used for server-side service invocation, but also support the interaction between browsers, mobile apps and IoT devices and back-end services. At the same time, Triple protocol seamlessly supports all service governance capabilities of Dubbo3.

Under the trend of Cloud Native, the requirement of interoperability between cross-platform, cross-manufacturer and cross-environment systems will inevitably give birth to RPC technology based on open standards, and gRPC conforms to the historical trend and has been more and more widely used. In the field of micro services, the proposal and implementation of Triple protocol is a big step for Dubbo3 to move towards cloud native micro services.

Appendix: Dubbo2 Protocol SPEC

Protocol SPEC!

  • Magic – Magic High & Magic Low (16 bits)Identifies dubbo protocol with value: 0xdabb
  • Req/Res (1 bit)Identifies this is a request or response. Request – 1; Response – 0.
  • 2 Way (1 bit)Only useful when Req/Res is 1 (Request), expect for a return value from server or not. Set to 1 if need a return value from server.
  • Event (1 bit)Identifies an event message or not, for example, heartbeat event. Set to 1 if this is an event.
  • Serialization ID (5 bit)Identifies serialization type: the value for fastjson is 6.
  • Status (8 bits)Only useful when Req/Res is 0 (Response), identifies the status of response
    • 20 – OK
    • 30 – CLIENT_TIMEOUT
    • 31 – SERVER_TIMEOUT
    • 40 – BAD_REQUEST
    • 50 – BAD_RESPONSE
    • 60 – SERVICE_NOT_FOUND
    • 70 – SERVICE_ERROR
    • 80 – SERVER_ERROR
    • 90 – CLIENT_ERROR
    • 100 – SERVER_THREADPOOL_EXHAUSTED_ERROR
  • Request ID (64 bits)Identifies an unique request. Numeric (long).
  • Data Length (32)Length of the content (the variable part) after serialization, counted by bytes. Numeric (integer).
  • Variable PartEach part is a byte[] after serialization with specific serialization type, identifies by Serialization ID.

Every part is a byte[] after serialization with specific serialization type, identifies by Serialization ID

  1. If the content is a Request (Req/Res = 1), each part consists of the content, in turn is:
    • Dubbo version
    • Service name
    • Service version
    • Method name
    • Method parameter types
    • Method arguments
    • Attachments
  1. If the content is a Response (Req/Res = 0), each part consists of the content, in turn is:
    • Return value type, identifies what kind of value returns from server side: RESPONSE_NULL_VALUE – 2, RESPONSE_VALUE – 1, RESPONSE_WITH_EXCEPTION – 0.
    • Return value, the real value returns from server.

Note: For the Variable Part, the current version of dubbo framework uses JSON serialization to add an extra line break between each Part of the content as a separator, please add an extra line break after each Part of the Variable Part, such as:

Dubbo version bytes (newline character) Service Name bytes (newline character)...Copy the code