This article has been synchronized to my public account Code4j, welcome to see the official to play.

This is my first blog post. I participated in the creation activity of gold diggers and started my writing path together.

1. What is remote procedure call

Before going through Dubbo’s service invocation process, let’s look at what a remote procedure call is.

Remote procedure Call, or Remote Producedure Call, is simply A cross-process Call that is transmitted over the network so that the application on machine A can Call the service on machine B as if it were calling the local service.

As a simple example, imagine an e-commerce system with service modules such as user service, coupon service, order service, etc. These different services do not run in the same JVM, but run separately in different JVMS. Therefore, when the order service wants to call the coupon service, it can only call the corresponding service through the network, rather than directly making local calls to the corresponding service like the previous single application.

So what does a simple remote procedure call look like? Take a look at the picture below.

In other words, the simplest RPC call is nothing more than that the caller sends the parameters of the call to the server over the network. After receiving the call request, the server completes the local call according to the parameters and sends the results back to the caller over the network.

In this process, details such as parameter encapsulation and network transmission will be completed by the RPC framework. To improve the above picture, a complete RPC call flow looks like this:

  • A Client invokes a remote service as a local invocation.
  • The Client Stub encapsulates the information about this request (the name of the class to be invoked, the name of the method, the method parameters, and so on)RequestAnd serialize it in preparation for network communication.
  • The Client Stub finds the address of the Server and communicates with it over the SocketRequestSend to the server.
  • After receiving the request from the Client, the Server Stub deserializes the binary data intoRequest.
  • The Server Stub invokes local methods based on the invocation information.
  • The Server Stub encapsulates the result of the call toResponseAnd serialize it and send it to the client over the network.
  • The Client Stub receives the response and deserializes it toResponse, the remote call ends.

2. Dubbo remote call process

This section is based on Dubbo 2.6.x and uses the Demo provided on the official website to analyze synchronous calls.

In the previous section, we learned a little bit about the process of service invocation. In fact, when Dubbo implements remote calls, the core process is exactly the same as the above image, but Dubbo has added some additional processes, such as cluster fault tolerance, load balancing, filter chains, and so on.

This article examines only the core invocation flow; other additional flows are left to your own devices.

Before explaining how Dubbo is called, let’s look at some of the concepts behind Dubbo.

  • Invoker: In Dubbo, the entity domain represents the object model to operate on. This is a bit like a Bean in Spring. All operations are performed around this entity domain.

    • Represents an executable to which you can initiateinvokeThe call. It could be a local implementation, a remote implementation, or a cluster implementation.

  • Invocation: Used as a session domain in Dubbo to represent the transient state of each operation, created before and destroyed after the Invocation.

    • In fact, it is the call information, which stores the call class name, method name, parameters and other information.

  • Protocol: As a service domain in Dubbo, responsible for the life cycle management of the entity domain and session domain.

    • BeanFactory in Spring is the entry point to the product.

2.1 The beginning of remote invocation – dynamic proxy

With these basic concepts in mind, let’s start tracing the flow of remote calls to Dubbo. In RPC frameworks, proxy objects are essential for remote calls because they mask many of the underlying details and make us unaware of remote calls.

If you’ve used JDK dynamic proxies or CGLIB dynamic proxies, you know that each proxy object has a handler that handles enhancements to dynamic proxies, Examples include the InvacationHandler used by the JDK or CGLIB’s MethodInterceptor. In Dubbo, dynamic proxies are implemented using javasisst by default, which uses InvocationHandler for proxy enhancement just like JDK dynamics.

The following is the result of decompilating the proxy class using javasisst and using JDK dynamic proxies, respectively.

As you can see from the above, all the InvacationHandler does is wrap the invocation information Invacation based on the method name and method parameters of the call and pass it to the holding Invoker object. This is where you really get into Dubbo’s core model.

2.2 Invoking links on the Client

Before we look at the client call link, we need to take a look at the overall design of Dubbo. The following is a framework design from the Dubbo website, which shows the structure of the entire framework.

To make it easier to understand, I have abstracted the Proxy layer, Cluster layer, and Protocol layer in the figure above.

As shown in the figure below, Dubbo’s Proxy layer first interacts with the lower Cluster layer. The function of the Cluster layer is to disguise multiple Invokers as a ClusterInvoker and expose it to the upper layer for use. The ClusterInvoker is responsible for fault-tolerant related logic, such as fast failure, failure retry and so on. The fault-tolerant logic at this level is transparent to the upper-level Proxy.

Therefore, when the InvocationHandler of the Proxy layer delegates the call request to the holding Invoker, it is actually passed down to the corresponding ClusterInvoker, and after obtaining the available Invoker, the Invoker is filtered according to the routing rules. And load balancing to select the Invoker to call a series of operations, will get a specific protocol Invoker.

This specific Invoker may be a remote implementation, such as DubboInvoker for the default Dubbo protocol, or a local implementation, such as Injvm for InjvmInvoker, etc.

For cluster-related Invoker, take a look at MockClusterInvoker for service degradation, AbstractClusterInvoker abstract parent class and FailoverClusterInvoker, the default and most common FailoverClusterInvoker cluster policy. In fact, by default, cluster invocation links pass through these three classes one by one.

By the way, the specific protocol Invoker is retrieved through a chain of filters, and each filter does some processing for the request, such as MonitorFilter for statistics, ConsumerContextFilter for current context information, and so on. Filter this part provides users with a lot of space to expand, if you are interested in their own understanding.

DubboInvoker (DubboInvoker, DubboInvoker, DubboInvoker, DubboInvoker)

As you can see, Dubbo makes some distinctions between synchronous, asynchronous, and single calls.

The first thing to be clear about is that synchronous or asynchronous calls are from the user’s point of view, but at the network level, all interactions are asynchronous, the network framework is only responsible for sending data out, or passing it up, The network framework does not know whether the binary data sent and received is one-to-one.

So, when the user chooses synchronous invocation, in order to convert the underlying asynchronous communication to synchronous operation, Dubbo needs to invoke a blocking operation that blocks the user thread until the result of this call is returned.

2.3 The network layer, the cornerstone of remote invocation

In DubboInvoker in the previous section, we saw that the request for a remote call is sent out through an ExchangeClient class that is in the Exchange message Exchange layer of the Dubbo framework’s telecommuting module.

As can be seen from the previous architecture diagram, the remote communication module is divided into three layers: Exchange, Transport and Serialize. Each layer has its own specific role.

Start with the Serialize layer, which is responsible for serialization/deserialization. It abstracts various serialization methods, such as JDK serialization, Hessian serialization, JSON serialization, etc.

On top is the Transport layer, which is responsible for one-way Message Transport and emphasizes the semantics of Message rather than the concept of interaction. This layer also abstracts various NIO frameworks, such as Netty, Mina, etc.

The Exhange layer, different from the Transport layer, is responsible for the Request/response interaction, emphasizing a kind of Request and Reponse semantics. It is precisely because of the existence of Request and response that there is a distinction between Client and Server.

After understanding the layered structure of the remote communication module, let’s take a look at the core concepts in the module.

In this module, Dubbo extracts the concept of an Endpoint Endpoint, where an Endpoint can be uniquely identified by an IP and a Port. Between these two endpoints, we can establish a TCP connection, which is abstracted by Dubbo as a Channel. The Channel processor, ChannelHandler, is responsible for processing the Channel, such as connection establishment events, connection disconnection events. Handles read data, sent data, caught exceptions, etc.

Meanwhile, to semantically distinguish endpoints, Dubbo abstracts the endpoint that initiates the request as the Client and the endpoint that sends the response as the Server. Dubbo abstracts a Transporter interface on top of the Client and Server to avoid the direct dependence of the upper layer interface on the specific NIO library because different NIO frameworks have different external interfaces. This interface is used to obtain Client and Server. If you need to change the NIO library, you only need to replace the relevant implementation classes.

Dubbo abstracts the processor responsible for the data Codec function into a Codec interface, if you are interested.

The main function of an Endpoint is to send data, so Dubbo defines a send() method for it. At the same time, let the Channel inherit the Endpoint, so that it has the function of adding K/V attributes on the basis of sending data.

For the client, a Cleint is associated with only one Channel, so it can directly inherit a Channel to send data, while the Server can accept Channel connections established by multiple CLEInts. So instead of having it inherit a Channel, Dubbo chose to have it inherit the Endpoint directly, and provided the getChannels() method to get the associated connection.

In order to reflect the interaction mode of request/response, based on Channel, Server and Client, ExchangeChannel, ExchangeServer and ExchangeClient interfaces are further abstracted. Request () method is added for the ExchangeChannel interface, as shown in the following class diagram.

With the concepts of the network layer behind it, let’s go back to the DubboInvoker, which initiates requests through an ExchangeClient it holds when invoked synchronously. In fact, this call ends up being received by the HeaderExchangeChannel class, which implements ExchangeChannel and therefore also has the capability to request it.

As you can see, the Request () method simply wraps the data into a Request object, constructs the semantics of the request, and ultimately sends the data one way through send(). The following is an invocation link diagram of the client sending the request.

Of note here is the creation of the DefaultFuture object. The DefaultFuture class is modeled by Dubbo on the Future class in Java, which means it can be used for asynchronous operations. Each Request object has an ID, and when DefaultFuture is created, the Request ID is mapped to the created DefaultFutrue, and the timeout is set.

The purpose of saving the mapping is because in the asynchronous case, the request and response are not one-to-one. In order to ensure that the subsequent received response can be properly processed, Dubbo will put the corresponding request ID in the response. When the response is received, the corresponding DefaultFuture can be found according to the request ID, and the response result is set to DefaultFuture. Allows the user thread blocking at get() to return in time.

The whole process can be abstracted into the sequence diagram below.

When ExchangeChannel calls send(), the data is sent out through the underlying NIO framework, but there is one last step to serialize and encode the data before it can be sent over the network.

Note that until the send() method is called, all the logic is handled by the user thread, while the coding is handled by Netty’s I/O thread. For those interested, see Netty’s threading model.

2.4 Protocol and Code

Protocols and codes have been mentioned many times, but what is a Protocol and what is a code?

In fact, colloquially speaking, protocol is a set of agreed communication rules. For example, if John and John want to communicate, they need to agree how to communicate before they communicate. For example, they agree that when they hear “Hello World”, the other party will start to speak. At this point, this agreement between Sam and Sam is their communication protocol.

Encoding, on the other hand, is actually assembling data into a format specified by the agreed protocol. When Sam wants to say “good morning”, all he has to do is add the agreed “Hello World” before “good morning”, which means that the final message is “Good morning Hello World”. As soon as Li Si hears “Hello World”, he knows that the following content is what Zhang SAN wants to say. Through this form, Zhang SAN and Li Si can complete normal communication.

Specific to the actual RPC communication, the so-called Dubbo protocol, RMI protocol, HTTP protocol and so on, they just have different corresponding communication rules, but the final function is the same, is to provide a set of rules for the assembly of communication data, nothing more.

Here is a graph from the official website showing the default Dubbo protocol packet format.

Dubbo packets are divided into a header and a body. The message header is a fixed-length format with a total of 16 bytes, which is used to store some meta information, such as the Magic Number of the start identifier of the message, the type of the packet, the ID of the serialization method used, and the length of the message body. The message body is in variable length format and the length is stored in the header that stores the Invocation information or the Invocation result, i.e. the sequence of bytes from the Invocation serialized or the object returned from the remote Invocation. The data in the message body is handled by serialization/deserialization.

As mentioned earlier, Dubbo abstracts the channel processor used to encode and decode the data into a Codec interface, so before the message is sent, Dubbo calls the encode() method of that interface to encode. The message body, that is, the call information Invacation of this call, is serialized through the Serialization interface.

When Dubbo starts the client and server, it ADAPTS the coDEC-related codecs to Netty’s pipeline using the adapter pattern. See also NettyCodecAdapter, NettyClient, and NettyServer.

The following is the relevant coding logic, which is better compared to the above.

Once encoded, the data is sent by the NIO framework and sent over the network to the server.

2.5 Invocation link on the Server

When the server receives data, the first step is to decode it because it is a sequence of bytes, which is ultimately handed over to the DECODE method of the Codec interface.

The header is parsed and the body is deserialized to the DecodeableRpcInvocation object (the invocation message) based on the meta information in the header, such as header length and message type.

In this case, the THREAD is the Netty I/O thread, which may not decode the Request object in the current thread, so it may obtain a partially decoded Request object. For detailed analysis, see the following.

It is worth noting that in version 2.6.x, decoding of requests is performed by default in the I/O thread, whereas in versions after 2.7.x it is handed over to the business thread.

I/O threads are the threads in the underlying communication framework that receive requests (essentially Worker threads in Netty), and business threads are the threads in the pool of threads inside Dubbo that handle requests/responses. If an event is likely to be too time consuming to execute on an I/O thread, it needs to be dispatched to a thread pool via a thread dispatcher.

When a server receives a request, it will dispatch the request to the thread pool for execution according to different thread dispatch policies. The Thread Dispatcher does not have thread-dispatch capability per se, it is simply used to create a ChannelHandler with thread-dispatch capability.

Dubbo has five thread dispatch policies, and the default policy is all. See the following table for specific policy differences.

strategy use
all All messages are dispatched to the thread pool, including requests, responses, connection events, disconnect events, and so on
direct None of the messages are dispatched to the thread pool and are executed directly on the IO thread
message Only request and response messages are dispatched to the thread pool; all other messages are executed on the IO thread
execution Only request messages are dispatched to the thread pool, with no response. All other messages are executed on the IO thread
Connection On an IO thread, disconnection events are queued, executed one by one, and other messages are dispatched to the thread pool

The data processed by the DubboCodec decoder is passed by Netty to the next inbound processor and eventually to the corresponding ChannelHandler, such as the default AllChannelHandler, according to the configured thread dispatching policy.

As you can see, for each event, AllChannelHandler simply creates a ChannelEventRunnable object and submits it to the business thread pool for execution. The Runnable object is really just a staging post. It is intended to avoid performing specific operations in the I/O thread, and ultimately delegate the actual operations to the holding ChannelHandler.

The following figure shows the process for the server to distribute the request.

As mentioned above, decoding is also possible in a business thread, because ChannelHandler held directly in ChannelEventRunnable is a DecodeHandler used for decoding.

If decoding is required, the channel handler calls the decode method of the DecodeableRpcInvocation object created in the I/O thread and deserializes the class name, method name, parameter information, and so on from the byte sequence for the invocation.

Once decoded, the DecodeHandler passes the fully decoded Request object to the next channel handler, HeaderExchangeHandler.

At this point, you can already see the benefits of Dubbo extracting ChannelHandler to avoid coupling to a specific NIO library, while using decorator pattern to process requests layer by layer, ultimately exposing only one specific Handler to the NIO library, giving you more flexibility.

Attached is a diagram of the server ChannelHandler structure.

The HeaderExchangeHandler decides what to do based on the type of the request. If the call is one-way, it is simply called backwards, with no need to return a response. If it is a two-way call, it needs to encapsulate the Response object after obtaining the specific call result, and send the Response of this call back to the client through the holding Channel object.

The HeaderExchangeHandler delegates the call to the holding ExchangeHandler handler, which is associated with the protocol used when the service is exposed, and is generally an internal class of a protocol.

Since the Dubbo protocol is used by default, the processors in the Dubbo protocol will be examined.

The ExchangeHandler inside the Dubbo protocol finds the Invoker for this call from the list of exposed services and makes a local call to it. Note, however, that the Invoker is a dynamically generated proxy object of type AbstractProxyInvoker that holds the real object that processes the business.

When an Invoke call is made, it is done with a real object held, wrapped in an RpcResult object and returned to the lower layer.

If you are interested in RpcResult, you can read about the changes to 2.7.x asynchronization. In short, RpcResult was replaced by AppResonse to hold call results or call exceptions, and a new intermediate state class AsyncRpcResult was introduced to represent outstanding RPC calls.

This proxy object is generated when the service is exposed on the server side. Javassist dynamically generates a Wrapper class and creates an anonymous internal object to delegate the invocation to the Wrapper.

The Wrapper class is decomcompiled below, and you can see that the processing logic is similar to the InvocationHandler on the client side, which calls the real object based on the method name of the call.

At this point, the server has completed the invocation. After the lower level ChannelHandler receives the call result, it will send the response back to the client through Channel, during which it will undergo encoding serialization and other operations. Since it is similar to the encoding serialization process of the request, it will not be described here. Check out ExchangeCodec#encodeResponse() and Dubbodec# encodeResponseData() if you’re interested.

Attached is a sequence diagram of how the server processes the request.

2.6 Client Processing Response

When the client receives the response to the call, it will no doubt still need to decode and deserialize the received byte sequence, which is similar to the server decoding request process, see ExchangeCodec#decode() and DubboCodec#decodeBody() for yourself. Also refer to the sequence diagram of the server decoding request above, with only a sequence diagram of the client processing the (partially) decoded response attached.

This is mainly about the client side of the decoded Reponse object processing logic. The client’s ChannelHandler structure is not much different from the server ChnnelHandler structure above, and the decoded response is eventually passed to the HeaderExchangeHandler processor for processing.

As we mentioned when the client initiated the request, each constructed request is identified with an ID that is carried with it when the corresponding response returns. When a response is received, Dubbo finds the corresponding DefaultFuture from the requested Future mapping collection based on the returned request ID, sets the result into DefaultFuture, and wakes up the blocked user thread. This completes the conversion of Dubbo’s business thread to user thread.

If you are interested, take a look at DefauFuture’s timeout handling and the changes to the threading model after Dubbo 2.7 asynchrony.

Finally, attach an image from the source website.

At this point, a complete RPC call ends.

Due to my limited level, some details may not be clear, if you have any questions, welcome to point out, exchange and study together.

3. Reference links

  • Dubbo official website – Service invocation process

  • In-depth Understanding of Apache Dubbo and Combat