This article is from the Internet Infrastructure Technology Team at OPPO. At the same time, you are welcome to follow our official account: OPPO_tech, and share OPPO cutting-edge Internet technology and activities with you.

OkHttp is probably the most widely used open source network library on the Android platform. After 6.0, Android also replaced the default implementation of internal HttpUrlConnection with OkHttp.

Most of the development of the students may have come in contact with OkHttp source, but there are few more fully read and understand, the bulk of the source code parsing the article on the current network are also point now, and put the source code for long, the analytic way is I can’t approved, hence the want to write an analytical OkHttp source of ideas.

The purpose of this article is to give a comprehensive introduction to the OkHttp source code, and to avoid Posting large sections of the source code. The parts involved in the source code will be shown as a call diagram as possible.

This selection of OkHttp source code is the latest version 4.4.0, for the reader is also a certain use of the foundation of Android development students.

The source code for OkHttp can be downloaded from Github (github.com/square/OkHt…

To get to the point, this article will break down the OkHttp source code from the following aspects:

  1. The overall structure
  2. The interceptor
  3. Task queue
  4. Connection overcommitment and connection pooling

1. Start with an example

Let’s start by looking at how one of the simplest Http requests is sent.

OkHttpClient client = new OkHttpClient();

Request request = new Request.Builder().url(“www.google.com”).build();

Response response = client.newCall(request).execute();

return response.body().string();

This piece of code is the most common use of OkHttp on a daily basis, and after you follow up the source code, you can get a more detailed flow chart to see how the internal logic flows.

There are several core classes involved, so let’s look at them one by one.

  1. OkHttpClient
  2. The Request and Response
  3. RealCall

OkHttpClient: this is the core of the OkHttp management class, all the internal logic and objects to the OkHttpClient unified management, it is generated through the Builder, the construction parameters and class members are many, here first do not do specific analysis.

Request and Response: Request is the encapsulation class of the Request we send, with url, header, method,body and other common parameters inside, Response is the result of the Request, including code, message, header,body; These two classes are defined in full compliance with the request and response content defined by the Http protocol.

RealCall: is responsible for scheduling the request (synchronously using the current thread, asynchronously using the internal thread pool of OkHttp); It is also responsible for constructing the internal logical chain of responsibility and executing the logic associated with the chain of responsibility until the results are obtained. Although OkHttpClient is the core management class of OkHttp, it is the RealCall class that actually makes the request and organizes the logic. It is responsible for both scheduling and chain of responsibility organization. Let’s focus on the RealCall logic.

The source address of the RealCal class: github.com/square/OkHt…

The RealCall class is not complex and has two of the most important methods, execute() and enqueue(), one for synchronous and one for asynchronous requests. Follow the enqueue source and found that it is only through the asynchronous thread and callback of an asynchronous invocation encapsulation, eventually logic will call to the execute () this method, and then call the getResponseWithInterceptorChain request () to obtain the results.

Seems getResponseWithInterceptorChain bearing the core logic of the entire request () method, then you just need to put this method analysis clear, whole OkHttp request process is generally understood. Since such an important method, or can not be exempt from Posting the complete source code.

val interceptors = mutableListOf<Interceptor>()
    interceptors += client.interceptors
    interceptors += RetryAndFollowUpInterceptor(client)
    interceptors += BridgeInterceptor(client.cookieJar)
    interceptors += CacheInterceptor(client.cache)
    interceptors += ConnectInterceptor
    if (!forWebSocket) {
      interceptors += client.networkInterceptors
    }
    interceptors += CallServerInterceptor(forWebSocket)

    val chain = RealInterceptorChain(interceptors, transmitter, null, 0, originalRequest, this,
        client.connectTimeoutMillis, client.readTimeoutMillis, client.writeTimeoutMillis)

    var calledNoMoreExchanges = false
    try {
      val response = chain.proceed(originalRequest)
     ............
Copy the code

From the source code as you can see, even getResponseWithInterceptorChain () method of logic is very simple, it generates a List of the List of Interceptors blocker, in order to:

  • client.Interceptors
  • RetryAndFollowUpInterceptor,
  • BridgeInterceptor
  • CacheInterceptor
  • ConnectInterceptor
  • client.networkInterceptors
  • CallServerInterceptor

These members are added to the List, creating a class called RealInterceptorChain, and the Response is obtained from chain.proceed.

By further analyzing the RealInterceptorChain and Interceptors, we get a conclusion that OkHttp divides the whole complex logic of the request into a single module named Interceptor, which is connected together through the design pattern of the chain of responsibility. Finally complete the request to get the response result.

The details of how these interceptors are strung together and what each of them does are discussed in more detail below.

2. The core of OkHttp: interceptors

Now that we know that the core logic of OkHttp is a bunch of interceptors, how are they constructed and related? This is where the RealInterceptorChain class comes in.

According to the analysis above, after RealCall adds Interceptors to the List one by one, it constructs a RealInterceptorChain object and calls chain.proceed to obtain the response result. So let’s analyze what the chain.proceed method does. In order not to let the space is too long, here is not posted source content, only to give the conclusion after analysis, we can quickly understand the source code comparison.

RealInterceptorChain source: github.com/square/OkHt…

According to the source code analysis of RealInterceptorChain, the following diagram can be obtained (some interceptors are omitted) :

Combined with the source code and the schematic, the following conclusions can be drawn:

  1. The interceptors are executed in the order they were added
  2. Execution of the interceptor from RealInterceptorChain. Proceed (), into the first interceptor execution logic
  3. Before executing, each interceptor will compose the remaining interceptors into a new RealInterceptorChain
  4. The interceptor logic is split into start, next. Proceed, and end by calling next. Proceed ()
  5. Next.proceed() represents the execution logic for all remaining interceptors
  6. All interceptors end up as a nested structure that is nested within layers

After understanding the construction process of the above interceptors, we will analyze the function and function of each interceptor one by one.

From the point of this local figure, add a total of five interceptor (do not include the custom interceptors if client. Interceptors and client.net workInterceptors, behind the two explain).

Let’s start with an overview of what each interceptor does

  • RetryAndFollowUpInterceptor – failure and redirect the interceptor
  • BridgeInterceptor — Encapsulates the request and Response interceptors
  • CacheInterceptor – A cache-related filter that updates the cache directly from the cache read
  • ConnectInterceptor – The connection service that is responsible for establishing a connection to the server — the actual request network
  • CallServerInterceptor — Performs flow operations (write the request body and obtain the response data), which is responsible for sending the request data to the server, reading the response data from the server, and encapsulating the HTTP request packet and parsing the request packet

Next, we will analyze the interceptors one by one.

2.1 RetryAndFollowUpInterceptor

Source address: github.com/square/OkHt…

According to the logic to the source, directly draw the corresponding flow chart (this logic in RetryAndFollowUpInterceptor intercept () method) :

Can be seen from the diagram above, RetryAndFollowUpInterceptor opened a while (true) cycle, and complete two important decision within the loop, as shown in figure in the blue box:

  1. When an exception is thrown inside the request, determine whether a retry is required
  2. When the response results in a 3XX redirect, a new request is built and sent

Retry logic is relatively complicated, there are the following decision logic (specific code in RetryAndFollowUpInterceptor type recover method) :

  • Rule 1: Set the retryOnConnectionFailure parameter of the client to false and no retry is performed
  • Rule 2: The requested body has already been sent, no retry is required
  • Rule 3: Special exception types are not retried (e.g. ProtocolException, SSLHandshakeException, etc.)
  • Rule 4: No more routes (including proxy and InetAddress), no retry

If none of the previous four rules is met, the current request is retried. The logic for redirection is relatively simple, and I won’t go into it here.

2.2 The difference between Interceptors and NetworkInterceptors

As mentioned earlier, the constructor of okHttpClient. Builder takes two arguments. Users can add custom interceptors via addInterceptor and addNetworkdInterceptor. Analysing the RetryAndFollowUpInterceptor we can know the difference between these two kinds of automatic interceptor.

From the front to add the order of the interceptor can know the Interceptors and networkInterceptors just a in front of the RetryAndFollowUpInterceptor, one in the back.

Combination in front of the chain of responsibility can call graph analysis, if a request within the interceptor RetryAndFollowUpInterceptor retry or redirect the N times, and all its internal nested interceptors will also be called N times, Similarly, networkInterceptors custom interceptors will also be called N times. An Interceptor, on the other hand, is called an Application Interceptor only once per request, so it is also called an Application Interceptor within OkHttp.

2.3 BridgeInterceptor and CacheInterceptor

BridageInterceptor provides the following functions:

  1. It is responsible for transforming user-constructed requests into requests sent to the server and the responses returned from the server into user-friendly responses, and is the bridge from application code to network code
  2. Set content length, content encoding
  3. Set up GZIP compression and decompress the content when it is received. Save the trouble of data decompression processing application layer
  4. Add a cookie
  5. Set other headers, such as user-agent,Host, keep-alive, etc. Keep-alive is a necessary step to realize connection reuse

The logical flow of the CacheInterceptor interceptor is as follows:

  1. Request to Cache, if caching is configured in OkHttpClient, it is not supported by default.

  2. Create a cache policy based on response,time, and request to determine how to use the cache.

  3. If network use is forbidden in the cache policy and the cache is empty, a Response is constructed and returned directly. Note that the return code is =504

  4. Cache policy Settings do not use network, but cache, directly return cache

  5. Proceed to the next filter, chain-.proceed (networkRequest)

  6. If the network returns a Resposne of 304 while the cache exists, the cached Resposne is used.

  7. Build the Resposne for the network request

  8. When caching is configured in OkHttpClient, the Resposne is cached.

  9. The caching step is to cache the header first and then the body.

  10. Returns the Resposne

The next two are probably the most important of all internal interceptors. One handles Dns and Socket connections, and the other sends the Http request body.

2.4 ConnectInterceptor

As mentioned above, The connectInterceptor is one of the most important interceptors, it is responsible for both Dns resolution and Socket connections (including TLS connections).

Source address: github.com/square/OkHt…

The class itself is very simple. From the source, there is only one key piece of code.

val exchange = transmitter.newExchange(chain, doExtensiveHealthChecks)
Copy the code

A new ExChange object was obtained from the Transmitter. This simple code contains a lot of logic, which involves the whole process of network connection establishment, including DNS process and socket connection process. Here we use two pictures to understand the whole process of network connection.

Take a look at the method call sequence diagram to tease out the key steps:

  1. ConnectInterceptor call transmitter. NewExchange
  2. The Transmitter first calls ExchangeFinder’s find() to get an exchange dec
  3. ExchangeFinder calls its own findHealthConnectio to get a RealConnection
  4. ExchangeFinder gets an ExchangeCodec through the codec() method of the RealConnection you just obtained
  5. The Transmitter gets the ExChange Ecodec and then new an ExChange, including the ExChange ecodec it just had

Through the previous 5 steps, Connectinterceptor finally obtains an Exchange class through the Transmitter. This class has two implementations, one is Http1ExchangeCodec, the other is Http2Exchangecodec, and the other is Http1ExchangeCodec. These protocols are Http1 and Http2 respectively.

So what’s the use of having an Exchange class? Look at the diagram for these classes as follows:

As you can see, the Exchange class you obtained earlier contains an ExchangeCodec object, which in turn contains a RealConnection object, RealConnection’s property members include socket, handlShake, protocol, etc. It should be a wrapper class for socket connections. An ExchangeCode object is the encapsulation of a RealConnection operation (writeRequestHeader, readResposneHeader).

From these two figures, it can be clearly known that the final obtained is a Socket object that has established a connection, that is, the Socket connection has been completed inside the ConnectInterceptor, so which step is completed?

Looking at the sequence diagram above, you can see that the findHealthConnection() method called by the ExchangeFinder getting the RealConnection, so the socket connection is both acquired and established here.

Similarly, before socket connection, there is actually a DNS process, which is also the internal logic hidden in findHealthConnection. The detailed process will be analyzed later in the DNS process. Here, the task of ConnectionInterceptor has been completed.

In addition, it is important to note that after the execution of ConnectInterceptor, in fact, the added custom networkInterceptors networkInterceptors, according to the order of execution, all networkInterceptors execute. The socket connection is actually established, and you can get the socket through realChain to do some things, which is why it is called network Interceptor.

2.5 CallServerInterceptor

The CalllServerInterceptor is the last interceptor. The previous interceptor has completed the socket connection and the TLS connection, so this step is to transfer the HTTP header and body data.

CallServerInterceptor source: github.com/square/OkHt…

The CallServerInterceptor consists of the following steps:

  1. The request header is sent to the server
  2. If there is a request body, send it to the server
  3. To read the Response header, construct a Response object
  4. If there is a Response body, we construct a new response object by adding the body to 3

As you can see here, the core work is done by the HttpCodec object, which actually uses Okio, which actually uses sockets, but one layer on top of another.

3. Overall structure

At this point, all interceptors are covered, and we know how a complete request flow occurs. At this point it will be clearer to look at the OkHttp architecture diagram

The overall OkHttp architecture consists of five internal interceptors vertically, which are divided into several parts horizontally. The vertical interceptors complete the entire request process by invoking horizontal layers. It is more comprehensive to understand OkHttp from these two aspects.

The horizontal sections will be analyzed in detail in the following sections.

3.1 Connection multiplexing, DNS and Socket connections

From the previous analysis, we know that both Socket connection and Dns process are completed by Transmitter and ExchangeFinder in ConnecInterceptor. As can be seen in the previous sequence diagram, The final way to establish the Socket connection is through the findConnection method of ExchangeFinder, so to say all the secrets are in the findConnection method.

So let’s take a closer look at findConnection(), where the source code and comments are posted.

Synchronized (connectionPool) {synchronized(connectionPool) {// delete.....if(result == null) {// 1, the first attempt to get a RealConnection from the buffer poolif (connectionPool.transmitterAcquirePooledConnection(address, transmitter, null, false)) {
          foundPooledConnection = true
          result = transmitter.connection
        } else if(nextRouteToTry ! SelectedRoute = nextRouteToTry nextRouteToTry = null} selectedRoute = nextRouteToTry nextRouteToTry = null}else if(retryCurrentRoute()) {//3, if this Route has been set to retry with the current Route, the Route selectedRoute = Transmitter will be used as the current Route selectedRoute = Transmitter. Connection!! .route() } } }if(result ! = null) {// 4. If there is any Connection that has transmiter or ConnectionPool that has transmiter that can be reused if there is any Connection that has transmiter that has transmiter, then it is returnedreturnresult!! } var newRouteSelection = {}}} // 5. If no Connection has been obtained before, you need to obtain a newRoute via routeSelectorfalse
    if(selectedRoute == null && (routeSelection == null || ! routeSelection!! .hasNext())) { newRouteSelection =trueRouteSelection = routeSelector. Next ()} var routes: List< route >? = null synchronized(connectionPool) {if (newRouteSelection) {
        // Now that we have a setof IP addresses, make another attempt at getting a connection from // the pool. This could match due to connection coalescing. routes = routeSelection!! .routes //7. If you get a new Route from a routeSelector, you get a new batch of IP. Again, you try to check from the ConnectionPool // to see if there are any connections you can reuseif (connectionPool.transmitterAcquirePooledConnection( address, transmitter, routes, false)) {
          foundPooledConnection = true
          result = transmitter.connection
        }
      }
 if(! foundPooledConnection) {ifSelectedRoute == null) {selectedRoute == null; selectedRoute == null; SelectedRoute = routeSelection!! .next() } // Create a connection and assign it to this allocation immediately. This makes it possible //for an asynchronous cancel() to interrupt the handshake we//9: create a RealConnection with a new route. Result = RealConnection(connectionPool, selectedRoute!!) connectingConnection = result } } // If we found a pooled connection on the 2nd time around, we'Re done. // 10, the comment makes it clear that if the Connection is fetched the second time from the connectionPool, it can be returned directlyif (foundPooledConnection) {
      eventListener.connectionAcquired(call, result!!)
      returnresult!! } // Do TCP + TLS handshakes. This is a blocking operation. .connect( connectTimeout,readTimeout, writeTimeout, pingIntervalMillis, connectionRetryEnabled, call, EventListener) // The next section is to put the successful RealConnection into the ConnectionPool, which will not be posted here}return result!!
Copy the code

As you can see from the above process, the findConnection method does several things:

  1. Check that the Connection currently saved in exchangeFinder satisfies the request
  2. Check whether the requested Connection exists in the current ConnectionPool
  3. Check the current list of RoutesElectors to see if any routes are available (routes are proxies, wrapper classes for IP addresses). If not, make a DNS request
  4. After the new Route is obtained through DNS, a second search is made from the ConnectionPool for a non-reusable Connection, otherwise a new RealConnection is created
  5. Use RealConnection for TCP and TLS connections, and save to ConnectionPool after successful connection

Connection multiplexing

As you can see, steps 2 and 4 reuse the ConnectionPool twice, and step 5 creates a new RealConnection and writes it to the ConnectionPool.

So, OkHttp connection reuse is actually implemented through a ConnectionPool. As reflected in the previous class diagram, the ConnectionPool contains a connections ArrayDeque object that is used to hold the cached ConnectionPool.

DNS process

As you can see from the previous parsing steps, the Dns process is hidden in the third RouteSelector check. The whole process is written in the findConnection method, and it may not be very easy to understand, but once you understand the RouteSelector, RouteSelection, In fact, it is easier to understand the relationship between the three classes: Route. The following figure shows the relationship between the three classes.

The following conclusions can be drawn from the figure:

  • RouteSelector calls Next to walk through different proxies to get the next Selection wrapper class. Selection holds a list of routes, that is, each proxy has a list of routes
  • Selection is an iterator that encapsulates the List and uses the next() method to obtain the next Route. The Route contains proxy, address, and inetAddress. It can be thought that the Route is an encapsulation of IP and proxy pairing
  • RouteSelector next () method calls the internal nextProxy (), nextProxy () and invokes the resetNextInetSocketAddres () method
  • ResetNextInetSocketAddres by address. DNS. Get InetSocketAddress lookup, also is the IP address

From the above process, we know that the IP address is finally obtained through the DNS of address, and how is this DNS built?

DNS is passed in as the built-in client. DNS is passed in as the client. DNS is passed in as the client. In the lookup is through the inetAddress. getAllByName method to obtain the IP address of the corresponding domain name, that is, the default Dns implementation.

At this point, the whole DNS process is clear. OkHttp is designed in this area to emphasize coupling and openness, and the entire process of DNS is hidden from view, which might not be easily discovered without careful debugging and code.

3.2 Establishing Socket Connections

After obtaining a Connectoin from Dns, the process of establishing a connection is as follows:

result!! .connect( connectTimeout,readTimeout,
        writeTimeout,
        pingIntervalMillis,
        connectionRetryEnabled,
        call,
        eventListener
)
Copy the code

In this case, the result is an object of type RealConnection, which means that the realConnection.connect method is called, finally leaving findConnection. Let’s look at the source code for the connect method.

// Omit the preceding paragraph....while (true) {
      try {
        if(rout.requirestunnel ()) {// The entry condition is that HTTPS requests are broked through HTTP, and there is a special protocol exchange process connectTunnel(connectTimeout,readTimeout, writeTimeout, call, eventListener)
        } else{// set up a socket connectionreadTimeout, call, eventListener)} So here is the TLS setup protocol for HTTPS: Connect SpecSelector, pingIntervalMillis, Call, eventListenerbreak} catch (e: IOException) {// Clean up resource socket? .closequietly () // Double wrap the exception and throw itif (routeException == null) {
          routeException = RouteException(e)
        } else {
          routeException.addConnectException(e)
        }
        if(! connectionRetryEnabled || ! connectionSpecSelector.connectionFailed(e)) { throw routeException } } }Copy the code

The connect method is not complicated. It first determines whether there is a proxy, and then calls the system method to establish the socket connection.

If it is an HTTPS request, there is a TLS connection to be established, and if an exception is thrown, it will be encapsulated and thrown.

4. To summarize

So far, basically OkHttp source code design has a full picture, there are some content because of daily use often encountered, such as OkHttpClient Builde parameters, such as Request and Response usage, here no longer talk about. In addition, support for HTTP2 and HTTPS, which involve more complex mechanisms and principles, will be discussed in another article in the future.