Network has always been an important module in the project, and many excellent network frameworks have appeared on the Android open source project. From the beginning is just some of the HttpClient and HttpUrlConnection simple encapsulation used tool classes, to later Google open source more complete and rich Volley, and then to today’s more popular Okhttp, Retrofit. There are similarities and differences between them. This series mainly tries to clarify their relationship and principles by analyzing the basic knowledge of the network and the Android network framework.

The whole series is divided into the following parts:

  1. Some of the basics of computer networks
  2. HttpUrlConnection and Volley parsing
  3. Okhttp parsing (more, will be divided interceptor parsing)
  4. Retrofit parsing
  5. conclusion

This is the first, mainly introduces some of the foundation of computer network, as well as some of the use of Android development and encountered problems and solutions, this article is mainly divided into the following parts:

  1. Computer network architecture
  2. Http related
  3. Tcp related
  4. Socket
  5. conclusion

For some basic knowledge of computer network, or need to do some understanding, it has been proved that in the daily development of Android and source code reading will often encounter relevant knowledge, so the first article to write some of the foundation of computer network.

1. Computer network architecture

This is often seen as the hierarchical structure of computer networks, and it is necessary to clarify this to prevent the tangle of Http and Tcp protocols, which are not in the same layer. Depending on the reference model, there are several versions of the hierarchical structure, such as the OSI model and the TCP/IP model. Here is an example of the more commonly seen 5-tier structure:

layered
The application layer (HTTP, FTP, DNS, SMTP, etc.)
The transport layer (TCP, UDP)
The network layer (IPEtc.)
Data link Layer (ARP etc.)
The physical layer

The five-tier architecture from the top to the bottom can finally achieve end-to-end data transmission and communication. What are they responsible for and how to achieve end-to-end communication?

1. Application layer: for example, HTTP protocol, it actually defines how to package and parse data. If the application layer is HTTP protocol, it will package data according to the protocol provisions, such as the request line, request header, request body, and then the data will be transmitted to the transport layer after the data is wrapped.

2. Transportation layer: There are two protocols in the transportation layer, TCP and UDP, which respectively correspond to reliable transportation and unreliable transportation. For example, because TCP needs to provide reliable transportation, internal solutions should be solved on how to establish connections, how to ensure reliable transmission without data loss, and how to adjust flow control and congestion control. On this layer, we usually deal with sockets, sockets are a group of programming call interface encapsulation, through it, we can operate TCP, UDP connection establishment and so on. This layer specifies the port number to send data to when we use sockets to establish connections.

3. Network layer: This layer of THE IP protocol, and some routing protocols, etc., so this layer of the data to be transmitted to the IP address. There are some optimal routes, routing algorithms and so on.

4. Data link layer: The deepest impression is the ARP protocol, which is responsible for resolving IP addresses into MAC addresses, that is, hardware addresses, so as to find the corresponding unique machine.

5. Physical layer: This layer is the bottom layer and provides the binary streaming service, which is when data is actually transmitted over the transmission medium (wired or wireless).

Therefore, through the above five layers of each role, to achieve the physical transmission medium -MAC address -IP address – port number – to obtain data according to the application layer protocol analysis data and finally realize the network communication and data transmission.

The following will focus on HTTP and TCP related things, about other layers, graduated so long also forget a lot, if you want to more detailed and specific understanding of the following three layers such as routing algorithm, ARP addressing and physical layer and so on or have to re-read the computer Network Principles ~

2. HTTP related

Here we mainly talk about some basic knowledge about Http, as well as some practical applications and problems encountered in Android and solutions.

One, no connection and no state

Http is connectionless and stateless.

Connectionless does not mean that no connection is required. Http is just an application layer protocol, which ultimately relies on services provided upward by the transport layer, such as TCP. Connectionless means that THE HTTP protocol processes only one request per connection and disconnects the connection after the request is completed. This is mainly to relieve the pressure on the server and reduce the occupation of server resources by connection. My understanding is that establishing connections is really a matter for the transport layer, and in the case of HTTP at the application layer, it is connectionless because the upper layer is not aware of the lower layer.

Stateless refers to the ability for each request to be independent of each other, with no memory of the previous request transaction. So there are things like cookies that are used to store some state.

Request packet and response packet

Here is a brief description of the format of the request packet and response packet:

Request message:

The name of the composition
The request line Request method post/ GET, request path Url, and protocol version
Request header It’s called a header, and it has a bunch of fields in it
Request body Sent data

Response message:

The name of the composition
The status line The status code is 200, protocol version, and so on
Response headers That’s the header that’s returned
Response body Response body data

On Get and Post: The differences between Get and Post that we are all familiar with are as follows:

  1. Get concatenates the request parameters to the URL and ends up in the address bar, while Post puts the request parameter data in the body of the request and does not show up in the address bar
  2. The length limit of the passed parameter

Question:

For the first point, it is not appropriate to expose private data to the address bar in the browser, but in the App development, there is no concept of the address bar, so this is still a constraint to choose post or get.

For the second point, the length limit should be a browser limit, regardless of the get itself, and should be ignored in App development.

Second, HTTP caching mechanism

The reason why I want to introduce the following Http caching mechanism is that Okhttp makes use of Http caching mechanism for network requests, rather than having the client write its own caching strategy like Volley and other frameworks.

Http caching is controlled by two fields in the header:

  1. Cache-controlIt mainly contains and several fields:
  • Private: Only the client can cache
  • Public: Both the client and proxy server can cache
  • Max-age: indicates the expiration time of the cache
  • No-cache: RequiredCompared to the cacheTo verify the cached data
  • No-store: No memory will be cached

This is where the cache policy is set, and the server sends it to the client the first time through the header.

Max-age indicates the expiration time of the cache. The request is made again later. If the expiration time is not exceeded, the cache can be used directly.

No – cache: it means you need to use the contrast cache to verify the cached data, if this field is open, even if Max – age cache without failure, is still need to initiate a request to the server confirm resource if there’s any update, whether to need to request data, as to how to do comparative cache, is this the Etag it. If the server confirms that the resource has not been updated, then 304 is returned, fetching the local cache. If the resource has been updated, then the latest resource is returned.

No-store: If this field is turned on, no caching will take place and no cache will be fetched.

2.ETag: Used for comparison cache. The ETag is an identifier of the server resource

When the client sends the first request, the server sends the Etag of the current resource. When the client sends the next request, the client carries the Etag with the if-none-match in the header. The server compares the Etag sent by the client with the latest resource Etag. The resource is not updated, and 304 is returned.

The Http caching mechanism is realized by the combination of cache-control and Etag.

Three, Cookie,

Http is stateless, and cookies are used to remember states in local caches. A Cookie usually contains domin, path, and Expires attributes. The server can write the state to the client’s Cookie by using set-cookies in the response header. The next time the client initiates a request, the Cookie can be included in the request.

Problems encountered in Android development and solutions:

Speaking of cookies, generally if you are only doing App development, it is not often encountered, but if it is related to the needs of WebView, it is possible to encounter, here is a worrying story about WebView cookies I have encountered in the project: The requirements are as follows: the H5 page in the loaded WebView should be logged in, so we need to manually write the ticket into the Cookie of the WebView after logging in to the native page, and then the H5 page loaded in the WebView should be verified by the server with the ticket in the Cookie. But there was a problem: If the WebView is closed by default, you can open it by setting the following code:

CookieManager cookieManager = CookieManager.getInstance();
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.LOLLIPOP) {
            cookieManager.setAcceptThirdPartyCookies(mWebView, true);
        } else {
            cookieManager.setAcceptCookie(true);
        }
Copy the code

Four, Https

We all know that Https guarantees the security of our data transmission, Https=Http+Ssl, the main principle of security is the use of asymmetric encryption algorithm, the common symmetric encryption algorithm is not secure, because both parties use the same key for encryption and decryption, as long as either party leaks the key, Then someone else can use the key to decrypt the data.

The core essence of asymmetric encryption algorithm is that the information encrypted by public key can only be unlocked by private key, and the information encrypted by private key can only be unlocked by public key

1. Describe why asymmetric encryption algorithm is safe:

Server application certificate issued by the CA authorities, to obtain the certificate of public and private keys, only the server knows the private key, and the public key can inform others, if can pass the public key to the client, so that the client by the server from the public key to encrypt their data transmission, and the server using the private key can decrypt the data. Since the data encrypted by the public key on the client can be decrypted only by the private key, which only the server has, the data transmission is secure.

The above is just a brief description of how asymmetric encryption algorithms ensure data security. In fact, the Https process is much more complex (space is limited, there are many articles on the Internet) :

  • For one thing, the client also needs to verify the validity and validity of the CA certificate sent by the server, because there is a risk of the CA certificate being transferred during the transmission process, which involves how the client verifies the validity of the server certificate to ensure the legitimate identities of the communication parties.

  • Another asymmetric algorithm although to ensure the security of the data, but the efficiency is relatively poor compared with the symmetric algorithm, how to optimize, to achieve both to ensure the security of the data, and improve the efficiency? HTTPS transport process in general are from the server certificate after the first time, the client obtain the public key, then the client will generate a symmetric key, then use the symmetric key, public-key encryption is used after the server receives the private key to decrypt the access to the symmetric key, then the server using the symmetric key to encrypt the follow-up data.

2. How does the client verify the validity of the certificate?

Firstly, a CA certificate usually contains the following contents:

  • The authority and version of the certificate
  • User of the certificate
  • Public key of certificate
  • Certificate validity
  • The digital signature Hash value of the certificate and the signature Hash algorithm (the digital signature Hash value is encrypted with the certificate’s private key)
  • , etc.

Client authentication server to get the certificate of legitimacy through: first use of access to the public key to decrypt the digital signature certificate Hash value 1 (because it is the use of private key encryption), and then use the signature in the certificate of the Hash algorithm to generate a Hash value 2, if the two values are equal, the said certificate legal, the server can be trusted.

Since HTTPS has a secure data encryption mechanism, how does the packet capture tool (such as Charles) that we usually use to capture HTTPS packets work?

In fact, Charles acts as an “intermediary agent”. When the server issues the certificate, Charles will intercept the certificate and obtain the public key, and then send Charles’s own certificate to the client. After that, the client will naturally get Charles’s certificate. Then Charles intercepts the request and uses his private key to decrypt the request data. Then he encrypts the data with the public key of the server and forwards the data to the server. The process of the server returning the packet is similar. Charles switches the certificate and acts as a middleman to decrypt and retrieve the HTTPS message. Of course, the most important thing is that we need to install and trust Charles certificates on our phones.

Problems encountered in Android development and solutions:

By an Android WebView loading is used during the project development company web pages on the server certificate expired results in page load out hang problem: the solution is temporarily ignore SSL error test environment, so that you can take out page load and, of course, don’t do it on the production, is a can have security issues, One is that Google Play should not be approved. Override WebViewClient’s onReceivedSslError():

@Override
    public void onReceivedSslError(WebView view, SslErrorHandler handler, SslError error) {
        
        if (ContextHolder.sDebug) {
            handler.proceed();
            return;
        }
       
        super.onReceivedSslError(view, handler, error);

    }
Copy the code

Fifth, the Http 2.0

Okhttp supports configuration using the Http2.0 protocol, which is a significant improvement over http1.x in the following aspects:

  • Binary format:X is a text protocol, and HTTP2.0 is a binary protocol with frame as the basic unit. A frame contains not only data but also the Identifier of the frame: Stream Identifier, which identifies which request the frame belongs to, making the network transmission very flexible.
  • Multiplexing:This is a big improvement over the old http1.x one connection one request situation, which was quite limited and caused a lot of problems, such as the cost of setting up multiple connections and efficiency issues.

In order to solve the efficiency problem, http1.x may initiate as many concurrent requests as possible to load resources. However, browsers have restrictions on concurrent requests under the same domain name, and the means of optimization is generally to request resources under different domain names to break through this limitation.

And http2.0 support multiplexing can be a good solution to this problem, multiple requests share a TCP connection, multiple requests can concurrently on this TCP connection, one is to solve the problem of establishing multiple TCP connection consumption, one also solves the problem of efficiency. So what is the mechanism that allows multiple requests to be concurrent on a TCP connection? The basic principle is the above binary frame division. Because each frame has an identity, different frames of multiple requests can be concurrently sent out without order. The server will organize each frame into the corresponding request according to the identity of each frame.

  • Header header compression:The header is compressed to reduce the size of requests, reduce traffic consumption, and improve efficiency. Because one of the problems was that every request had a header, and the data in that header was usually one layer unchanged.
  • Server push is supported

3. The TCP

TCP is connection-oriented and provides reliable data transmission. At this level, we usually use the Socket Api to manipulate TCP, establish connections, and so on.

One or three handshakes to establish a connection

First time: Send SNY=1 to indicate that the handshake is a request to establish a connection, then seq generates a random number X for the client

SNY=1,ACK=1, seq+1, seq=Y, seq=Y, seq=Y, seq=Y

3: ACK=1. Seq = client random number +1, ack= server random number +1

Why is a three-way handshake required to establish a connection?

Very clear above all is two shake hands is the most basic, shaking hands for the first time, the C terminal sends a connection request message to the S, S end S after receiving end can know oneself with the C is the connection is successful, but C end right now does not know whether the S end receives the message, so I have to reply S end after the message is received, the c-terminal get S after the reply, To make sure that you can connect to the S end, this is the second handshake. C can’t start sending data until it knows it can connect to S. So two handshakes must be basic.

So why the third handshake? Let’s say we don’t have a third handshake, we have two handshakes and we think the connection is made, what happens?

The third handshake is to prevent an invalid connection request segment from being suddenly retransmitted to the server, resulting in an error

Specific situation is: the c-terminal sent the first network connection request for some reason stranded in the network node, lead to delay, until the connection release at some point in time to reach S side, this is a failure message already, but this time the s-video still think this is the first time C side to establish connections to shake hands, then responded to the c-terminal S end, a second hand. If there are only two handshakes, then at this point, the connection is established, but the C end has no data to send, and the S end will wait stupidly, resulting in a huge waste of resources. So you need to do a third handshake, and only if the C side responds again, can you avoid that.

The second and fourth handshakes disconnect

After the above analysis of the connection diagram, this diagram should not be difficult to understand, there is a major problem:

Why is there one more handshake than the connection?

As you can see, the server’s ACK(reply to client) and FIN(terminate) messages are not sent at the same time. Instead, the server ACK first and then the FIN. This makes sense because when the client requests to disconnect, the server may still have data to send. So that’s four handshakes.

The main feature of TCP is to provide reliable transmission, so how to ensure that the data transmission is reliable, this is the next to talk about the sliding window protocol

Sliding window protocol

The sliding window protocol is essential to ensure reliable transmission of TCP, because the sending window only moves back to send another frame when it receives an acknowledgement frame.

Here’s an example: Suppose the send window is 3 frames

A began to send window frame in the first 3 [1, 2, 3], the first three frames can be sent later, temporarily can not send, such as [1] frame after send out, received a confirmation message from the receiver, then send window can move back 1 frame, at this time send window to [4] 2, only send the window frame can also be sent, One analogy.

The receiving window will put the frame into the corresponding position after receiving it, and then move the receiving window. The interface window has the same size as the sending window. If the receiving window is 5 frames, the frames outside the receiving window will be discarded.

The different sizes of the send and receive Windows extend to different protocols:

agreement The characteristics of
Stop-wait protocol Send window size =1, receive window size =1
Back N frame protocol Send window size >1, receive window size =1,
Select a retransmission protocol Send window size >1, receive window size >1

Stop-wait protocol: each frame must wait for an acknowledgement message before sending the next frame. Disadvantages: poor efficiency.

Back N frames protocol: the accumulative acknowledgement is adopted. After receiving N frames correctly, the receiver sends an accumulative acknowledgement message to the sending window to confirm that N frames have been correctly received. If the sender does not receive the acknowledgement message within the specified time, it considers timeout or data loss, and resends all frames after the acknowledgement frame. Disadvantages: The PDU following the error number has been sent, but it still needs to be sent again, which is wasteful.

Select the retransmission protocol: If an error occurs, only the required PDUs are retransmitted. This improves transmission efficiency and reduces unnecessary retransmission.

That leaves one last problem: TCP has a set of traffic control and congestion control mechanisms to solve congestion caused by a mismatch between the sending window and the receiving window.

Iv. Flow control and congestion control

1. Flow control

Flow control is to control the traffic on a communication path. In this way, the sender dynamically adjusts the transmission rate by obtaining feedback from the receiver to control the traffic. The purpose is to ensure that the transmission speed of the sender does not exceed that of the receiver.

2. Congestion control

Congestion control is to control the traffic of the entire communication subnet, which belongs to global control.

① Slow Start + Congestion Avoid First look at a classic picture:

Using a slow start at the beginning, the congestion window is set to 1, then the congestion window exponential growth to slow start threshold (ssthresh = 16), the switch for congestion avoidance, namely additive increase, such growth to a certain extent, lead to network congestion, at this time will reduce the congestion window to drop to 1, namely to slow start, at the same time adjust to 12 new slow start threshold method, And so on.

② Fast retransmission + fast recovery

Fast retransmission: The retransmission mechanism mentioned above is to wait until the timeout and no reply from the receiver. The design idea of fast retransmission is as follows: If the sender receives three repeated ACK messages from the receiver, it can determine that a packet segment is lost. In this case, the sender can immediately retransmit the lost packet segment without waiting until the timeout period expires, which improves the retransmission efficiency.

Fast recovery: The above congestion control will reduce the congestion window to 1 when the network is congested, and slow start again. One problem with this is that the network does not recover quickly. Fast recovery is to optimize this problem. If fast recovery is used, when congestion occurs, the congestion window will only decrease to the new slow start threshold (that is, 12), but not to 1, and then directly start into congestion to avoid the addition growth, as shown in the figure below:

Fast retransmission and fast recovery are further improvements to congestion control.

4.Socket

A Socket is a set of apis that operate on TCP/UDP. HttpURLConnection and Okhttp are all low-level network requests that are ultimately sent through sockets. Volley and Retrofit are upper-level encapsulation. Finally, it relies on HttpURLConnection or Okhttp to make the final connection and send the request.

The simple use of Socket should be all, two end each establish a Socket, server end is called ServerSocket, and then establish a connection.

5. To summarize

Of course, these are only the basic computer network knowledge I know and think is very important, there are a lot of network basic knowledge to understand and explore. Wrote a lot of, be a sort out to oneself network foundation, the likelihood also can have flaw. The following will also be a simple parsing of HttpUrlConnection and Volley, source code parsing Okhttp and Retrofit.