preface

Learning resources from Geek Time – Teacher Li Bing “Browser working principle and Practice”. Next, let’s check in every day

  • Day 01 Chrome architecture: Why 4 processes with only 1 page open?
  • Day 02 TCP: How to ensure that a page file can be delivered to the browser in its entirety?
  • Day 03 HTTP Request Flow: Why do many sites open quickly the second time?

What will you learn from reading this article?

An HTTP request in a browser goes through the following eight stages:

  • Build request
  • Find the cache
  • The IP address and port are ready
  • Waiting for TCP queue
  • Establishing a TCP Connection
  • Making an HTTP request
  • The server processes the request
  • The server returns the request
  • disconnect

There are two common problems with requesting paths over HTTP

  1. The Cache flow is involved
  2. How do you use cookies for state management

As you may recall from the previous article, TCP ensures complete data transmission. A TCP connection consists of three stages: establishing a connection, transferring data, and disconnecting the connection.

The HTTP protocol is based on TCP connections. HTTP is a protocol that allows the browser to obtain resources from the server. It is the basis of the Web. Usually, the browser initiates requests to obtain different types of files, such as HTML files, CSS files, JavaScript files, images, and videos. In addition, HTTP is also the most widely used protocol in browsers, so it’s important to understand HTTP in depth to learn about browsers.

I don’t know if you have the following questions:

  1. Why is it that a site is usually slow to open the first time you visit it, and fast the second time you visit it?
  2. When you log in to a site, the next time you visit the site, you are already logged in. How does this work?

All of this is hidden in the HTTP request process. So, in today’s article, I’ll take you through the entire HTTP request process by analyzing the state of each step in the HTTP request process. Hopefully, you’ll have a new understanding of the HTTP protocol after reading this article.

The browser initiates an HTTP request process

If you type in your browser’s address bar geek time website address: time.geekbang.org/index.html, then, the browser will finish what action? Let’s follow up step by step.

1. Build the request

First, the browser builds the request line information (as shown below), and once it’s built, the browser is ready to make the web request.

GET /index.html HTTP11.
Copy the code

2. Search the cache

Before actually making a web request, the browser looks in the browser cache to see if there is a file to request. Among them, browser caching is a technique for saving a copy of a resource locally for immediate use on the next request.

When the browser discovers that the requested resource already has a copy in the browser cache, it intercepts the request, returns a copy of the resource, and ends the request without going to the source server for a new download. The benefits of this are:

  • Reduces server stress and improves performance (resource acquisition time is shorter).
  • For web sites, caching is an important part of fast resource loading.

Of course, if the cache lookup fails, it enters the network request process.

3. Prepare IP addresses and ports

Before we look at network requests, however, we need to look at HTTP and TCP. Because the browser uses HTTP protocol as the application layer protocol, used to encapsulate the requested text information; It uses TCP/IP as the transport layer protocol to send it to the network, so the browser needs to establish a connection with the server over TCP before HTTP work can begin. This means that HTTP content is implemented through the TCP data transfer phase. You can better understand the relationship between the two by combining the following diagram.

Here’s a list of questions you can think about:

  • What is the first step in an HTTP web request? As shown in the figure above, the TCP connection is established with the server.
  • Do we have the information to establish a connection? In the last article, we learned that the first step in establishing a TCP connection is to prepare the IP address and port number.
  • How do I get the IP address and port number? It depends on what we have, we have a URL, can we use the URL to get IP and port information?

In the previous article, we explained that packets are sent to the recipient via IP addresses. Because IP addresses are numeric identifying-geekTime’s IP is 39.106.233.176, for example, which is hard to remember, but much easier to remember using geekTime’s domain name (time.geekbang.org), a service has emerged based on this need, Mapping domain names to IP addresses. The System that maps Domain names to IP addresses is called the Domain Name System (DNS).

So, along the way, you’ll see that in the first step the browser will ask DNS to return the IP of the domain name. Of course, the browser also provides DNS data caching service. If a domain name has been resolved, the browser will cache the result for the next query, which also reduces the network request.

Once you have the IP, the next step is to get the port number. In general, HTTP defaults to port 80 if the URL does not specify a port number.

4. Wait for the TCP queue

Now that you have the port and IP address ready, can you set up a TCP connection?

Again, the answer is “no”. Chrome has a mechanism that allows you to establish a maximum of six TCP connections under the same domain name. If 10 requests occur at the same time under the same domain name, four of the requests will be queued until the ongoing requests are completed.

Of course, if the number of current requests is less than 6, the next step is to establish a TCP connection.

5. Establish the TCP connection

When the queue is over, you can finally happily shake hands with the server, with the browser establishing a connection via TCP before the HTTP work can begin. The way TCP works is described in detail in my last article, so you can review it for yourself if you need to. I won’t repeat it here.

6. Send an HTTP request

Once the TCP connection is established, the browser can communicate with the server. The data in HTTP is transferred during this communication.

You can see how the browser sends requests to the server.

First, the browser sends the request line to the server, which contains the request method, Uniform Resource Identifier (URI), and HTTP version protocol.

Sending a request line tells the server what resources the browser needs. The most common request method is Get. For example, typing the geek Time domain name (time.geekbang.org) directly into the browser address bar tells the server to Get its home page resources.

Another common request method is POST, which is used to send some data to the server. For example, when you log in to a website, you need to send the user information to the server through POST. If the POST method is used, the browser also prepares data for the server, which is sent in the request body.

After the browser sends a request line command, it sends additional information in the form of a request header that tells the server the basics of the browser. For example, it contains the operating system used by the browser, the browser kernel and other information, as well as the domain name information of the current request, Cookie information of the browser, and so on.

The server side handles the HTTP request flow

After a long time, the HTTP request was finally delivered to the server. The server then prepares the content based on the browser’s request information.

1. Return the request

Once the server has finished processing, the data can be returned to the browser. Curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl

curl -i  https://time.geekbang.org/
Copy the code

Note that the -i is used to return the data for the response row, header, and body, as shown below. You can use this data to understand how the server is responding to the browser.

First the server returns a response line, including the protocol version and status code.

However, not all requests can be handled by the server, so what about some messages that cannot be processed or processed incorrectly? The server tells the browser what it has done with the status code on the request line, for example:

  • The most commonly used status code is 200, indicating that the processing is successful.
  • If the page is not found, a 404 is returned.

Then, just as the browser sends the request header along with the request, the server sends the response header to the browser along with the response. The response header contains information about the server itself, such as when the server generated the returned data, the returned data type (JSON, HTML, streaming media, and so on), and the Cookie that the server wants to save on the client.

After sending the response header, the server can continue sending the data in the response body, which usually contains the actual content of the HTML.

This is how the server responds to the browser.

2. Disconnect the device

Normally, once the server returns the request data to the client, it closes the TCP connection. But if the browser or server adds the following header:

Connection:Keep-Alive 
Copy the code

The TCP connection will remain open after being sent, so the browser can continue sending requests over the same TCP connection. Maintaining a TCP connection saves the time required to establish a connection for the next request and speeds up resource loading. For example, the images embedded in a Web page are all from the same Web site, and if you initialize a persistent connection, you can reuse that connection to request other resources without having to re-establish a new TCP connection.

3. The redirection

At this point, the request process seems to be coming to an end, but there is one more thing you need to know. For example, when you open Geekbang.org in your browser, you will find that the final page is www.geekbang.org.

The two urls are different because a redirection operation is involved. You can also use curl to see what geekbang.org will return.

Enter the following command on the console:

curl -I geekbang.org
Copy the code

Note that the input parameter is -i, which is different from -i. -I means that only the data of the response header and the response row need to be retrieved, but the data of the response body need not be retrieved. Finally, the returned data is as follows:

As you can see, the response line returns a status code 301, which tells the browser that I need to redirect to another url, and that url is contained in the Location field of the response header. Then the browser retrieves the address in the Location field. And use that address to navigate again, this is a complete redirection of the execution process. This explains why geekbang.org was typed in and www.geekbang.org was opened instead.

But don’t take the jump for granted. If you open 12306.cn, you will find that the site is not open. This is because 12306’s server doesn’t handle jumps, so you have to manually enter the full www.12306.cn to open the page.

Problem solving

With that said, you now understand the HTTP request flow, so let’s go back to the question posed at the beginning of this article.

1. Why do many sites open quickly the second time?

If the second page is opened quickly, the main reason is that some time-consuming data was cached during the first page loading.

So what data is cached? As you can see from the core request path above, the DNS cache and the page resource cache are two pieces of data that are cached by the browser. Among them, DNS cache is relatively simple, it is mainly in the browser local IP and domain name associated, not too much analysis here.

Let’s focus on the browser resource cache. Here’s how it works:

First, how does the server get the browser to cache data?

As you can see from the first request above, when the server returns an HTTP response header to the browser, the browser uses the cache-Control field in the response header to set whether or not to Cache the resource. Usually, we also need to set a Cache expiration time for this resource, which is set by the max-age parameter in cache-control, such as 2000 seconds.

Cache-Control:Max-age=2000
Copy the code

This means that the cached resource will be returned to the browser if it is requested again if it has not expired.

But if the cache expires, the browser continues to make web requests with HTTP headers:

If-None-Match:"4f80f-13c-3a1xb12a"
Copy the code

After receiving the request header, the server determines whether the requested resource has been updated based on the if-none-match value.

  • If there is no update, the 304 status code is returned, which is equivalent to the server telling the browser, “This cache can continue to be used and I won’t send you the data again this time.”
  • If the resource is updated, the server returns the latest resource directly to the browser.

There are a lot of details about caching, but you can refer to HTTP caching for details, which I won’t go into here.

In short, many sites are able to get second visits in seconds because they cache a lot of their resources locally. The browser cache saves time by responding to requests directly with a local copy rather than generating actual network requests. DNS data is also cached by the browser, which eliminates DNS queries.

2. How is the login status maintained?

From the introduction above, you have seen how caching works. Let’s take a look at how login status is maintained.

  • The user opens the login page, enters the user name and password in the login box, and clicks ok. Clicking the button triggers the page script to generate user login information, which is then submitted to the server by calling the POST method.
  • After receiving the message from the browser, the server queries the background to verify that the user’s login information is correct. If so, it generates a string representing the user’s identity, writes the string to the set-cookie field in the response header, as shown below, and sends the response back to the browser.
Set-Cookie: UID=3431uad;
Copy the code
  • After receiving the response header from the server, the browser parses the response header. If the response header contains a set-cookie field, the browser saves this field locally. For example, keep UID=3431uad locally.
  • When the user visits again, the browser will make an HTTP request, but before making the request, the browser will read the saved Cookie data and write the data into the Cookie field in the request header (as shown below), and then the browser will send the request to the server.
Cookie: UID=3431uad;
Copy the code
  • After receiving the HTTP request header data, the server will search for the “Cookie” field information in the request header. When it finds the information containing UID= 3431UAD, the server queries the background, determines that the user is logged in, and then generates the page data containing the user information, and sends the generated data to the browser.
  • After the browser receives the page data containing the current user, it can correctly display the status information of the user login.

Ok, this process lets you know that the state of the browser page is achieved by using cookies. The Cookie process can be referred to the following figure:

Simply put, if the response header sent by the server has a set-cookie field in it, the browser keeps the contents of that field locally. The next time a client sends a request to the server, the client automatically adds the Cookie value to the request header and then sends the request. After discovering the Cookie sent by the client, the server will check which client sent the connection request, and then compare the records on the server to obtain the status information of the user.

thinking

If the network load time of a page is too long, how do you analyze the stage of the card?

  1. The number of concurrent requests exceeds 6, and the TCP connection enters the waiting state. Procedure
  2. Large resources and long download time
  3. For repeated requests for some resources, TCP connections do not have connection=keep-alive, which consumes connection time
  4. The network is slow or the connection times out
  5. Packets are lost during network transmission and need to be retransmitted continuously