“This is the third day of my participation in the Gwen Challenge in November. Check out the details: The last Gwen Challenge in 2021.”

Hello, everyone, I am front-end LAN Feng, a second-tier city program yuan, today mainly to share with you my notes 2021 front end test series: HTTP request and HTTP cache control, this aspect of the content is often used in our work, but also the interviewer often ask questions, I hope the following article is helpful to you.

1. A complete HTTP service process

Problem analysis

What happens when we type www.xxxx.com into the address bar of our Web browser?

  1. rightwww.xxxx.comThis url performs DNS domain name resolution to obtain the corresponding IP address
  2. Based on this IP address, find the corresponding server and initiate the TCP three-way handshake
  3. After a TCP connection is established, an HTTP request is sent
  4. The server responds to the HTTP request, and the browser gets the HTML code
  5. The browser parses the HTML code and requests resources in the HTML code (such as JS, CSS, images, etc.).
  6. The browser renders the page to the user
  7. Server Shutdown The TCP connection is disabled

Note:

  1. How does DNS find the domain name?

DNS domain name resolution is a recursive query, the process is, first go to the DNS cache -> cache can not find the root domain name server -> the root domain name will go to the next level, such a recursive search, found, to our Web browser

  1. Why is HTTP implemented based on TCP?

TCP is a reliable end-to-end connection protocol. HTTP is based on transport layer TCP and does not have to worry about various problems of data transmission (when errors occur, it will be retransmitted).

  1. As a final step, how does the browser render the page?

A) Parsing HTML files to form a DOM tree

B) Parsing CSS files to form a rendering tree c) parsing while rendering d) JS runs in a single thread. JS may modify the DOM structure, meaning that it is unnecessary to download all subsequent resources before JS execution is completed. Therefore, JS is a single thread, which will block subsequent resource downloads

Details of each step

DNS resolution (DNS server)

  1. The browser’s own DNS cache is searched first (the cache is short, about 1 minute, and can hold only 1000 caches)
  2. If it is not found in the browser’s own cache, the browser searches the system’s own DNS cache
  3. If you haven’t found it yet, try looking for it in the hosts file
  4. In the first three procedures failed to obtain the case, recursively to the domain name server to find, the detailed process is as follows

DNS optimization includes DNS cache and DNS load balancing

TCP connection establishment (three-way handshake)

After obtaining the IP address corresponding to the domain name, the User-agent (generally referred to the browser) sends a random port (1024< port <65535) to the server’s WEB application (HTTPD, nginx, etc.) port 80. Once the connection request (the original HTTP request is wrapped in the TCP/IP 4-layer model) arrives at the server (with various routing devices, except in the LAN), goes to the nic, and then to the kernel’s TCP/IP stack (used to identify the connection request, unpack the packet, layer by layer unpack). It may also pass through the Netfilter firewall (a module belonging to the kernel) and eventually reach the WEB application, which establishes the TCP/IP connection.

Making an HTTP request (after establishing a connection)

An HTTP request packet consists of three parts: the request line, the request header, and the blank line/request body

** Request line: ** describes the request method (GET/POST, etc.), the requested resource name (URL), and the VERSION of the HTTP protocol used

** Request header: ** is used to describe which host and port the client requests, and some environment information of the client, etc

** empty line: ** empty line is \r\n (present in POST request)

** Request body: ** When using methods such as POST, the client usually needs to pass data to the server. This data is stored in the body of the request (GET is stored after the URL, not here).

For example:

A GET request

The following is the browser to http://localhost:8081/test? Name =XXG&age=23 GET request sent to server:

You can see that the request consists of the request line and the first two parts of the request. The request line contains method (such as GET and POST), URI (common resource identifier), and protocol version, separated by Spaces. The request line and each request header are on one line, separated by the newline character CRLF (i.e. \r\n).

A POST request

Below is a browser for http://localhost:8081/test POST request data sent to the server, the message body in take a parameter name = XXG&age = 23

As you can see, the above request contains three parts: the request line, the request header, and the space/message body. There is one more request message than the previous GET request, where the header and the message body are separated by an empty line. The parameters of the POST request are not in the URL, but in the message body. The content-Length parameter in the request header is used to indicate the number of bytes in the message body, so that the server knows whether the request is finished. This is the main difference between a GET request and a POST request.

So what are the request methods in the start line?

GET: to request a resource in its entirety.

PUT: (webdav) Upload a file (but browsers do not support this method.) DELETE: (webdav) DELETE OPTIONS: methods that return methods supported by the requested resource TRACE: The agent that goes through a resource request (this method cannot be issued by the browser)

So what is URL, URI, URN?

URI Uniform Resource Identifier Specifies the Uniform Resource Identifier

URL Uniform Resource Locator URN Uniform Resource Name Specifies the Uniform Resource Name

Both URLS and UrNs are URIs, and for the sake of convenience both are temporarily referred to the same thing

The server responds to the HTTP request, and the browser gets the HTML code

An HTTP response also consists of three parts: a status line, a response header, a space, and a message body

Status lines include protocol version, status code, and status code description

** Status code: The ** status code is used to indicate the result of the server’s processing of the request

1XX: Indicating message – indicating that the request has been accepted and processing continues

2xx: Success: The request is successfully received, understood, or accepted. 3xx: redirection – further action must be taken to complete the request 4XX: client error – the request has a syntax error or the request cannot be implemented 5XX: server error – the server failed to implement a valid request.

Name a few common ones:

200 (No problem)

404 404 404 server does not have this resource. 500 Server has a problem.

** Response header: The ** response header describes basic information about the server and how the client handles the data

** space: **CRLF (i.e. \r\n) split

** Message body: ** Data returned by the server to the client

The response format is shown below

In the HTTP response above, content-Length in the response header is also used to indicate the number of bytes in the message body. Content-type indicates the Type of message body, usually HTML for web browsing, but also other types, such as images, videos, etc.

The browser parses the HTML code and requests resources within the HTML code

When the browser gets the HTML file, it starts to parse the HTML code in it. When it encounters static resources such as JS/CSS /image, it requests the server to download it (multi-threaded download will be used, and the number of threads in each browser is different). This is the time to use keep-Alive and establish an HTTP connection. Multiple resources can be requested and downloaded in the same order as in the code, but since each resource is of different size and the browser is a multi-threaded request for resources, the order shown here is not necessarily the order in the code.

The browser renders the page to the user

Finally, the browser makes use of its own internal working mechanism to render the static resources and HTML code requested and render it to the user. The browser is a process of parsing and rendering.

First, the browser parses the HTML file to build the DOM tree, then the CSS file to build the render tree, and once the render tree is built, the browser lays out the render tree and draws it to the screen.

The process is complicated and involves two concepts: reflow(reflow) and repain(redraw).

Each element in the DOM node exists in the form of a box model, which requires the browser to calculate its position and size, a process called relow. Once the location, size, and other properties of the box model, such as color and font, are determined, the browser begins to draw the content, a process called repain.

Pages will inevitably experience reflow and Repain when they first load.

Reflow and Repain processes can be very performance draining, especially on mobile devices, and can ruin the user experience, sometimes causing pages to stagnate. So we should reduce reflow and repain as little as possible.

JS parsing is done by the JS parsing engine in the browser.

JS runs in a single thread. JS may modify the DOM structure, which means that it is unnecessary to download all subsequent resources before the completion of JS execution. Therefore, JS is a single thread, which will block subsequent resource downloads.

The following figure shows the HTML parsing process

Server Shutdown The TCP connection is disabled

Normally, once the Web server sends the request data to the browser, it closes the TCP connection, and then if the browser or server adds this line of code to its header:

Connection:keep-alive
Copy the code

The TCP connection will remain open after it is sent, so the browser can continue sending requests over the same connection. Keeping the connection saves the time required to establish a new connection for each request and saves network bandwidth.

The above process is a complete HTTP service process.

The interview questions point

  1. Enter the URL in the address bar to start the domain name to IP process
  2. Get the IP and start setting up the HTTP request
  3. The browser rendering process after getting the HTML

Answer the train of thought

I'll start with the process from the URL to getting the HTML, and then focus on the HTML rendering process. How the interviewer responds to the next question (e.g., rearrange, redraw, TCP three times shake hands and four times wave).Copy the code

Relevant extension

  1. TCP three handshakes four waves

2. HTTP cache control

Interview Question Analysis

Web cache can be roughly divided into database cache, server-side cache (proxy server cache, CDN cache), and browser cache.

Browser caches also contain many things: HTTP caches, indexDB, cookies, localstorage, and so on. We will only discuss HTTP caching here.

Before we dive into HTTP caching, let’s clarify a few terms:

  • Cache hit ratio: The ratio of the number of requests for data from the cache to the total number of requests. Ideally, the higher the better.
  • Expired content: Content that is marked as “stale” after the set expiration date. Usually expired content cannot be used to reply to client requests and must be re-requested to the source server for new content or to verify that cached content is still ready.
  • Validation: Verifies that the expired content in the cache is still valid. If the validation passes, refresh the expiration time.
  • Invalidation: Invalidation is the removal of content from the cache. Defunct content must be removed when it changes.

Browser caching is primarily a caching mechanism defined by the HTTP protocol. HTML meta tags, for example, meaning that the browser does not cache the current page. But proxy servers do not parse HTML content, and HTTP headers are commonly used to control caching.

Browser Cache classification

The browser cache is divided into strong cache and negotiated cache. The simple process for the browser to load a page is as follows:

  1. The browser determines whether a strong cache has been hit based on the HTTP header for the resource. If hit, the resource is added directly to the cache and the request is not sent to the server. (Strong cache)
  2. If the strong cache is not hit, the browser sends a resource load request to the server. The server determines whether the browser’s local cache is invalid. If available, the server does not return the resource information and the browser continues to load the resource from the cache. (Negotiated cache)
  3. If the negotiated cache is not hit, the server returns the full resource to the browser, which loads the new resource and updates the cache. (New request)
1. Strong cache

When a strong cache is hit, the browser does not send the request to the server. In Chrome developer tools, the HTTP return code is 200, but the Size column will display as (from cache).

Strong caching is controlled using the Expires or cache-Control fields in the HTTP return header to indicate how long a resource is cached.

Expires

Cache expiration time, used to specify the expiration time of resources, is a specific point in time on the server. That is, Expires=max-age + request time needs to be used in combination with last-Modified. But as we mentioned above, cache-control has a higher priority. Expires is a Web server response header field that tells the browser in response to an HTTP request that the browser can cache data directly from the browser before the expiration date without having to request it again.

This field returns a time, such as Expires:Thu,31 Dec 2037 23:59:59 GMT. This time represents the expiration time of the resource, which is valid until 23:59:59 on December 31, 2037, i.e., hitting the cache. One obvious disadvantage of this approach is that the expiration time is an absolute time, so when the client local time is changed, the server and client time deviation becomes large, resulting in cache confusion. Cache-control was developed.

Cache-Control

Cache-control is a relative time. For example, cache-control :3600 indicates that the resource is valid for 3600 seconds. Since the time is relative and compared to the client time, server-client time deviations do not cause problems. Cache-control and Expires can be both or both enabled on the server. Cache-control takes a higher priority when both are enabled.

Cache-control can be composed of multiple fields. The values are as follows:

  1. Max-age Specifies the length of time, in seconds, during which the cache is valid. For example, cache-control :max-age=31536000, which means that the Cache Expires in (31536000/24/60 * 60) days. The first time the resource is accessed, the server also returns the Expires field with an expiration time of one year.

Revisiting the resource hits the cache without disabling the cache and without the expiration time, without asking the server for the resource and fetching it directly from the browser cache.

Revisiting the resource hits the cache without disabling the cache and without the expiration time, without asking the server for the resource and fetching it directly from the browser cache.

  1. S-maxage, like max-age, overrides max-age and Expires, but only applies to shared caches and is ignored in private caches.
  2. Public indicates that the response can be cached by any object (the client sending the request, the proxy server, and so on).
  3. Private Indicates that the response can only be cached by a single user (operating system user or browser user). The response is non-shared and cannot be cached by the proxy server.
  4. No-cache forces all users who have cached the response to send a request with a validator to the server before using the cached data. Not literally not cached.
  5. No-store disallows caching and refetches data from the server for each request.

7.must-revalidateSpecifies that if the page is expired, go to the server to fetch it. This instruction is not commonly used and will not be discussed too much.Strong cache flow chart

2. Negotiate cache

If the strong cache is not hit, the browser sends the request to the server. The server determines whether the negotiation cache is matched according to last-modify/if-modify-since or Etag/ if-none-match in the HTTP header. If a hit is made, the HTTP return code is 304 and the browser loads the resource from the cache.

Last-Modify/If-Modify-Since

When the browser requests a resource for the first time, last-modify is added to the header returned by the server. Last-modify indicates the time when the resource was Last modified, for example, last-modify: Thu,31 Dec 2037 23:59:59 GMT.

When the browser requests the resource again, the request header will contain if-modify-since, which is the last-modify value returned before the cache. After receiving if-modify-since, the server determines whether the resource matches the cache based on the last modification time.

If the cache is hit, HTTP304 is returned, the resource content is not returned, and last-modify is not returned. Because of the contrasting server time, the client-server time gap does not cause problems. However, sometimes it is not accurate to determine whether a resource has been modified or not by the last modification time (the last modification time of the resource can also be consistent). Hence ETag/ if-none-match.

ETag/If-None-Match

Unlike last-modify/if-modify-since, Etag/ if-none-match returns an entity tag (Etag). ETag guarantees that each resource is unique, and resource changes will result in ETag changes *. If the ETag value changes, the resource status has been modified. The server determines whether the cache is hit based on the if-none-match value sent by the browser.

ETag extension description

We expect ETag to generate unique values for each URL, and ETag will change as resources change. How is the mysterious Etag generated? In the case of Apache, ETag generation depends on the following factors

  1. I-node number of the file. This i-node is not the other iNode. Is the number used by Linux/Unix to identify files. Yes, the file is not identified with a file name. You can see this using the command ‘ls -i’.
  2. The time when the file was last modified
  3. The file size

When generating an Etag, one or more of these factors can be used, using a collision-proof hash function. Therefore, ETag can be repeated in theory, but the probability is negligible.

How can last-Modified produce Etag?

You might think that last-Modified is enough to let the browser know if the local cached copy is new enough, so why Etag? Etag was introduced in HTTP1.1 to solve several last-Modified problems:

  1. The last-Modified tag is accurate only to the second level. If a file has been Modified more than once in a second, it will not accurately mark the modification time of the file
  2. If some files are generated regularly, sometimes the contents are unchanged but last-Modified changes, making the file uncacheable

3. The server may not obtain the correct file modification time or the time on the proxy server may be different from that on the proxy server

Etag is the unique identifier of the corresponding resource on the server side that is automatically generated by the server or generated by the developer, which can more accurately control the cache. Last-modified and ETag can be used together. The server will verify the ETag first. If the ETag is consistent, the server will continue to compare last-Modified, and finally decide whether to return 304.

First request from the browser

The browser made the second request

The interview questions point

  1. HTTP cache scope

HTTP caching can help the server improve concurrency performance, as many resources are cached directly from the browser without repeated requests

  1. HTTP Cache classification

Strong cache negotiates cache

  1. HTTP cache implementation technology

Strong caching: Controls through Expires and cache-control Negotiated caching: controls through last-modify and e-tag

Other:

  1. Why does Expires have to be cache-control

    Because expires has a problem with server and browser time being out of sync. Expires is absolute event cache-control is relative timeCopy the code

  2. Last-modify and Etag last-modify it has an accuracy problem and there is no accuracy problem with e-tag as soon as the file changes the e-tag value

Answer the train of thought

HTTP cache can be divided into strong cache and negotiation cache. Finally, the configuration implementation of strong cache and negotiated cache and the usage of related HTTP response header fields are discussed.Copy the code

Relevant extension

  1. User behavior is related to cache browser cache behavior and user behavior!!

  1. Server side cache CDN, REDIS, database cache, etc
  2. Cache-control Nginx cache-control Nginx