preface
This article is the third in a series of HTTP articles. See more articles at juejin.cn/post/684490… It’s both a simple and complex thing, and you can explore every point as you go along. The purpose of this series of courses is to have an outline of HTTP, if you’re interested in the details. Later, if there is time, I will also write about HTTP security and HTTP interview, because for many front-end children, in the short term, what is more needed is to be able to skillfully answer the interviewer’s questions about HTTP.
The characteristics of HTTP
1. Simple, flexible and easy to expand
HTTP is a flexible and extensible transport protocol.
HTTP protocol was relatively simple when it was first born. People who came into contact with HTTP for the first time would think that HTTP protocol is “simple”. The basic packet format is “header+body”, and the header information is also a simple text format with common English words. Just by guessing, you can guess “close to ten”. The simplicity of the features lowers the barriers to learning and using HTTP, enabling more people to learn and use HTTP. Second, simplicity implies the possibility of evolution and expansion. It is much easier to make a simple system complex than to make a complex system simple.
Under the basic design concept of “simplicity”, with the development of the Internet, HTTP protocol has more advantages of “flexibility and easy to expand”. In this process, HTTP protocol gradually added request method, version number, status code, header fields and other features. Body is no longer limited to TXT or HTML in the form of text, but can transmit pictures, audio and video data.
Those RFC documents can actually be understood as “recognition and standardization” of existing extensions, realizing a virtuous circle of “from practice to practice”.
It is precisely because of this feature that HTTP can “stand tall” in the long history of thirty years and always maintain a strong vitality.
2. Reliable transmission
HTTP is a “reliable” transport protocol.
Because HTTP is based on TCP/IP, and TCP itself is a reliable transport protocol, HTTP naturally inherits this feature to transfer data “reliably” between requestor and responder.
However, we must understand the meaning of “reliable” properly. HTTP is not 100% guarantee that data will be sent to the other end, and it can also fail when the network is busy, connection quality is poor, and so on. “Reliable” simply provides a “promise” to the user that the data will be delivered “to the best of their ability”.
3. Application layer protocol
HTTP is an application-layer protocol.
In the decades after the birth of TCP/IP, there have been many application layer protocols, but they only focus on a small area of application, limited to a few application scenarios. For example, FTP can only transfer files and SSH can only log in remotely. In general data transmission, FTP cannot be called at all.
HTTP with the packet structure that can carry any header field and entity data, as well as connection control, cache proxy and other easy-to-use features, the emergence of the “dominant”.
To paraphrase a popular joke on the Web, HTTP could easily be said jokingly: “Don’t get me wrong, I’m not talking about FTP, I’m talking about the application layer in this room, they’re all rubbish.”
4. Wide application and mature environment
Another advantage of HTTP protocol is that it is “widely used” and the software and hardware environments are very mature.
With the popularity of the Internet, especially the mobile Internet, HTTP has extended its tentacles to every corner of the world: From simple Web pages to complex JSON and XML data, from the browser on the desktop to various apps on the mobile phone, from reading news, visiting forums to shopping, financial management, “eating chicken”, you can hardly find a place that does not use HTTP.
HTTP is widely used behind a lot of hardware infrastructure support, various Internet companies and traditional industry companies are sparing no effort to develop, buy servers to open websites, build data centers, CDN and high-speed fiber, continue to optimize the Internet experience, let HTTP run more and more smoothly.
5. Request-reply mode
The HTTP protocol uses the request-response communication mode.
This request response mode is the most fundamental communication model in THE HTTP protocol, which is commonly referred to as “one round, one back”.
6. Stateless
The HTTP protocol is stateless, which is a double-edged sword
What is statelessness
Statelessness, figuratively speaking, is the inability to remember. There is no state specified in the entire protocol, the client and server are in a physical state. The two devices do not know each other before the connection is established, and each packet sent and received is independent of each other.
For example, the browser sends A request that says, “This is Xiao Ming, please give me file A.” After receiving the message, the server checks the permission to see that Ming can really access file A and sends the file back to the browser. The browser then wants the B file, but the server does not record the status of the request and does not know that the second request and the first request are from the same browser, so the browser must repeat its identity: “THIS is Xiao Ming, please give me the B file again.”
Advantages of statelessness
Since the server has no “memory”, no additional resources are required to record state information, which is not only easier to implement, but also reduces the burden on the server, allowing more CPU and memory to be used for external services.
In addition, “stateless” also means that the servers are the same, there is no difference in “state”, so it is easy to form a cluster, let the load balance to forward requests to any server, there will be no error processing because of inconsistent state, easily achieve high concurrency and high availability.
Disadvantages of statelessness
The server does not have “memory” and therefore cannot support “transaction” operations that require multiple consecutive steps. Electrical business shopping, for example, to log in first, and then add a shopping cart, order, settlement, payment again, this a series of operations need to know the identity of the user, but the “stateless server” is not know these requests are interrelated, each time to ask again identity information, not only trouble, but also increased the unnecessary quantity of data transmission.
But don’t forget that HTTP is “flexible and extensible.” While there is no “state” in the standard, it can be implemented in other ways, such as cookies.
7. Plaintext transmission
Another “double-edged sword” in HTTP is plaintext transmission.
“Plaintext” means that the packets in the protocol (specifically, the header part) do not use binary data, but use simple and readable text.
Advantages of plaintext
Compared with binary protocols such as TCP and UDP, it has obvious advantages. Without using any external tools, you can view or modify packets captured using a browser, Wireshark, or tcpdump with your naked eyes, which greatly facilitates development and debugging.
The disadvantage of clear HTTP message plaintext drawback is obvious, all the information that will be exposed to “light”, in the long transmission link on every link of them have no privacy at all, malicious people invaded in the link of some equipment, simply “bypass” the flow, can realize the communication of peep.
HTTP method of transferring large files
We know that HTTP can transfer many kinds of data, not only text, but also pictures, audio, and video. It’s a big change from the early days of the Internet, when text and pictures were mostly just a few kilobytes in size. Web pages contain too much information, just like a home page HTML may be hundreds of K, high-quality pictures are M, not to mention those movies, TV dramas, a few gigabytes, dozens of gigabytes are possible. In contrast, a 100M fibre optic fixed network or 4G mobile network becomes a “water pipe” under the pressure of these large files, which will overflow the network transmission links, whether uploading or downloading.
Therefore, how to efficiently and quickly transfer these large files under the limited bandwidth becomes an important topic. It’s like having opened the refrigerator door (connection), how do you get the elephant (file) in and then close the door (transfer)?
Today we’ll take a look at some of the ways HTTP can solve this problem.
1. Data compression
Usually the browser sends the request with an accept-encoding header, which contains a list of supported compression formats, such as gzip, so that the server can select a compression algorithm and place it in the content-encoding header. The original data is then compressed and sent to the browser. If the compression rate can have 50%, that is to say, 100K of data can be compressed into 50K size, then it is equivalent to in the case of constant bandwidth network speed has been doubled, the acceleration effect is very obvious.
2. Block transmission
Data compression is to make a large file smaller as a whole. If a large file cannot be made smaller as a whole, it is “broken down” into smaller pieces, which are sent to the browser in batches, and then assembled and restored.
In HTTP, the chunked Encoding is chunked, and the header field transfer-encoding is used in response messages. “Chunked” means that the body part of the packet is not sent at once, but is sent in chunks one by one.
Let’s take a look at the coding rules of block transfer:
- Each block contains two parts, a length header and a data block;
- The length header is a line of plain text ending in CRLF (carriage return newline, i.e. \r\n), with a hexadecimal number to indicate the length;
- The data block follows the length header and also ends with CRLF, but the data does not contain CRLF;
- The end is indicated by a block of length 0.
When the browser receives the data in chunks, it automatically removes the chunk code and reassembles the content.
3. Scope request
With the chunking transfer encoding, the server can easily send and receive large files, but there are some issues to consider for large files on G.
For example, if you are watching a popular time travel show and want to skip the opening title and go straight to the main movie, or if you have a boring story and want to drag the progress bar to fast forward for a few minutes, you actually want to get a piece of data from a large file, and chunking doesn’t have the ability.
To meet these requirements, THE HTTP protocol introduced the concept of “range requests,” allowing clients to use specialized fields in request headers to indicate that only a portion of a file is being retrieved.
The request header Range is a special field for HTTP Range requests in the format of bytes=x-y, where x and y are data ranges in bytes.
HTTP connection management
Short connection
The HTTP protocol was originally (0.9/1.0) a very simple protocol, and the communication process also adopted a simple “request-reply” mode.
The underlying data transmission is based on TCP/IP. Each time a request is sent, a connection must be established with the server. After receiving the response packet, the connection is immediately closed.
Because the entire connection between a client and a server is short and does not remain connected for a long time, it is called a short-lived connection. The early HTTP protocol was also known as a “connectionless” protocol.
The disadvantage of “short connections” is quite serious, because establishing and closing a connection in TCP is an “expensive” operation. TCP requires a three-way handshake to establish a connection, and a four-way handshake to close a connection, resulting in low transmission efficiency.
A long connection
In response to the shortcomings of short connections, HTTP proposes a “persistent connections” communication method, also known as “persistent connections”, “keep alive”, “connection reuse”.
In fact, the solution is very simple, with the idea of “cost sharing”, since the TCP connection and closing is very time-consuming, so the time cost of the original “request-reply” evenly spread to multiple “request-reply”.
HTTP/1.1 enables long connections by default. Multiple requests and responses are sent and received on one connection, improving transmission efficiency. Too many long connections can consume server resources, so the server selectively closes long connections using policies.
The diagram below shows a short connection compared to a long connection:
Header fields associated with the connection
Because long connections can significantly improve performance, they are enabled by default for HTTP/1.1 connections. You don’t need a special header field to specify that once the first request is sent to the server, subsequent requests will reuse the first TCP connection, known as a long connection, to receive and receive data on this connection.
Of course, we can also specify the long Connection mechanism in the request header, using the Connection field and the value “keep-alive”.
However, regardless of whether the client explicitly requests long connections or not, if the server supports long connections, it will always put a “Connection: keep-alive” field in the response message telling the client, “I support long connections, so use this TCP to send and receive data.”
Disadvantages of long connections
If the TCP connection is left open for a long time, the server must keep its state in memory, which consumes the server’s resources. If there are a large number of idle long connections, the resources of the server will soon be exhausted and the server will not be able to provide services to the users who really need them.
Therefore, long connections also need to be closed at the appropriate time and cannot remain connected to the server forever, which can be done on either the client or the server.
On the client side, you can add a “Connection: close” field to the request header to tell the server, “Close the Connection after this communication.” When the server sees this field, it knows that the client is actively closing the connection. Therefore, the server adds this field to the response packet and closes the TCP connection after sending it. In addition, you can disable a long connection by setting the timeout period and the maximum number of requests that can be sent over a long connection.
Queue Head blocking
Queue head blocking is caused by the basic REQUEST-reply model of HTTP.
Because HTTP dictates that packets must be received on a first-in, first-out (FIFO), a “serial” queue is formed. Requests in the queue have no priority, only the order in which they were queued, with the first request being processed first. If the request at the head of the queue is delayed because it is being processed too slowly, then all the requests at the back of the queue have to wait too, resulting in other requests incurring undue time costs.
Use the analogy of a punch card. In our company, we punch in by fingerprint. When we go to work, everyone is queuing to punch in, but at this time, the first person in front of us cannot be recognized by the punch in machine, so we cannot successfully punch in, and others cannot jump the queue, which will cause the people behind us to be late.
Performance optimization
Because the request-reply model cannot be changed, the “queue head blocking” problem cannot be solved in HTTP /1.1. The “queue head blocking” problem, which causes performance degradation, can be mitigated by “concurrent connections” and “domain sharding” techniques.
- Concurrent connections
The company can buy more punch machines to put on the front desk, so that we can not crowded in a queue, scattered punch, a team occasionally blocked it does not matter, the people behind can go to other punch machines.
This is known in HTTP as “concurrent connections”, which means making multiple long connections to the same domain name.
The HTTP protocol recommends that clients use concurrency, but not abuse it. The RFC provides that a client can create a maximum of 6-8 concurrent connections.
- Domain name subdivision
Another way to use quantity to solve quality is domain name sharding.
HTTP protocol and browser limit the number of concurrent connections, so I will open more domain names, such as nginx1.com, nginx2.com, nginx3.com, and all these domain names point to the same server nginx.com, so that the actual number of long connections up.
HTTP Cache
Cache is an important concept in the field of computer, and it is a powerful tool to optimize system performance.
Due to long links and uncontrollable network latency, it is costly for a browser to obtain resources using HTTP. Therefore, it is necessary to cache the “hard-won” data, and reuse it as much as possible when the next request is made, so as to avoid the communication cost of multiple response requests, save network bandwidth, and speed up the response speed.
In fact, almost every part of the HTTP transport has a cache, which is very complex.
Based on the characteristics of the “request-reply” pattern, it can be roughly divided into client-side cache and server-side cache.
Server cache control
The initial process for the client to initiate a request to the server is as follows:
- The browser finds no data in the cache and sends a request to obtain resources from the server.
- The server responds to the request, returns the resource and marks the expiration date of the resource.
- The browser caches resources for reuse.
Here is a detailed request-reply process:
- Resources have an expiration date
The header field used by the server to mark the validity period of the resource is “cache-control”, in which the value “max-age=30” is the validity period of the resource, which is equivalent to telling the browser, “This page can only be cached for 30 seconds, after which it will expire, it can not be used again.”
-
Duration In this case, max-age indicates the duration. The starting point of time calculation is the creation time of the response packet (the Date field, that is, the time when the client leaves the server), not the time when the client receives the response packet, which includes the time consumed during link transmission.
-
Max-age is the most commonly used attribute for HTTP cache control, and there are other attributes that can be used in response messages to more precisely indicate how the browser should use the cache:
- No-store: cache is not allowed. It is used for some data that changes very frequently, such as seckill pages.
- No-cache: It can be cached, but before using it, the server must check whether it has expired and the latest version.
- Must -revalidate: If the cache does not expire, it can continue to be used, but if it expires, it must be verified by the server.
The following is a Cache policy flow for the server:
Client cache control
In fact, not only the server can issue a cache-control header, but also the browser can issue a cache-control header, which means that request-responder parties can use this field for Cache Control and negotiate Cache usage policies.